.. _ci-jenkins:

Zoe Continuous Integration with Jenkins
=======================================

Overview
--------

  - Integrate Zoe repository to Jenkins and SonarQube

  - Each commit to Zoe repository trigger a build at Jenkins:

    - Run SonarQube Scanner to analyze the codebase
    - Create two containers for zoe-master, zoe-api
    - Run integration test [testing rest api]
    - Build new images if no errors happen
    - Deploy Zoe with latest images

Software Stack
--------------

  - Jenkins - version 2.7.4
  - SonarQube - version 6.1

Configuration
-------------

- Jenkins: all the configurations in this section is configured on Jenkins Server

  - Required:

    - Plugins: Github plugin, SonarQube Plugin, Quality Gates, Email Plugin (optional), Cobertura Coverage Report (optional)
    - Softwares: Java, Python, Docker
      - Go to **Manage Jenkins**, then  **Global Tool Configuration** to setup Java SDK, SonarQube Scanner

  - SonarQube server configuration: this aims to connect Jenkins and SonarQube together

    - Go to **Manage Jenkins** then **Configure System**
    - SonarQube servers: input name, server URL, server version, **sever authentication token** (created on SonarQube Server)

  - Quality Gates configuration:

    - Go to **Manage Jenkins** then **Configure System**
    - Quality Gates: input name, server URL, username and password to login into SonarQube server

  - Github Servers configuration:

    - Go to **Manage Jenkins** then **Configure System**
    - Github: **Add Github Server**, the API URL would be ``https://api.github.com``. The credentials creation is well defined in the document of Github plugin:

      - You can create your own [personal access token](https://github.com/settings/tokens/new) in your account GitHub settings.
      - Token should be registered with scopes:
        - admin:repo_hook - for managing hooks (read, write and delete old ones)
        - repo - to see private repos
        - repo:status - to manipulate commit statuses
      - In Jenkins create credentials as «Secret Text», provided by Plain Credentials Plugin

  - Create credentials for Github account: this is similar when you want to [connect to Github over SSH](https://help.github.com/articles/connecting-to-github-with-ssh/), here, beside adding your public key to Github, you also need to add your private key to Jenkins.

    - Create SSH key pair on the machine run Jenkins:

      - Add public key to Github
      - Add private key to Jenkins credentials

  - Create new item as a **freestyle project**: this aims to create a Jenkins job with the github repository

    - General

      - Select Github project
      - Insert project URL

    - Source Code Management

      - Select **Git**
      - Repositories

        - Repository URL: use **SSH URL** of Github repository
        - Credentials: select the one created above

    - Build Triggers

      - For Github plugin with version before 1.25.1: Select **Build when a change is pushed to Github**
      - For Github plugin with version from 1.25.1: Select **GitHub hook trigger for GITScm polling**

    - Build

      - Add **Execute SonarQube Scanner** to do SonarQube Analysis
      - Add **Quality Gates** to break the build when the SonarQube quality gate is not passed
      - Add **Execute Shell** to run script for testing, deploying. Please refer to the Appendix section for the script.

    - Post-build Actions [Optional]

      - Add **Publish Covetura Coverage Report** for getting report from coverage. Due to the shell script in Appendix, the xml file generated by coverage is located at ``test`` folder, so, we should put ``**/tests/coverage.xml`` as the input of the field **Cobertura xml report pattern**.
      - Add **E-mail Notification** for notifying when jobs finish

- Github

  - Add new SSH key (the one created on Jenkins server)
  - Go to the project (which is integrated to Jenkins) settings

    - Integration & Services

      - Add Service, choose **Jenkins (Github plugin)**
      - Add Jenkins hook url

        - For github plugin, this one would have the format: http://your-jenkins.com/github-webhook
        - In case your Jenkins doens't expose to the world, try https://ngrok.com/

- SonarQube: all the configurations in this section is configured on SonarQube Server

  - On **Administration**, go to **My Account**, then **Security**
  - Generate Tokens, copy this and paste to **server authentication token** on Jenkins configuration
  - The project needs to provides **sonar-properties** file in the repo:(http://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner)
  - Then, on System then Update Center, install two plugins for Python and TypeScript.

Appendix
--------

- Sonar properties files

  - Take a look at sonar-project.properties files in root, ``zoe_api``, ``zoe_master``, ``zoe_lib``, ``zoe_fe`` folders.

- Execute Shell Script

  - Push this script inside the execute shell script of Jenkins job you created above, the zoe_rest_api can be changed in the ``test_config.py`` file.

::

  # Run Style checker for Sphinx RST documentation
  doc8 docs/

  # Build new container images
  python3 ci/zoeci.py 1 tcp://192.168.12.2:2375 192.168.12.2:5000/zoe:$BUILD_ID

  # Deploy new zoe with the above images for testing
  python3 ci/zoeci.py 0 tcp://192.168.12.2:2375 ci/docker-compose-test.yml 192.168.12.2:5000/zoe:$BUILD_ID

  # Run integration test
  cd tests
  coverage run -p basic_auth_success_test.py
  coverage run -p cookie_auth_success_test.py
  coverage combine
  coverage xml
  cd ..

  # Push the built images above to local registry
  python3 ci/zoeci.py 2 tcp://192.168.12.2:2375 192.168.12.2:5000/zoe:$BUILD_ID

  # Redeploy zoe with new images
  python3 ci/zoeci.py 0 tcp://192.168.12.2:2375 ci/docker-compose-prod.yml 192.168.12.2:5000/zoe:$BUILD_ID

- Screenshots

  - Jenkins Server configuration

    - Plugin configuration

      - Java SDK Configuration

      .. image:: imgs/1.java.config.png

      - SonarQube Scanner Configuration

      .. image:: imgs/1.2.sonar.config.PNG

      - SonarQube Server Configuration

      .. image:: imgs/2.sonar.config.png

      - Quality Gates Configuration

      .. image:: imgs/2.1.sonar.quality.gates.png

      - Github Server Configuration

      .. image:: imgs/4.1.github.server.config.png

      - Github Server Credential Creation

      .. image:: imgs/4.1.github.server.credential.png

      - Email Notification Configuration

      .. image:: imgs/3.email.config.png

    - Create Github credentials

      .. image:: imgs/4.github.credential.png

    - Create Freestyle project

      .. image:: imgs/5.1.freestyle.project.png

      .. image:: imgs/5.2.freestyle.project.png

      .. image:: imgs/5.3.freestyle.project.png

      .. image:: imgs/5.4.1.freestyle.project.png

      .. image:: imgs/5.4.2.freestyle.project.png

      .. image:: imgs/5.4.3.freestyle.project.png

      .. image:: imgs/5.5.freestyle.project.png

  - SonarQube Configuration

      .. image:: imgs/6.sonar.token.png

  - Github Repository Configuration

    - Create webhook service

    .. image:: imgs/7.github.repo.png

    - Create access token

    .. image:: imgs/7.1.github.access.token.png