Zoe Continuous Integration with Jenkins

Overview

  • Integrate Zoe repository to Jenkins and SonarQube
  • Each commit to Zoe repository trigger a build at Jenkins:
    • Run SonarQube Scanner to analyze the codebase
    • Create two containers for zoe-master, zoe-api
    • Run integration test [testing rest api]
    • Build new images if no errors happen
    • Deploy Zoe with latest images

Software Stack

  • Jenkins - version 2.7.4
  • SonarQube - version 6.1

Configuration

  • Jenkins: all the configurations in this section is configured on Jenkins Server
    • Required:
      • Plugins: Github plugin, SonarQube Plugin, Quality Gates, Email Plugin (optional), Cobertura Coverage Report (optional)
      • Softwares: Java, Python, Docker - Go to Manage Jenkins, then Global Tool Configuration to setup Java SDK, SonarQube Scanner
    • SonarQube server configuration: this aims to connect Jenkins and SonarQube together
      • Go to Manage Jenkins then Configure System
      • SonarQube servers: input name, server URL, server version, sever authentication token (created on SonarQube Server)
    • Quality Gates configuration:
      • Go to Manage Jenkins then Configure System
      • Quality Gates: input name, server URL, username and password to login into SonarQube server
    • Github Servers configuration:
      • Go to Manage Jenkins then Configure System
      • Github: Add Github Server, the API URL would be https://api.github.com. The credentials creation is well defined in the document of Github plugin:
        • You can create your own [personal access token](https://github.com/settings/tokens/new) in your account GitHub settings.
        • Token should be registered with scopes: - admin:repo_hook - for managing hooks (read, write and delete old ones) - repo - to see private repos - repo:status - to manipulate commit statuses
        • In Jenkins create credentials as «Secret Text», provided by Plain Credentials Plugin
    • Create credentials for Github account: this is similar when you want to [connect to Github over SSH](https://help.github.com/articles/connecting-to-github-with-ssh/), here, beside adding your public key to Github, you also need to add your private key to Jenkins.
      • Create SSH key pair on the machine run Jenkins:
        • Add public key to Github
        • Add private key to Jenkins credentials
    • Create new item as a freestyle project: this aims to create a Jenkins job with the github repository
      • General
        • Select Github project
        • Insert project URL
      • Source Code Management
        • Select Git
        • Repositories
          • Repository URL: use SSH URL of Github repository
          • Credentials: select the one created above
      • Build Triggers
        • For Github plugin with version before 1.25.1: Select Build when a change is pushed to Github
        • For Github plugin with version from 1.25.1: Select GitHub hook trigger for GITScm polling
      • Build
        • Add Execute SonarQube Scanner to do SonarQube Analysis
        • Add Quality Gates to break the build when the SonarQube quality gate is not passed
        • Add Execute Shell to run script for testing, deploying. Please refer to the Appendix section for the script.
      • Post-build Actions [Optional]
        • Add Publish Covetura Coverage Report for getting report from coverage. Due to the shell script in Appendix, the xml file generated by coverage is located at test folder, so, we should put **/tests/coverage.xml as the input of the field Cobertura xml report pattern.
        • Add E-mail Notification for notifying when jobs finish
  • Github
    • Add new SSH key (the one created on Jenkins server)
    • Go to the project (which is integrated to Jenkins) settings
  • SonarQube: all the configurations in this section is configured on SonarQube Server
    • On Administration, go to My Account, then Security
    • Generate Tokens, copy this and paste to server authentication token on Jenkins configuration
    • The project needs to provides sonar-properties file in the repo:(http://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner)
    • Then, on System then Update Center, install two plugins for Python and TypeScript.

Appendix

  • Sonar properties files
    • Take a look at sonar-project.properties files in root, zoe_api, zoe_master, zoe_lib, zoe_fe folders.
  • Execute Shell Script
    • Push this script inside the execute shell script of Jenkins job you created above, the zoe_rest_api can be changed in the test_config.py file.
# Run Style checker for Sphinx RST documentation
doc8 docs/

# Build new container images
python3 ci/zoeci.py 1 tcp://192.168.12.2:2375 192.168.12.2:5000/zoe:$BUILD_ID

# Deploy new zoe with the above images for testing
python3 ci/zoeci.py 0 tcp://192.168.12.2:2375 ci/docker-compose-test.yml 192.168.12.2:5000/zoe:$BUILD_ID

# Run integration test
cd tests
coverage run -p basic_auth_success_test.py
coverage run -p cookie_auth_success_test.py
coverage combine
coverage xml
cd ..

# Push the built images above to local registry
python3 ci/zoeci.py 2 tcp://192.168.12.2:2375 192.168.12.2:5000/zoe:$BUILD_ID

# Redeploy zoe with new images
python3 ci/zoeci.py 0 tcp://192.168.12.2:2375 ci/docker-compose-prod.yml 192.168.12.2:5000/zoe:$BUILD_ID
  • Screenshots

    • Jenkins Server configuration

      • Plugin configuration

        • Java SDK Configuration
        ../_images/1.java.config.png
        • SonarQube Scanner Configuration
        ../_images/1.2.sonar.config.PNG
        • SonarQube Server Configuration
        ../_images/2.sonar.config.png
        • Quality Gates Configuration
        ../_images/2.1.sonar.quality.gates.png
        • Github Server Configuration
        ../_images/4.1.github.server.config.png
        • Github Server Credential Creation
        ../_images/4.1.github.server.credential.png
        • Email Notification Configuration
        ../_images/3.email.config.png
      • Create Github credentials

        ../_images/4.github.credential.png
      • Create Freestyle project

        ../_images/5.1.freestyle.project.png ../_images/5.2.freestyle.project.png ../_images/5.3.freestyle.project.png ../_images/5.4.1.freestyle.project.png ../_images/5.4.2.freestyle.project.png ../_images/5.4.3.freestyle.project.png ../_images/5.5.freestyle.project.png
    • SonarQube Configuration

      ../_images/6.sonar.token.png
    • Github Repository Configuration

      • Create webhook service
      ../_images/7.github.repo.png
      • Create access token
      ../_images/7.1.github.access.token.png