Jenkins Basics

-
Jenkins Build Parameters

Jenkins is the widely adopted open source continuous integration tool. You can orchestrate any application deployments using Jenkins with a wide range of plugins and native Jenkins workflows. In This Tutorial we will be demonstrates interesting features of jenkins pipeline.

In this tutorial declarative pipeline syntax will be used.

Note
 official documentation for jenkins pipeline syntax

Jenkins Build Parameters

Build Parameters makes jenkins job more dynamic in nature ,which prompt user before triggering the job. Following different types of build parameters that jenkins supports.

build_parameters.groovy
node {
    env.GRADLE_USER_HOME = "${workspace}/.gradle" // pipeline specific env variable
}

pipeline { agent { label "${params.AGENT}" }

    parameters {
        string(name: 'AGENT', defaultValue: 'node-1 || node-2 ')
        choice(name: 'BRANCH', choices: ['master', 'develop'])
        booleanParam(name: 'RUN_TESTS', defaultValue: true)
        password(name: 'PASSWORD', defaultValue: 'SECRET')
        text(name: 'DESCRIPTION', defaultValue: 'description')
        extendedChoice(name: 'OPTIONS', defaultValue: 'foo', description: '', descriptionPropertyValue: '', multiSelectDelimiter: ',', quoteValue: true, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'foo,bar', visibleItemCount: 2)
    }

    stages {
        stage('One') {
            steps {
                print 'Running stage one'
            }
        }

        stage('Two') {
            steps {
                print 'Running stage two'
            }
        }
    }

}

Invoking Downstream Job with Build Parameters

In Jenkins CI-CD Pipeline , a parent job can trigger child job with build parameters.

invoke_downstream_job.groovy
node { env.GRADLE_USER_HOME = "${workspace}/.gradle" }

pipeline {
    agent any

    parameters {
        string(name: 'AGENT', defaultValue: 'node-1||node-2 ')
        choice(name: 'BRANCH', choices: ['master', 'develop'])
        booleanParam(name: 'RUN_TESTS', defaultValue: true)
        password(name: 'PASSWORD', defaultValue: 'SECRET')
        text(name: 'DESCRIPTION', defaultValue: 'description')
    }

    stages {
        stage('Trigger Test Job') {
            when {
                expression {
                    // Conditional flow
                    return params.RUN_TESTS
                }
            }
            steps {
                script {
                    def jobResult = build job: "integration_tests",
                            propagate: true,
                            wait: true,
                            quietPeriod: 2,
                            parameters: [
                                    string(name: 'BRANCH', value: params.BRANCH),
                                    booleanParam(name: 'RUN_TESTS', value: params.RUN_TESTS),
                                    string(name: 'AGENT', value: params.AGENT)]

                }
            }
        }
    }

}
Tip
 propagate true means child job result sends back to parent, wait true means , parent waits until child job completes.

Reading Job results from child Job

When Job-Parent invokes Job-Child , Job-Child can send result back to parent using enviroment variables.

job-child.groovy
pipeline {
    agent any
    stages {
        stage('ChildJob Stage'){
            steps { // run you job work
                // set env variable which is available to caller (parent job)
                env.setProperty("ARTIFACT_VERSION", 'GENERATED_ARTIFACT_VERSION') //(1)
            }
         }
    }
}

  1. Add env variable in the child job

job-parent.groovy
pipeline {

    agent any

    stages {
        stage('Parent Job') {

            steps {

                script {

                    def jobResult = build job: "integration_tests",
                            propagate: true,
                            wait: true,
                            quietPeriod: 2,
                            parameters: [
                                    string(name: 'BRANCH', value: params.BRANCH),
                                    booleanParam(name: 'RUN_TESTS', value: params.RUN_TESTS),
                                    string(name: 'AGENT', value: params.AGENT)
                            ]

                    def envVars = jobResult.getBuildVariables() // (2)
                    print "Received ARTIFACT_VERSION " + envVars['ARTIFACT_VERSION'] //(3)

                }
            }
        }
    }
}

  1. Read env variables from child job through jobResult.getBuildVariables() method.

  2. Read key value map of env variables using envVars['ARTIFACT_VERSION'].

Global Tools

Install JDK as global tool and use it under pipeline.

global_tools.groovy
pipeline {

    agent any

    options {
        timeout(time: 80, unit: 'MINUTES')// Max time to finish this job
    }

    tools {
        jdk 'jdk-1.8.0_212' //which will refer JDK at scratch/comdevus/jenkins/tools/hudson.model.JDK/jdk-1.8.0_212
    }

}

Wait until block

wait_until.groovy
node {
env.GRADLE_USER_HOME = "${workspace}/.gradle"
 currentBuild.displayName = "#${env.BUILD_NUMBER}[${params.BRANCH}]" //custom build number
}

pipeline {
    agent any

    stages {
        stage('Wait for') {

            steps {
                waitUntil {
                    script {

                        try {
                            sleep(time: 1, unit: "MINUTES")

                            def status = sh(returnStdout: true, script: """

                            docker inspect --format='{{.State.Health.Status}}' mydocker

                            """).trim()

                            (status == 'healthy')

                        } catch (e) {
                            e == 'error'
                        }

                    }
                }
            }
        }
    }

}

Edit properties/ json file within Jenkins

edit.groovy
def updateGradleProperties() {

  def fileText = readFile(file: "${env.WORKSPACE}/gradle.properties")
  fileText = (fileText =~ /mykey=.*/).replaceFirst("myvalue")
  writeFile(file: "${env.WORKSPACE}/gradle.properties", text: fileText)

}

def updateJson() {

  def inputFile = readFile(file: "${env.WORKSPACE}/my-custom.json")
  def manifestJSON = new JsonSlurper().parseText(inputFile)
  def builder = new JsonBuilder(manifestJSON)

  builder.content.'child' = 'value'

  String result = builder.toPrettyString()

  writeFile(file: "${env.WORKSPACE}/my-custom.json", text: result)

}

Jenkins Job kill forcibly

This article explains different ways to kill jenkins job forcibly.

Kill Jenkins Job

How to push Code to Git repository from Jenkins

git-commit-push.groovy
stage("Commit") {
    steps {
        dir('myrepo') {

            sh("""
                    git checkout -b pipeline_${env.GIT_BRANCH}
                    git config user.name 'xxxxxx'
                    git config user.email 'xxxxxx@gmail.com'
                    git add -A
                    git diff-index --quiet HEAD || git commit -m 'Pipeline updated'
            """)
        }
    }
}



stage("Push") {

    environment {
        GIT_AUTH = credentials('github-credential-id')
    }

    steps {
        dir('myrepo') {

            sh("""
                git config --local credential.helper "!f() { echo username=\\$GIT_AUTH_USR; echo password=\\$GIT_AUTH_PSW; }; f"
                git push origin HEAD:pipeline_${env.GIT_BRANCH}
            """)

        }
    }
}

How to run parallel stages

parallel-stages.groovy
stage('Build Parallel') {

    parallel {

        stage('p1'){

        }

        stage('p2'){

        }

        stage('p3'){

        }

    }
}

Jacoco Integration in build.gradle

Configure codeCoveragePlugin for gradle multi module project and publish report using Jenkins Job.

add below code in main build.gradle file

build.gradle
allprojects {
  apply plugin: 'jacoco'
}


def jacocoClassDirs = fileTree(project.rootDir.absolutePath)
                        .include('**/build/classes/java/main/**')
                        //Exclude two classes with the same name for a successful build
                        .exclude('**/Config.class')
                        //Exclude swagger-generated code
                        .exclude('**/model/**')
                        .exclude('**/*Mock*.class')



task codeCoverageReport(type: JacocoReport) {

    // Gather execution data from all sub-projects
    // (change this if you e.g. want to calculate unit test/integration test coverage separately)
    executionData fileTree(project.rootDir.absolutePath).include("**/build/jacoco/*.exec")
    getClassDirectories().setFrom(jacocoClassDirs)

    // Add all relevant sourcesets from the subprojects
    subprojects.each {
       sourceSets it.sourceSets.main
    }

    reports {
      xml.enabled true
      html.enabled true
      html.destination "${buildDir}/reports/jacoco"
      csv.enabled false
    }

}

// always run the tests before generating the report
codeCoverageReport.dependsOn {
    subprojects*.test
}

alternative way from official doc

build..gradle
plugins {
    id 'java'
    id 'jacoco'
}

repositories {
    jcenter()
}

// A resolvable configuration to collect source code
def sourcesPath = configurations.create("sourcesPath") {
    visible = false
    canBeResolved = true
    canBeConsumed = false
    extendsFrom(configurations.implementation)
    attributes {
        attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, Usage.JAVA_RUNTIME))
        attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category, Category.DOCUMENTATION))
        attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named(DocsType, 'source-folders'))
    }
}

// A resolvable configuration to collect JaCoCo coverage data
def coverageDataPath = configurations.create("coverageDataPath") {
    visible = false
    canBeResolved = true
    canBeConsumed = false
    extendsFrom(configurations.implementation)
    attributes {
        attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, Usage.JAVA_RUNTIME))
        attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category, Category.DOCUMENTATION))
        attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named(DocsType, 'jacoco-coverage-data'))
    }
}

// Task to gather code coverage from multiple subprojects
def codeCoverageReport = tasks.register('codeCoverageReport', JacocoReport) {
    additionalClassDirs(configurations.runtimeClasspath)
    additionalSourceDirs(sourcesPath.incoming.artifactView { lenient(true) }.files)
    executionData(coverageDataPath.incoming.artifactView { lenient(true) }.files.filter { it.exists() })

    reports {
        // xml is usually used to integrate code coverage with
        // other tools like SonarQube, Coveralls or Codecov
        xml.enabled true

        // HTML reports can be used to see code coverage
        // without any external tools
        html.enabled true
    }
}

// Make JaCoCo report generation part of the 'check' lifecycle phase
tasks.named("check") {
    dependsOn(codeCoverageReport)
}


// Override coverageThreashold value
task codeCoverageVerificationRoot(type: JacocoCoverageVerification) {
  executionData jacocoExecutionData
  getClassDirectories().setFrom(jacocoClassDirs)

  doFirst {
    subprojects.findAll { subproject ->
      subproject.pluginManager.hasPlugin('java')
    }.each { subproject ->
      additionalSourceDirs files(subproject.sourceSets.main.allJava.srcDirs)
    }
  }

  violationRules {
    rule {
      limit {
        minimum = new BigDecimal(codeCoverageMinimum)
      }
    }
  }
}

Run below task to generate report

run.sh
$ ./gradlew codeCoverageReport

Jacoco PublishReport Plugin

This Jacoco jenkins Publish Report plugin will helps to generate report without adding Jacoco related code as shown in Jacoco Integration in build.gradle

JenkinsPipeline.groovy
      steps {
        step([$class                    : 'JacocoPublisher',
              execPattern               : '**/build/jacoco/*.exec',
              classPattern              : '**/build/classes',
              sourcePattern             : '**/src/main/java',
              exclusionPattern          : '**/*Exception*,**/*Mock*,**/*Config*,**/*Log*,**/*Test*,**/*.model.*/**',
              //Check this to set the build status to unstable if coverage thresholds are violated.
              changeBuildStatus         : true,
              //Check this to set the build status to failure if the delta coverage thresholds are violated.
              // Delta coverage is the difference between coverage of last successful and current build.
              buildOverBuild            : true,
              minimumInstructionCoverage: "30",
              minimumBranchCoverage     : "30",
              minimumComplexityCoverage : "30",
              minimumLineCoverage       : "30",
              minimumMethodCoverage     : "30",
              minimumClassCoverage      : "30",
              skipCopyOfSrcFiles        : false, //Check this to disable display of source files for each line coverage
              maximumInstructionCoverage: "50",
              maximumBranchCoverage     : "45",
              maximumComplexityCoverage : "45",
              maximumLineCoverage       : "65", // line coverage is base for code-coverage
              maximumMethodCoverage     : "70",
              maximumClassCoverage      : "70"
        ])
      }
Note
 To make build unstable you need to specify both min, max values as (min <max < Actual)

Publishing Artifacts and HTMLReport to Build Job

If your Jenkins Job Generates TestReport / any other artifacts. you can attach it to Jenkins Job.

publish.groovy
  post {

    always {
        //This will attach build artifacts such as logs , jars
      archiveArtifacts artifacts: 'reports/**/*', allowEmptyArchive: true, onlyIfSuccessful: false

      script {

        try{
          // This will attach HTML report and add link in the left side panel of your Jenkins Job.
          publishHTML (target: [
            allowMissing: false,
            alwaysLinkToLastBuild: true,
            keepAll: true,
            reportDir: 'reports',
            reportFiles: 'test-report.html',
            reportName: 'Test Report',
            escapeUnderscores: true
          ])
        }catch(e){
          print "Report Publish error "+e
        }
      }
    }

  }

Jenkins HTML Display Error

run below command in jenkins script console (https://www.thetopsites.net/article/54362637.shtml)

console.groovy
  System.setProperty("hudson.model.DirectoryBrowserSupport.CSP", "default-src * 'unsafe-inline' 'unsafe-eval'; script-src * 'unsafe-inline' 'unsafe-eval'; connect-src * 'unsafe-inline'; img-src * data: blob: 'unsafe-inline'; frame-src *; style-src * 'unsafe-inline';")

Reset Jenkins Job Number

If you want to delete range of failed jobs from Jenkins pipeline , run below script from jenkins console with your job name and provide resetNumberTarget.

reset_jobId.groovy
resetNumberTarget = 100
item = Jenkins.instance.getItemByFullName("[your_job_name]")
item.builds.each() { build ->

  if(build.number >= resetNumberTarget)
  {
    build.delete()
  }
}

item.updateNextBuildNumber(resetNumberTarget)

Comments

Post a Comment

Popular posts from this blog

IBM Datapower GatewayScript

-
Introduction IBM API Assembly Language Introduction GatewayScript is a new transformation technology for API, Web, and Mobile and is available in DataPower. GatewayScript is based on the JavaScript programming language. More precisely, it supports the ECMAScript 5.1 version of the specification. On the DataPower appliance, JavaScript runs in ‘strict’ mode only. The strict mode provides a more consistent and more disciplined behavior. For example, if a script attempts to write a property of a DataPower API object and that object is frozen, an immediate exception is thrown. The ‘strict’ mode behavior supports stronger error checking and handling and increases security. DataPower Playground (GatewayScript Fiddle) IBM DataPower Playground is an interactive website that lets you write GatewayScript code and execute it on a cloud-hosted DataPower Gateway for learning purposes. Available at : https://datapower-playground.mybluemix.net/ GatewayScript provides
Read more

Spring boot Kafka Integration

-
Image
Introduction Introduction K afka is an open source streaming platform which scales very well in a horizontal way without compromising speed and efficiency. The Kafka core is written in Scala , and Kafka Streams and KSQL are written in Java . Features Kafka is a service bus To connect heterogeneous applications, we need to implement a message publication mechanism to send and receive messages among them. A message router is known as message broker . Kafka is a message broker , a solution to deal with routing messages among clients in a quick way. Kafka architecture has two directives The first is to not block the producers (to deal with back pressure). The second is to isolate producers and consumers . The producers should not know who their consumers are, hence the kafka follows the dumb broker and smart clients model. Tip Back pressure is to make applications robust against data surges. when producer generates data higher rate tha
Read more

Spring boot SOAP Web Service Performance

-
Image
Consumng This tutorial walks you through the process of consuming a SOAP-based web service with Spring. It elucidates following items Making parallel calls with RxJava (or WebFlux ) Sending large SOAP message payloads using AXIOM( AXis Object Model) AxiomSoapMessageFactory .(which improves performance) Marshalling( is the process of transforming the memory representation of an object to a data format suitable for storage or transmission) Marshalling (XML Serialization) done with O/X Mappers, different types of mappers out there and choose appropriate one for your need. Castor XML mapping is an open source XML binding framework. It allows you to transform the data contained in a java object model into/from an XML document. By default, it does not require any further configuration, though a mapping file can be used to have more control over the behavior of Castor. The Spring integration classes reside in the org.springframework.oxm.castor pa
Read more