Russ Treadwell

· 10 min read

Jenkins Jobs as Code

How I do Jenkins now: jobs live in git, with a shared pipeline library, Job DSL job files, and one seed job that builds the rest.

Views: loading

Back in 2020 I wrote Jenkins and UE4. It’s eight steps of clicking through the Jenkins UI to make one job. Create a freestyle project, tick a box, add a parameter, add two more, paste three batch commands into a text box.

That works fine until you have more than a couple of jobs. Then you’re copying jobs by hand, the copies drift apart, nobody remembers who changed what, and the only copy of your build setup is whatever is sitting on the Jenkins box.

The 200 IQ move is to stop configuring jobs in Jenkins at all. Everything lives in a git repo, and Jenkins pulls that repo and builds its own job list from it.

This is how builds work at GhostJam now. One repo holds the job definitions and the pipeline code. A build syncs from Perforce, builds and packages the game, uploads symbols to BugSplat, pushes to Steam, and tells Discord how it went.

I switched for the convenience. Being able to change a pipeline in a script, push it, and have Jenkins pull it makes everything so much easier.

Fair warning: my friend Sunny knows a lot more about devops than I do, so she may have a problem with some of this haha.

The idea

There are three pieces:

  1. A shared library. The pipeline logic, written once.
  2. Job definitions. One small file per job that says what its parameters and settings are, then calls the library.
  3. A seed job. The one job that reads the job definitions and creates everything else.

Pipeline changes take effect on the next build, because Jenkins pulls the library fresh every time. Job changes take effect the next time the seed job runs. Nobody clicks anything.

What you need

  • Jenkins with the Pipeline, Folders, Job DSL and Environment Injector plugins
  • Plugins for whatever your pipeline talks to. For us that’s P4 and Discord Notifier
  • A Windows build agent labeled Windows, and Jenkins credentials for Perforce, Steam and the Discord webhook
  • A git repo for your CI config
  • Patience (still)

The repo

jenkins-config/
├── jenkins/
│   └── job_dsl_config.groovy
├── jobs/
│   ├── my_game.groovy
│   ├── my_game_editor.groovy
│   └── other_game.groovy
└── vars/
    ├── unrealGamePipeline.groovy
    ├── unrealEditorPipeline.groovy
    ├── workspaceUtils.groovy
    ├── p4Checkout.groovy
    ├── steamDeploy.groovy
    └── discordNotify.groovy

The shared library

vars/ is a Jenkins convention. Every file in it becomes a step you can call from any pipeline, named after the file. So vars/discordNotify.groovy gives you a discordNotify step:

GROOVY 
// vars/discordNotify.groovy
def call(String message) {
    withCredentials([string(credentialsId: 'discord-webhook', variable: 'WEBHOOK_URL')]) {
        discordSend(
            webhookURL: env.WEBHOOK_URL,
            title: env.JOB_NAME,
            description: message,
            link: env.BUILD_URL,
            result: currentBuild.currentResult
        )
    }
}

The whole pipeline is one of those steps too. It takes a config map and runs the stages:

GROOVY 
// vars/unrealGamePipeline.groovy
def call(Map config = [:]) {
    def buildConfig = params.BUILD_CONFIGURATION ?: 'Development'

    node('Windows') {
        def client = workspaceUtils.p4Client(env.PROJECT_NAME, config.p4Stream)

        ws(workspaceUtils.buildDir(env.PROJECT_NAME, config.p4Stream)) {
            try {
                stage('Checkout') {
                    p4Checkout(stream: config.p4Stream, workspaceName: client, changelist: config.p4Changelist)
                }

                stage('Build and Package') {
                    def uat = "${env.UE_PATH}\\Build\\BatchFiles\\RunUAT.bat"
                    def project = "${env.WORKSPACE}\\${env.PROJECT_NAME}.uproject"
                    bat "\"${uat}\" BuildCookRun -project=\"${project}\" -platform=Win64 -clientconfig=${buildConfig} -build -cook -stage -pak"
                }

                if (params.ENABLE_STEAM_DEPLOY) {
                    stage('Deploy to Steam') {
                        steamDeploy(appId: env.STEAM_APP_ID, depotId: env.STEAM_DEPOT_ID)
                    }
                }
            } catch (e) {
                currentBuild.result = 'FAILURE'
                throw e
            } finally {
                archiveArtifacts artifacts: '**/Saved/Logs/**/*.log', allowEmptyArchive: true
                discordNotify("${env.GAME_NAME} build ${env.BUILD_NUMBER}: ${currentBuild.currentResult}")
            }
        }
    }
}

The steps it calls are more files in vars/. Ours are longer than these, but that’s the shape of it.

vars/workspaceUtils.groovy
GROOVY 
// vars/workspaceUtils.groovy

// "//MyGame/EngineUpgrade" becomes "EngUpg": the first three letters of each
// word in the last part of the stream.
def shortStream(String stream) {
    return stream.tokenize('/').last()
        .split(/(?=[A-Z])/)
        .collect { it.take(3) }
        .join('')
}

// The name the build directory and the Perforce client are both made from.
// "MyGame_EngUpg", plus the executor number when an agent runs more than one
// build at a time, so two builds never share a directory or a client.
def baseName(String project, String stream) {
    def name = "${project}_${shortStream(stream)}".replaceAll('[^a-zA-Z0-9_]', '')
    def executor = env.EXECUTOR_NUMBER ?: '0'
    return executor == '0' ? name : "${name}_${executor}"
}

// Short build directory. Jenkins' default workspace path is long, and Unreal
// on Windows runs into the 260 character path limit fast.
def buildDir(String project, String stream) {
    return "C:\\B\\${baseName(project, stream)}"
}

// One Perforce client per stream and agent. Sharing a client between streams
// makes Perforce re-sync files with fresh timestamps on every switch, and
// Unreal Build Tool answers that with a full rebuild.
def p4Client(String project, String stream) {
    def agent = (env.NODE_NAME ?: 'agent').replaceAll('[^a-zA-Z0-9]', '')
    return "jenkins-${baseName(project, stream)}-${agent}"
}
vars/p4Checkout.groovy
GROOVY 
// vars/p4Checkout.groovy
def call(Map args) {
    checkout([
        $class: 'PerforceScm',
        credential: 'perforce',
        workspace: [
            $class: 'ManualWorkspaceImpl',
            name: args.workspaceName,
            spec: [
                allwrite: true, clobber: true, compress: false,
                line: 'LOCAL', locked: false, rmdir: false,
                // Keep the submit time on synced files. Without this every
                // sync looks like a change and you rebuild everything.
                modtime: true,
                streamName: args.stream,
                view: "${args.stream}/... //${args.workspaceName}/...",
                backup: true
            ]
        ],
        populate: [
            $class: 'SyncOnlyImpl',
            have: true, modtime: true, quiet: true,
            pin: args.changelist ?: ''
        ]
    ])
}
vars/steamDeploy.groovy
GROOVY 
// vars/steamDeploy.groovy
def call(Map args) {
    def contentRoot = args.contentRoot ?: "${env.WORKSPACE}\\Saved\\StagedBuilds\\Windows"
    def steamCmd = env.STEAM_CMD_PATH ?: 'C:\\steamcmd\\steamcmd.exe'
    def vdfDir = "${env.WORKSPACE}\\steam_vdf"
    def appVdf = "${vdfDir}\\app_${args.appId}.vdf"

    writeFile file: appVdf, text: """
"AppBuild"
{
    "AppID" "${args.appId}"
    "Desc" "${env.VERSION} build ${env.BUILD_NUMBER}"
    "ContentRoot" ""
    "SetLive" "${args.branch ?: ''}"
    "Preview" "0"
    "Depots"
    {
        "${args.depotId}" "depot_${args.depotId}.vdf"
    }
}
"""

    writeFile file: "${vdfDir}\\depot_${args.depotId}.vdf", text: """
"DepotBuildConfig"
{
    "DepotID" "${args.depotId}"
    "ContentRoot" "${contentRoot}"
    "FileMapping"
    {
        "LocalPath" "*"
        "DepotPath" "."
        "Recursive" "1"
    }
    "FileExclusion" "*.pdb"
}
"""

    // Use a dedicated Steam account, and log in with it once by hand on the
    // build machine so Steam Guard is out of the way.
    withCredentials([usernamePassword(credentialsId: 'steam',
                                      usernameVariable: 'STEAM_USER',
                                      passwordVariable: 'STEAM_PASS')]) {
        bat "\"${steamCmd}\" +login %STEAM_USER% %STEAM_PASS% +run_app_build \"${appVdf}\" +quit"
    }
}

There’s a second pipeline in there too. unrealEditorPipeline builds the editor binaries and submits them to Perforce, so the rest of the team gets them through UnrealGameSync instead of compiling anything. A Perforce trigger kicks that job off whenever code is submitted.

vars/unrealEditorPipeline.groovy
GROOVY 
// vars/unrealEditorPipeline.groovy

// Runs the engine's own BuildGraph script, so the engine source has to be in
// the same stream as the project. archiveStream is where the zipped editor
// binaries get submitted, which is where UnrealGameSync picks them up.
// P4_PORT and P4_USER need to be set, for example as global environment
// variables in Jenkins.
def call(Map config = [:]) {
    node('Windows') {
        def client = workspaceUtils.p4Client(env.PROJECT_NAME, config.p4Stream)

        ws(workspaceUtils.buildDir(env.PROJECT_NAME, config.p4Stream)) {
            try {
                stage('Checkout') {
                    p4Checkout(stream: config.p4Stream, workspaceName: client, changelist: config.p4Changelist)
                }

                stage('Build Editor') {
                    def uat = "${env.WORKSPACE}\\Engine\\Build\\BatchFiles\\RunUAT.bat"
                    def graph = "${env.WORKSPACE}\\Engine\\Build\\Graph\\Examples\\BuildEditorAndTools.xml"
                    def args = [
                        'BuildGraph',
                        "-Script=\"${graph}\"",
                        '-Target="Submit To Perforce For UGS"',
                        "-set:EditorTarget=${env.PROJECT_NAME}Editor",
                        "-set:ArchiveStream=${config.archiveStream}",
                        "-P4 -P4Port=${env.P4_PORT} -P4User=${env.P4_USER} -P4Client=${client}",
                        '-buildmachine -submit'
                    ].join(' ')

                    bat "\"${uat}\" ${args}"
                }
            } catch (e) {
                currentBuild.result = 'FAILURE'
                throw e
            } finally {
                discordNotify("${env.GAME_NAME} editor build ${env.BUILD_NUMBER}: ${currentBuild.currentResult}")
            }
        }
    }
}

To register the library, go to Manage Jenkins, System, Global Trusted Pipeline Libraries (older versions say Global Pipeline Libraries). Give it a name, a default branch, and the repo URL. I’m calling it pipeline-library here.

The jobs

First the folders. This script runs before the job scripts.

GROOVY 
// jenkins/job_dsl_config.groovy
folder('Games')
folder('Games/MyGame')

Then one file per job.

GROOVY 
// jobs/my_game.groovy
pipelineJob('Games/MyGame/MyGame_Full') {
    displayName('My Game Full Pipeline')
    description('Generated by the seed job. Edits made in the UI get overwritten.')

    properties {
        disableConcurrentBuilds()
    }

    definition {
        cps {
            script('''
                @Library('pipeline-library') _

                unrealGamePipeline([
                    p4Stream: params.P4_STREAM,
                    p4Changelist: params.P4_CL
                ])
            ''')
            sandbox()
        }
    }

    parameters {
        choiceParam('BUILD_CONFIGURATION', ['Development', 'Shipping'], 'Build configuration')
        choiceParam('P4_STREAM', ['//MyGame/Main', '//MyGame/Release'], 'Perforce stream to build from')
        stringParam('P4_CL', '', 'Changelist to sync to (leave empty for latest)')
        booleanParam('ENABLE_STEAM_DEPLOY', true, 'Deploy to Steam')
    }

    environmentVariables {
        env('GAME_NAME', 'My Game')
        env('PROJECT_NAME', 'MyGame')
        env('UE_PATH', 'C:\\Program Files\\Epic Games\\UE_5.6\\Engine')
        env('STEAM_APP_ID', '480')
        env('STEAM_DEPOT_ID', '481')
        env('VERSION', '1.2.0')
    }

    logRotator {
        numToKeep(30)
        daysToKeep(30)
    }
}

Parameters are the things you pick per build. Environment variables are the things that are fixed for the game. The script hands both to the library, and sandbox() lets it run without admin approval. The underscore after @Library is there because the annotation needs something to attach to.

A new game is one more file with its own names and IDs. None of the pipeline logic gets copied. Our second project keeps the engine source inside its own Perforce stream instead of using an installed engine, and that was one extra option passed to the same pipeline. An editor job is the same file with unrealEditorPipeline in the script instead.

It also makes the weird cases cheap. Bumping the game version for an update is a one-line commit. When we were upgrading engine versions, the upgrade stream needed a different engine than main, and that was a few lines in the job script instead of a second set of jobs:

GROOVY 
if (params.P4_STREAM == '//MyGame/EngineUpgrade') {
    env.UE_PATH = 'C:\\\\Program Files\\\\Epic Games\\\\UE_5.7\\\\Engine'
}

The seed job

This is the only job you make by hand.

  1. Create a new job as a freestyle project.
  2. Under Source Code Management choose Git and connect it to your config repo.
  3. Under Build Triggers check “Build periodically” and put @daily.
  4. Under Build Steps add “Process Job DSLs”, choose “Look on Filesystem”, and put this in DSL Scripts.
jenkins/job_dsl_config.groovy
jobs/*.groovy
  1. Save and run it.

The config script goes first because it creates the folders, and Job DSL can’t put a job in a folder that doesn’t exist yet. After the first run it updates the jobs once a day on its own. Hit Build Now when you don’t want to wait.

The build step has an “Action for removed jobs” setting. The default ignores jobs that are no longer in the scripts, Disable turns them off, and Delete removes them along with their build history.

To retire a job without throwing its definition away, I rename the file to something like my_game_demo.groovy.deprecated. The glob stops matching it.

Things that will bite you

properties() wipes your environment variables

Calling properties([...]) inside the pipeline replaces the job’s entire property list. The environment variables from the job DSL are one of those properties, so they got wiped, and the builds after that had no GAME_NAME or PROJECT_NAME. Job properties like disableConcurrentBuilds() go in the job DSL, not in the pipeline.

?: eats your false

The obvious way to default an option in the library is the Elvis operator. It falls through on anything falsy, not just null, so passing false gets you the default.

GROOVY 
// false becomes true
def enableDeploy = config.enableDeploy ?: true

// what you actually meant
def enableDeploy = config.containsKey('enableDeploy') ? config.enableDeploy : true

Backslashes get unescaped twice

The pipeline script in a job file is a string inside a Groovy script. Windows paths in there need four backslashes, like the engine upgrade example above. In environmentVariables it’s the usual two.

The rest

  • The first seed run fails. You get script not yet approved for use. An admin has to approve the scripts under Manage Jenkins, In-process Script Approval, and again every time one changes. You can turn script security off for Job DSL in the security settings, but then anyone who can push to the config repo is effectively a Jenkins admin.
  • A bad library push breaks every job. To test a change, point one job at a branch with @Library('pipeline-library@my-branch') _ first.
  • Anything that can hang needs a timeout. SteamCMD waiting on a Steam Guard code will hold the build machine until someone notices.
  • Script file names are picky. Letters, digits and underscores only, no leading digit. my-game.groovy won’t load.
  • Secrets don’t go in the repo. That includes webhook URLs. Put them in Jenkins credentials, like discordNotify does above.
  • Don’t guess at the DSL. Your Jenkins lists exactly what’s available at /plugin/job-dsl/api-viewer/index.html.

until next time