How to upload (big) files to Jenkins job as build parameter

Adrian Citu

Context

Originally, Jenkins had a mechanism to upload files as build parameters but this mechanism was rather faulty (see JENKINS-27413 and JENKINS-29289 ).

A new mechanism was proposed (for Jenkins2 only) via the File Parameter plug-in. The plug-in offers the possibility to capture files as build parameters like this:

def fb64 = input message: 'upload', parameters:  [base64File('file')]
node {
    withEnv(["fb64=$fb64"]) {
        sh 'echo $fb64 | base64 -d'
    }
}

Problem

If you look closer to the File Parameter plug-in documentation it said that: “You can use Base64 parameters for uploading small files in the middle of the build”. What does it means “small files” in terms of size is not mentioned but if you try the previous example with files bigger than 2 kBytes then the job will fail with the following error:

java.io.IOException: error=7, Argument list too long
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
        at java.lang.ProcessImpl.start(ProcessImpl.java:134)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
...
Caused: java.io.IOException: Cannot run program "nohup" (in directory "/var/jenkins_cache/workspace/testproject"): error=7, Argument list too long

Solution

What is the root cause of this exception ? I’m not exactly sure but I think that  sh ‘echo $fb64 | base64 -d‘ command will transfer as environment variable the file to the Jenkins slave executing the job and something into this transfer mechanism is not very robust.

I propose two ways to workaround this problem:

Solution 1: Don’t send the uploaded file as environment variable to sh

Don’t send the uploaded file as environment variable to ‘sh‘ and write the file directly into the workspace:

withEnv(["fb64=$fb64"]) {
    script{
        def  decoded = new String(fb64.decodeBase64())
        writeFile file:"uploaded_file.txt", text: decoded
        sh 'cat ${WORKSPACE}/uploaded_file.txt'
    }

The drawback of this solution is that you’ll have to write the uploaded file somewhere into your workspace, so if you want to store it into another location then you’ll have to add some extra steps to the pipeline.

Solution 2: Don’t use withEnv pipeline step

The second solution is not using the withEnv pipeline step and just directly use the sh ‘echo $fb64 | base64 -d‘ command from a script step:

script{
         sh "set +x; echo '$fb64' | base64 -d > /tmp/uploaded_file.txt"
         cat '/tmp/uploaded_file.txt'
      }

Please note that I’m using the “set +x” before the echo command in order to inhibit the output of the command so the Jenkins console/log is not filled-in with base64 encoded characters. Also in this solution you have the freedom to chose the destination of the uploaded file.