Wiring it into your CI server

Run in CI covers what the execution server does headless: pick a suite or a chain, run it, exit with a code your pipeline can gate on. This page is the wiring for the three CI systems people ask about most.

There is no plugin to install for any of them. The execution server is a container that runs one job and exits, so every CI system already knows how to run it. What differs between them is only how you publish the report afterwards.

Before you start

Your tests need to be in git. Point VP_WORKSPACE_DIR at a workspace folder in the repository the pipeline checks out. Mount it read-only; the run never writes to it.

Your runner needs the image. It is not published for anonymous download, so your runner authenticates to wherever we placed it for you, or loads it from an archive you keep internally. See Getting the image.

Decide where the traffic comes from. Two shapes, and the choice is about network position rather than preference:

• The runner does the testing. The container runs the suite itself. Nothing else needs to exist, and there is no token and no network call to VirtuProbe. Use this when your CI runner can already reach the systems under test.
• The runner calls an execution server. You run the execution server inside the target network and your pipeline asks it to run the suite over HTTP. Use this when the runner cannot reach the systems under test, which in most corporate networks it cannot.

Every example below is the first shape. The second is at the end.

GitHub Actions

name: Integration tests
on: [push]

jobs:
  virtuprobe:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run the smoke suite
        run: |
          docker run --rm \
            -v "$PWD/tests:/workspace:ro" -e VP_WORKSPACE_DIR=/workspace \
            -v "$PWD/results:/out" -e VP_OUTPUT_DIR=/out \
            -e VP_RUN=suite:checkout-smoke \
            -e VP_RUN_ENV=staging \
            virtuprobe-execution-server

      - name: Publish the report
        if: always()
        uses: dorny/test-reporter@v1
        with:
          name: VirtuProbe
          path: results/junit.xml
          reporter: java-junit

The docker run step fails the job on a non-zero exit, so you need no extra wiring to make a failed assertion fail the build. if: always() on the reporting step matters: without it the report is skipped precisely when the run failed, which is when you want it.

GitLab CI

integration-tests:
  image: docker:latest
  services:
    - docker:dind
  script:
    - |
      docker run --rm \
        -v "$PWD/tests:/workspace:ro" -e VP_WORKSPACE_DIR=/workspace \
        -v "$PWD/results:/out" -e VP_OUTPUT_DIR=/out \
        -e VP_RUN=suite:checkout-smoke \
        -e VP_RUN_ENV=staging \
        virtuprobe-execution-server
  artifacts:
    when: always
    paths:
      - results/
    reports:
      junit: results/junit.xml

reports: junit: is what puts failures on the merge request rather than only in the job log. Keep when: always for the same reason as above.

If your runner uses the Docker executor rather than docker:dind, drop the image and services lines and run the execution server as the job image directly, passing the same variables.

Jenkins

pipeline {
  agent any
  stages {
    stage('Integration tests') {
      steps {
        sh '''
          docker run --rm \
            -v "$PWD/tests:/workspace:ro" -e VP_WORKSPACE_DIR=/workspace \
            -v "$PWD/results:/out" -e VP_OUTPUT_DIR=/out \
            -e VP_RUN=suite:checkout-smoke \
            -e VP_RUN_ENV=staging \
            virtuprobe-execution-server
        '''
      }
    }
  }
  post {
    always {
      junit 'results/junit.xml'
      archiveArtifacts artifacts: 'results/report.html', allowEmptyArchive: true
    }
  }
}

junit in the post block marks the build unstable on a failed test, which is usually what you want from a test stage. If a failed suite should fail the build outright rather than mark it unstable, leave the step to fail on its own and do not swallow the exit code.

Passing values in

Anything your tests need at run time goes in through VP_RUN_VARS, as key=value pairs separated by newlines or semicolons. Take the values from your CI system’s secret store; nothing is written to the workspace.

- run: |
    docker run --rm \
      -e VP_RUN_VARS="api_token=${{ secrets.API_TOKEN }}
    base_url=https://staging.example.com" \
      ...

Caution

A value passed this way is visible to anything that can read the container’s environment, and it appears in the run’s evidence file. Use it for per-run values, not for passwords.

Credentials your tests reuse

A credential stored in a workspace keeps its host, its username and its scheme in the files you commit. Its secret is never written there, which is what makes the workspace safe to put in git, and also what leaves a fresh runner holding a credential with no password.

Supply the secret at run time instead. A mounted file is the better of the two, because it can be read-only and does not end up in the environment of everything the run spawns:

docker run --rm \
  -v "$PWD/secrets.json:/run/secrets/vp.json:ro" \
  -e VP_CREDENTIALS_FILE=/run/secrets/vp.json \
  ...

where the file is keyed by credential name:

{
  "staging-db": { "username": "svc", "password": "..." },
  "partner-api": { "token": "..." }
}

Write that file from your CI system’s secret store as a step before the run, and delete it afterwards.

For one or two values an environment variable is simpler. The name is VP_CREDENTIAL_, the credential’s name upper-cased with anything other than a letter or digit replaced by an underscore, then the field:

-e VP_CREDENTIAL_STAGING_DB_PASSWORD="..."

An environment variable wins over the file, so you can mount a shared file and override one value for a single job.

Note

These values overlay what is stored rather than replacing the credential, so the same tests work unchanged on a desktop where the secret is in the credential store. Unlike VP_RUN_VARS, secrets supplied this way never reach the run’s variables and so never appear in evidence.json or in run history.

Choosing what to report

By default a run writes three files into VP_OUTPUT_DIR:

File	What it is for
junit.xml	your CI system’s own test view
report.html	a readable summary to attach to the build
evidence.json	the full record of the run, for an audit trail

Narrow it with VP_FORMATS if you only want some of them, for example VP_FORMATS=junit.

report.html is a single self-contained file with no external references, so it renders correctly when your CI server serves build artifacts from an isolated network.

Calling a remote execution server

When the runner cannot reach the systems under test, deploy the execution server where it can and have the pipeline ask it to run the suite:

curl -sS -X POST \
  -H "Authorization: Bearer $VP_TOKEN" \
  "https://exec.internal.example.com:10101/data/suite/$SUITE_ID/run/junit" \
  -o results/junit.xml

Issue that pipeline its own token. In Settings, create a named token for each pipeline rather than sharing the one the desktop application uses. A named token can be given an expiry and can be withdrawn on its own, so retiring one pipeline’s access leaves every other client working.

Note

A named token still carries full access to that execution server; what it gives you is revocation and an audit trail, not restricted permissions. Treat it as you would any deployment credential.

The token value is shown once when you create it and cannot be read back afterwards. Store it in your CI system’s secret store at that moment.