CI Integration

Integrate Starlark code coverage into your CI/CD pipeline. This guide covers the major CI platforms with copy-paste configurations.

Quick Setup

Configure coverage output

Use Cobertura XML for best CI compatibility:
sky.toml
```
[test.coverage]
enabled = true
output = "coverage.xml"
fail_under = 80.0
```
Add CI configuration (see platform-specific sections below)
Upload to coverage service (optional: Codecov, Coveralls)

GitHub Actions

Complete Example (Tests + Coverage + PR Annotations)

This workflow runs tests, reports results with PR annotations, and uploads coverage:

name: Test

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: '1.22'

      - name: Install Sky
        run: go install github.com/example/sky/cmd/skytest@latest

      - name: Run tests
        run: |
          skytest -junit -r tests/ > test-results.xml
          skytest --coverage --coverage-output=coverage.xml -r tests/

      - name: Test Report
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: Starlark Tests
          path: test-results.xml
          reporter: java-junit
          fail-on-error: false

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          files: coverage.xml
          token: ${{ secrets.CODECOV_TOKEN }}

Test Reporting Actions

Several GitHub Actions can display JUnit XML test results:

dorny/test-reporter

Shows test results in PR checks with file annotations. Marketplace

mikepenz/action-junit-report

Adds test results as PR check with annotations. Marketplace

Using dorny/test-reporter

      - name: Test Report
        uses: dorny/test-reporter@v1
        if: always()  # Run even if tests fail
        with:
          name: Starlark Tests
          path: test-results.xml
          reporter: java-junit
          fail-on-error: false

Using mikepenz/action-junit-report

      - name: Test Report
        uses: mikepenz/action-junit-report@v4
        if: always()
        with:
          report_paths: test-results.xml
          check_name: Starlark Tests
          fail_on_failure: false

Job Summary (Native GitHub Feature)

Write test results directly to the GitHub Actions job summary:

      - name: Run tests with summary
        run: |
          skytest -r tests/ | tee test-output.txt

          # Write summary
          echo "## Test Results" >> $GITHUB_STEP_SUMMARY
          echo '```' >> $GITHUB_STEP_SUMMARY
          tail -5 test-output.txt >> $GITHUB_STEP_SUMMARY
          echo '```' >> $GITHUB_STEP_SUMMARY

Or generate a detailed summary from JSON output:

      - name: Generate test summary
        if: always()
        run: |
          skytest -json -r tests/ > results.json || true

          echo "## 🧪 Test Results" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY

          PASSED=$(jq '.passed' results.json)
          FAILED=$(jq '.failed' results.json)

          if [ "$FAILED" -eq 0 ]; then
            echo "✅ All $PASSED tests passed" >> $GITHUB_STEP_SUMMARY
          else
            echo "❌ $FAILED failed, $PASSED passed" >> $GITHUB_STEP_SUMMARY
          fi

Basic Coverage Only

name: Test

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: '1.22'

      - name: Install Sky
        run: go install github.com/example/sky/cmd/skytest@latest

      - name: Run tests with coverage
        run: skytest --coverage --coverage-output=coverage.xml -r tests/

      - name: Upload coverage artifact
        uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: coverage.xml

With Codecov

      - name: Upload to Codecov
        uses: codecov/codecov-action@v4
        with:
          files: coverage.xml
          token: ${{ secrets.CODECOV_TOKEN }}
          fail_ci_if_error: true

Coverage Threshold Check

      - name: Check coverage threshold
        run: |
          # skytest will exit non-zero if coverage < threshold
          skytest --coverage --coverage-output=coverage.xml --coverage-fail-under=80 tests/

PR Coverage Comment

      - name: Generate coverage report
        uses: irongut/CodeCoverageSummary@v1.3.0
        with:
          filename: coverage.xml
          badge: true
          format: markdown
          output: both

      - name: Add coverage PR comment
        uses: marocchino/sticky-pull-request-comment@v2
        if: github.event_name == 'pull_request'
        with:
          recreate: true
          path: code-coverage-results.md

GitLab CI

Basic Coverage

stages:
  - test

test:
  stage: test
  image: golang:1.22
  before_script:
    - go install github.com/example/sky/cmd/skytest@latest
  script:
    - skytest --coverage --coverage-output=coverage.xml tests/
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
  coverage: '/Coverage: (\d+\.?\d*)%/'

Coverage Visualization in MRs

GitLab automatically shows coverage diff in merge requests when using Cobertura:

test:
  stage: test
  image: golang:1.22
  script:
    - go install github.com/example/sky/cmd/skytest@latest
    - skytest --coverage --coverage-output=coverage.xml tests/
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

The MR diff view will show:

Green highlighting for covered new lines
Red highlighting for uncovered new lines
Coverage percentage change

With Coverage Threshold

test:
  stage: test
  script:
    - skytest --coverage --coverage-output=coverage.xml --coverage-fail-under=80 tests/
  allow_failure: false

Jenkins

Declarative Pipeline

pipeline {
    agent any

    tools {
        go 'go-1.22'
    }

    stages {
        stage('Install') {
            steps {
                sh 'go install github.com/example/sky/cmd/skytest@latest'
            }
        }

        stage('Test') {
            steps {
                sh 'skytest --coverage --coverage-output=coverage.xml tests/'
            }
            post {
                always {
                    cobertura coberturaReportFile: 'coverage.xml',
                              onlyStable: false,
                              failNoReports: true,
                              sourceEncoding: 'UTF-8'
                }
            }
        }
    }
}

Scripted Pipeline

node {
    stage('Checkout') {
        checkout scm
    }

    stage('Install') {
        sh 'go install github.com/example/sky/cmd/skytest@latest'
    }

    stage('Test') {
        sh 'skytest --coverage --coverage-output=coverage.xml tests/'
    }

    stage('Publish Coverage') {
        cobertura coberturaReportFile: 'coverage.xml',
                  onlyStable: false,
                  failNoReports: true,
                  sourceEncoding: 'UTF-8',
                  lineCoverageTargets: '80, 60, 0',
                  conditionalCoverageTargets: '80, 60, 0'
    }
}

Azure DevOps

trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

steps:
  - task: GoTool@0
    inputs:
      version: '1.22'
    displayName: 'Set up Go'

  - script: go install github.com/example/sky/cmd/skytest@latest
    displayName: 'Install Sky'

  - script: skytest --coverage --coverage-output=coverage.xml tests/
    displayName: 'Run tests with coverage'

  - task: PublishCodeCoverageResults@2
    inputs:
      summaryFileLocation: '$(System.DefaultWorkingDirectory)/coverage.xml'
      codecoverageTool: 'Cobertura'
    displayName: 'Publish coverage'

CircleCI

version: 2.1

orbs:
  go: circleci/go@1.7
  codecov: codecov/codecov@3

jobs:
  test:
    executor:
      name: go/default
      tag: '1.22'
    steps:
      - checkout
      - run:
          name: Install Sky
          command: go install github.com/example/sky/cmd/skytest@latest
      - run:
          name: Run tests
          command: skytest --coverage --coverage-output=coverage.xml tests/
      - store_artifacts:
          path: coverage.xml
      - codecov/upload:
          file: coverage.xml

workflows:
  test:
    jobs:
      - test

Coverage Services

Codecov

Upload coverage to Codecov for tracking and PR comments.

Configuration

coverage:
  status:
    project:
      default:
        target: 80%
        threshold: 1%
    patch:
      default:
        target: 80%

comment:
  layout: "reach,diff,flags,files"
  behavior: default

Upload Methods

- uses: codecov/codecov-action@v4
  with:
    files: coverage.xml
    token: ${{ secrets.CODECOV_TOKEN }}

- curl -Os https://cli.codecov.io/latest/linux/codecov
- chmod +x codecov
- ./codecov upload-process -f coverage.xml -t $CODECOV_TOKEN

# Download uploader
curl -Os https://cli.codecov.io/latest/linux/codecov
chmod +x codecov

# Upload
./codecov upload-process -f coverage.xml -t $CODECOV_TOKEN

Coveralls

Upload coverage to Coveralls.

GitHub Actions
Generic

- name: Upload to Coveralls
  uses: coverallsapp/github-action@v2
  with:
    file: coverage.xml
    format: cobertura

# Install goveralls
go install github.com/mattn/goveralls@latest

# Convert and upload (requires LCOV format)
skytest --coverage --coverage-output=coverage.lcov tests/
goveralls -coverprofile=coverage.lcov -service=local

Coverage Badges

Codecov Badge

[![codecov](https://codecov.io/gh/USER/REPO/branch/main/graph/badge.svg)](https://codecov.io/gh/USER/REPO)

Dynamic Badge from CI

Generate a badge based on coverage percentage:

      - name: Extract coverage
        id: coverage
        run: |
          # Extract percentage from JSON output
          COVERAGE=$(skytest --coverage --coverage-output=/dev/stdout --coverage-format=json tests/ | jq -r '.percentage')
          echo "percentage=$COVERAGE" >> $GITHUB_OUTPUT

      - name: Update badge
        uses: schneegans/dynamic-badges-action@v1.6.0
        with:
          auth: ${{ secrets.GIST_TOKEN }}
          gistID: your-gist-id
          filename: coverage.json
          label: coverage
          message: ${{ steps.coverage.outputs.percentage }}%
          valColorRange: ${{ steps.coverage.outputs.percentage }}
          maxColorRange: 100
          minColorRange: 0

Best Practices

Set Thresholds

Use fail_under to prevent coverage regressions. Start at current coverage and increase gradually.

Track Trends

Use Codecov or Coveralls to track coverage over time and catch regressions early.

Require on PRs

Block PR merging if coverage drops significantly.

Cache Installation

Cache the Sky binary to speed up CI runs.

- name: Cache Sky
  uses: actions/cache@v4
  with:
    path: ~/go/bin/skytest
    key: ${{ runner.os }}-skytest-${{ hashFiles('go.sum') }}

- name: Install Sky (if not cached)
  run: |
    if [ ! -f ~/go/bin/skytest ]; then
      go install github.com/example/sky/cmd/skytest@latest
    fi

cache:
  key: skytest
  paths:
    - /go/bin/skytest

before_script:
  - |
    if [ ! -f /go/bin/skytest ]; then
      go install github.com/example/sky/cmd/skytest@latest
    fi

Troubleshooting

Coverage Not Appearing in GitLab MR

Ensure the artifact path matches exactly: coverage.xml
Verify the file is valid Cobertura XML
Check that the artifact is uploaded in the same job that runs tests

Codecov Upload Failing

Verify CODECOV_TOKEN is set in secrets
Check the file path matches what you’re uploading
Use fail_ci_if_error: false temporarily to debug

Coverage Percentage Parsing

GitLab extracts coverage from stdout using regex. Ensure your output matches:

coverage: '/Coverage: (\d+\.?\d*)%/'

The skytest output format:

Coverage: 85.0% (17/20 lines)