Skip to content
Sky

Getting Started with Coverage

Starlark Code Coverage

Section titled “Starlark Code Coverage”

Measure which lines of your Starlark code execute during tests. Track coverage over time and enforce minimum thresholds in CI.

Quick Start

Section titled “Quick Start”

  1. Enable coverage in your config

    Create or update sky.toml:

    [test.coverage]
    enabled = true
    output = "coverage.lcov"

  2. Run your tests

    Terminal window
    skytest tests/

    Output:

    PASS  test_add (0.002s)
    PASS  test_subtract (0.001s)
    

    Results: 2 passed, 0 failed
    Coverage: 85.0% (17/20 lines)
    Coverage written to coverage.lcov

  3. View the report

    Open in your IDE or generate HTML:

    Terminal window
    genhtml coverage.lcov -o coverage-html/
    open coverage-html/index.html

How Coverage Works

Section titled “How Coverage Works”

Coverage collection uses the OnExec hook in starlark-go-x to track each line executed during test runs.

┌─────────────────────────┐
│   Your Starlark files   │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│   skytest --coverage    │ ← OnExec hook records each line
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│   Coverage Report       │ ← LCOV, Cobertura, JSON, HTML, Text
└─────────────────────────┘

Example Workflow

Section titled “Example Workflow”

Create a module and test:

math.star
def add(a, b):
    return a + b


def divide(a, b):
    if b == 0:
        return None  # This line may not be covered
    return a / b
math_test.star
load("math.star", "add", "divide")


def test_add():
    assert.eq(add(2, 3), 5)
    assert.eq(add(-1, 1), 0)


def test_divide():
    assert.eq(divide(10, 2), 5)
    # Missing: test for divide by zero

Run with coverage:

Terminal window
$ skytest --coverage math_test.star
PASS  test_add (0.001s)
PASS  test_divide (0.001s)


Results: 2 passed, 0 failed
Coverage: 83.3% (5/6 lines)

The report shows line 6 (return None) was never executed.

Coverage Metrics

Section titled “Coverage Metrics”

Line Coverage

Tracks how many times each line executed. A line with 0 hits is uncovered.

Function Coverage

Tracks which functions were called. Inferred from line hits at function entry points.

MetricStatusDescription
Line coverage✅ SupportedWhich lines executed and how many times
Function coverage✅ SupportedWhich functions were called
Branch coverage🔜 PlannedWhich conditional branches were taken

Configuration

Section titled “Configuration”

TOML Configuration

Section titled “TOML Configuration”
sky.toml
[test.coverage]
# Enable coverage collection
enabled = true


# Output file path (extension determines format)
output = "coverage.lcov"


# Fail if coverage falls below threshold (0 = disabled)
fail_under = 80.0

Starlark Configuration

Section titled “Starlark Configuration”

For dynamic configuration based on environment:

config.sky
def configure():
    ci = getenv("CI", "") != ""


    return {
        "test": {
            "coverage": {
                "enabled": ci,  # Only collect in CI
                "output": "coverage.xml" if ci else "coverage.html",
                "fail_under": 80 if ci else 0,
            },
        },
    }

CLI Flags

Section titled “CLI Flags”

Override config from the command line:

Terminal window
# Enable coverage
skytest --coverage tests/


# Specify output file
skytest --coverage --coverage-output=coverage.xml tests/


# Set minimum threshold
skytest --coverage --coverage-fail-under=80 tests/

Output Format Selection

Section titled “Output Format Selection”

The output format is determined by the file extension:

ExtensionFormatBest For
.txtTextTerminal viewing
.jsonJSONCustom tooling
.xmlCoberturaCI systems (Jenkins, GitLab)
.lcovLCOVIDE extensions, genhtml
.htmlHTMLBrowser viewing, sharing
sky.toml
[test.coverage]
enabled = true
# Change extension to change format:
output = "coverage.html"  # HTML report
# output = "coverage.xml"   # Cobertura XML
# output = "coverage.lcov"  # LCOV tracefile

Enforcing Coverage Thresholds

Section titled “Enforcing Coverage Thresholds”

Prevent coverage regressions by setting a minimum threshold:

sky.toml
[test.coverage]
enabled = true
output = "coverage.lcov"
fail_under = 80.0

When coverage drops below the threshold:

Terminal window
$ skytest tests/
PASS  test_add (0.001s)


Results: 1 passed, 0 failed
Coverage: 75.0% (15/20 lines)
FAIL: Coverage 75.0% is below threshold 80.0%

Next Steps

Section titled “Next Steps”