notebook-ci-pipeline-author

Composes the three notebook testing tools into one GitHub Actions pipeline: papermill executes parameterized notebooks, nbval validates output regression, testbook runs function-level unit tests, and nbstripout gates committed output. Per-tool depth lives in references/papermill.md (parameterized execution) and references/nbval.md (output regression); this SKILL.md covers the wiring and integration decisions, plus the PR review checklist for notebook changes.

When to use

Teams using all three tools but assembling the pipeline by hand: no consistent artifact naming, no shared caching, duplicate install steps, no HTML report on failure. Also: any PR that adds or modifies .ipynb files and needs the review checklist below.

Hard-reject conditions

Do not proceed if any of the following apply:

• The repository has no parameters-tagged notebook cell. Papermill injection silently uses notebook defaults when the tag is absent per the Papermill execute docs (opens in new window); the pipeline will report green against stale values.
• nbval ground-truth outputs are absent from the committed .ipynb files. Re-running cells with no stored outputs produces no diff, so regressions are invisible.
• testbook tests use execute=True per-function without a scope="module" fixture. Each test re-executes the kernel; CI timeouts follow per the testbook docs (opens in new window).

State the blocker to the user and stop.

Step 1 - Install nbstripout as a pre-commit filter

Install once per clone so committed notebooks carry no output noise per the nbstripout README (opens in new window):

pip install nbstripout
nbstripout --install                    # writes .git/config filter entry
nbstripout --install --attributes .gitattributes  # repo-wide via .gitattributes

Add to .gitattributes:

*.ipynb filter=nbstripout

For pull-request verification without modifying files, use the kynan/nbstripout action (opens in new window):

- name: Verify notebooks are stripped
  uses: kynan/nbstripout@main
  with:
    paths: '**/*.ipynb'

The action runs a dry-run check and fails if any notebook carries uncommitted output.

Step 2 - Install dependencies with pip caching

Per GitHub Actions: Building and Testing Python (opens in new window), the setup-python action accepts cache: 'pip' and locates requirements.txt automatically:

- uses: actions/setup-python@v5
  with:
    python-version: '3.11'
    cache: 'pip'

- name: Install dependencies
  run: |
    python -m pip install --upgrade pip
    pip install -r requirements.txt
    pip install papermill nbval pytest testbook nbconvert

Keep papermill nbval pytest testbook nbconvert pinned in requirements.txt so the cache key (hashFiles('**/requirements.txt')) reflects version changes.

Step 3 - Stage 1: papermill parameterized execution

Papermill executes the notebook with injected parameters and writes a fully-rendered output notebook:

- name: Execute notebook (papermill)
  run: |
    papermill notebooks/analysis.ipynb \
      artifacts/analysis-executed.ipynb \
      -p seed 42 \
      -p n_samples 1000

Use -p for numeric/boolean parameters and -r for string parameters to prevent type-coercion surprises per the Papermill execute docs (opens in new window). Full papermill depth - parameter flags, Python API, matrix sweeps, regression-test wiring - is in references/papermill.md. Store the output path (artifacts/analysis-executed.ipynb) in an env var shared across stages:

env:
  EXECUTED_NB: artifacts/analysis-executed.ipynb

Step 4 - Stage 2: nbval output regression

Run nbval in lax mode on the executed notebook. Strict mode fails on every non-deterministic output; lax mode fails only on errors unless cells carry #NBVAL_CHECK_OUTPUT per the nbval docs (opens in new window):

- name: Output regression (nbval-lax)
  run: |
    pytest --nbval-lax $EXECUTED_NB \
      --sanitize-with sanitize.cfg \
      -v

sanitize.cfg example for timestamps and memory addresses:

[regex1]
regex: \d{1,2}/\d{1,2}/\d{2,4}
replace: DATE-STAMP

[regex2]
regex: 0x[0-9a-fA-F]+
replace: MEMORY-ADDR

Pin per-cell markers on cells that emit timestamps or large floats: # NBVAL_IGNORE_OUTPUT. Use # NBVAL_RAISES_EXCEPTION to validate expected error paths. Full nbval depth - strict vs lax mode, all per-cell controls, sanitize patterns, discovery - is in references/nbval.md.

Step 5 - Stage 3: testbook function unit tests

Run testbook tests against the source notebook (not the executed artifact) using a module-scoped fixture so the kernel executes once per pytest session per the testbook docs (opens in new window):

- name: Unit tests (testbook)
  run: pytest tests/test_notebook_functions.py -v

The scope="module" fixture is the load-bearing wiring decision - it stops each test re-executing the kernel:

@pytest.fixture(scope="module")
def tb():
    with testbook("notebooks/analysis.ipynb", execute=True) as tb:
        yield tb

The full tests/test_notebook_functions.py, with per-function tb.ref() assertions, is in references/notebook-ci-pipeline.md.

Step 6 - Stage 4: HTML report via nbconvert

Convert the executed notebook to a self-contained HTML report per the nbconvert docs (opens in new window):

- name: Convert to HTML
  if: always()
  run: |
    jupyter nbconvert --to html \
      --template lab \
      --embed-images \
      $EXECUTED_NB \
      --output artifacts/analysis-report.html

if: always() per GitHub Actions expressions (opens in new window) ensures the report generates even when nbval or testbook failed; the HTML is the primary debugging artifact.

Step 7 - Artifact upload with failure-aware retention

Upload both the executed notebook and the HTML report. Use if: always() so artifacts surface on failure per actions/upload-artifact@v4 (opens in new window):

- name: Upload artifacts
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: notebook-ci-${{ github.run_id }}
    path: |
      artifacts/analysis-executed.ipynb
      artifacts/analysis-report.html
    if-no-files-found: warn
    retention-days: 14

Set retention-days within the 1-90 day range allowed by actions/upload-artifact@v4 (opens in new window); 14 days covers sprint cycles without excessive storage.

Step 8 - Complete workflow

Steps 1-7 assemble into one workflow file. The full assembled YAML is in references/notebook-ci-pipeline.md; paste it to .github/workflows/notebook-ci.yml and adjust the notebook path, papermill parameters, and test path to match the repo.

Step 9 - Review checklist for notebook PRs

When a PR adds or modifies .ipynb files, walk each notebook through six checks and emit a finding table with a verdict. Every finding must trace to an observable file pattern or a cited source, never intuition.

1. Untested notebooks. Cross-reference conftest.py, pytest.ini, CI workflow YAML, and tests/. Each notebook needs at least one of: an --nbval / --nbval-lax CI step, a testbook test file referencing it, or a papermill execution step. No coverage = UNTESTED.
2. --nbval-lax misuse. --nbval-lax "collects notebooks and runs them, failing if there is an error" but skips output comparison unless cells carry #NBVAL_CHECK_OUTPUT (nbval docs (opens in new window)). Flag when the notebook is a regression test (its purpose is validating outputs) AND no cell carries the marker; fix is strict --nbval or adding markers to load-bearing cells.
3. Hardcoded credentials. Scan source cells for API-key assignments (api_key =, token =, password =, secret =), connection strings (://user:pass@), and long base64 blobs in string literals - and scan cell outputs too (secrets echoed in stream / execute_result blocks are equally dangerous and often missed). A match is a match; do not judge intent.
4. Non-deterministic output cells. Stored outputs containing timestamps, memory addresses, UUIDs, or floats with > 4 significant figures, with no #NBVAL_IGNORE_OUTPUT marker and no --sanitize-with config in CI, produce spurious diffs or false failures.
5. parameters tag. For papermill-executed notebooks, exactly one cell must carry the parameters tag; with no tag, papermill inserts injected-parameters at the top, potentially before imports (Papermill parameterize docs (opens in new window)). Flag zero tags (MISSING-PARAMS-TAG) and multiple tags (AMBIGUOUS-PARAMS-TAG).
6. Committed outputs. Any code cell with a non-empty outputs array, in a repo without an nbstripout hook, bloats diffs and causes merge conflicts per nbstripout README (opens in new window) - flag COMMITTED-OUTPUTS. With the hook configured, the flag means the PR bypassed it.

Severity: BLOCK = credentials, untested; WARN = lax misuse, committed outputs, non-deterministic outputs; INFO = params-tag findings. Verdict is BLOCK if any BLOCK-severity finding is present, PASS otherwise.

Anti-patterns

Limitations

• Papermill, nbval, and testbook each spawn a separate kernel session. For very large notebooks (> 5 min execution), total CI time triples. Use selective cell execution in testbook (pass a list of tags to execute) and split long notebooks into composable units.
• nbval and testbook conflict when run against the same notebook in the same pytest session per the testbook docs (opens in new window). This workflow avoids the conflict by pointing each tool at a different target (nbval on the executed artifact, testbook on the source).
• The kynan/nbstripout action checks output presence but does not enforce kernel metadata stripping; add --extra-keys flags locally if kernel version metadata causes diff noise.

References

• references/papermill.md - papermill in tool depth: parameters tag, CLI + Python API, sweeps, regression wiring
• references/nbval.md - nbval in tool depth: strict/lax, per-cell controls, sanitize config, discovery
• Papermill execute docs (opens in new window) - Python API, CLI flags, parameter types, papermill_description TQDM integration
• nbval docs (opens in new window) - strict/lax modes, per-cell markers, sanitize config
• testbook docs (opens in new window) - @testbook decorator, tb.ref(), tb.inject(), tb.patch(), module-scoped fixture
• nbstripout README (opens in new window) - install, git filter, --install --attributes, --verify flag
• kynan/nbstripout action (opens in new window) - GitHub Actions dry-run check
• nbconvert docs (opens in new window) - --to html, --template, --embed-images flags
• GitHub Actions: Building and Testing Python (opens in new window) - setup-python cache: 'pip' pattern
• actions/upload-artifact@v4 (opens in new window) - name, path, if-no-files-found, retention-days parameters
• GitHub Actions expressions (opens in new window) - always() and failure() status check functions in step if conditions