Composes the three notebook testing tools into one GitHub Actions pipeline: papermill executes parameterized notebooks, nbval validates output regression, testbook runs function-level unit tests, and nbstripout gates committed output. Per-tool depth lives in references/papermill.md (parameterized execution) and references/nbval.md (output regression); this SKILL.md covers the wiring and integration decisions, plus the PR review checklist for notebook changes.
When to use
Teams using all three tools but assembling the pipeline by hand: no consistent artifact naming, no shared caching, duplicate install steps, no HTML report on failure. Also: any PR that adds or modifies .ipynb files and needs the review checklist below.
Hard-reject conditions
Do not proceed if any of the following apply:
State the blocker to the user and stop.
Step 1 - Install nbstripout as a pre-commit filter
Install once per clone so committed notebooks carry no output noise per the nbstripout README (opens in new window):
pip install nbstripout
nbstripout --install # writes .git/config filter entry
nbstripout --install --attributes .gitattributes # repo-wide via .gitattributesAdd to .gitattributes:
*.ipynb filter=nbstripoutFor pull-request verification without modifying files, use the kynan/nbstripout action (opens in new window):
- name: Verify notebooks are stripped
uses: kynan/nbstripout@main
with:
paths: '**/*.ipynb'The action runs a dry-run check and fails if any notebook carries uncommitted output.
Step 2 - Install dependencies with pip caching
Per GitHub Actions: Building and Testing Python (opens in new window), the setup-python action accepts cache: 'pip' and locates requirements.txt automatically:
- uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install papermill nbval pytest testbook nbconvertKeep papermill nbval pytest testbook nbconvert pinned in requirements.txt so the cache key (hashFiles('**/requirements.txt')) reflects version changes.
Step 3 - Stage 1: papermill parameterized execution
Papermill executes the notebook with injected parameters and writes a fully-rendered output notebook:
- name: Execute notebook (papermill)
run: |
papermill notebooks/analysis.ipynb \
artifacts/analysis-executed.ipynb \
-p seed 42 \
-p n_samples 1000Use -p for numeric/boolean parameters and -r for string parameters to prevent type-coercion surprises per the Papermill execute docs (opens in new window). Full papermill depth - parameter flags, Python API, matrix sweeps, regression-test wiring - is in references/papermill.md. Store the output path (artifacts/analysis-executed.ipynb) in an env var shared across stages:
env:
EXECUTED_NB: artifacts/analysis-executed.ipynbStep 4 - Stage 2: nbval output regression
Run nbval in lax mode on the executed notebook. Strict mode fails on every non-deterministic output; lax mode fails only on errors unless cells carry #NBVAL_CHECK_OUTPUT per the nbval docs (opens in new window):
- name: Output regression (nbval-lax)
run: |
pytest --nbval-lax $EXECUTED_NB \
--sanitize-with sanitize.cfg \
-vsanitize.cfg example for timestamps and memory addresses:
[regex1]
regex: \d{1,2}/\d{1,2}/\d{2,4}
replace: DATE-STAMP
[regex2]
regex: 0x[0-9a-fA-F]+
replace: MEMORY-ADDRPin per-cell markers on cells that emit timestamps or large floats: # NBVAL_IGNORE_OUTPUT. Use # NBVAL_RAISES_EXCEPTION to validate expected error paths. Full nbval depth - strict vs lax mode, all per-cell controls, sanitize patterns, discovery - is in references/nbval.md.
Step 5 - Stage 3: testbook function unit tests
Run testbook tests against the source notebook (not the executed artifact) using a module-scoped fixture so the kernel executes once per pytest session per the testbook docs (opens in new window):
- name: Unit tests (testbook)
run: pytest tests/test_notebook_functions.py -vThe scope="module" fixture is the load-bearing wiring decision - it stops each test re-executing the kernel:
@pytest.fixture(scope="module")
def tb():
with testbook("notebooks/analysis.ipynb", execute=True) as tb:
yield tbThe full tests/test_notebook_functions.py, with per-function tb.ref() assertions, is in references/notebook-ci-pipeline.md.
Step 6 - Stage 4: HTML report via nbconvert
Convert the executed notebook to a self-contained HTML report per the nbconvert docs (opens in new window):
- name: Convert to HTML
if: always()
run: |
jupyter nbconvert --to html \
--template lab \
--embed-images \
$EXECUTED_NB \
--output artifacts/analysis-report.htmlif: always() per GitHub Actions expressions (opens in new window) ensures the report generates even when nbval or testbook failed; the HTML is the primary debugging artifact.
Step 7 - Artifact upload with failure-aware retention
Upload both the executed notebook and the HTML report. Use if: always() so artifacts surface on failure per actions/upload-artifact@v4 (opens in new window):
- name: Upload artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: notebook-ci-${{ github.run_id }}
path: |
artifacts/analysis-executed.ipynb
artifacts/analysis-report.html
if-no-files-found: warn
retention-days: 14Set retention-days within the 1-90 day range allowed by actions/upload-artifact@v4 (opens in new window); 14 days covers sprint cycles without excessive storage.
Step 8 - Complete workflow
Steps 1-7 assemble into one workflow file. The full assembled YAML is in references/notebook-ci-pipeline.md; paste it to .github/workflows/notebook-ci.yml and adjust the notebook path, papermill parameters, and test path to match the repo.
Step 9 - Review checklist for notebook PRs
When a PR adds or modifies .ipynb files, walk each notebook through six checks and emit a finding table with a verdict. Every finding must trace to an observable file pattern or a cited source, never intuition.
Severity: BLOCK = credentials, untested; WARN = lax misuse, committed outputs, non-deterministic outputs; INFO = params-tag findings. Verdict is BLOCK if any BLOCK-severity finding is present, PASS otherwise.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Run nbval on the source notebook before papermill | nbval re-executes from scratch; parameter injection never happens | Run nbval on the papermill output notebook (Stage 2) |
| Run testbook tests against the executed artifact | testbook needs the source notebook to resolve cell tags; .ipynb with injected-parameters cell confuses selective execution | Point testbook at the source notebook, not the artifact |
Omit nbstripout --install from onboarding | Developers commit outputs; nbval diffs against stale ground truth in CI | Document nbstripout --install in CONTRIBUTING.md; enforce via the kynan/nbstripout action (Step 1) |
| Upload artifacts only on success | Failures produce no HTML; engineers cannot inspect which cell errored | Use if: always() on the convert and upload steps (Steps 6-7) |
| Module-scope fixture missing from testbook tests | Each test re-executes the full notebook kernel; multi-minute CI runs per test | Add @pytest.fixture(scope="module") (Step 5) |
Limitations
References
The output-regression stage of the pipeline (SKILL.md Step 4) in tool depth: strict vs lax mode, per-cell controls, sanitize config, and discovery.
nbval is a pytest plugin that validates Jupyter notebooks by re-executing cells and comparing outputs against stored results, "ensuring that the notebook is behaving as expected and that changes to underlying source code haven't affected the results" per the nbval docs (opens in new window).
When to use
Step 1 - Install
pip install nbval pytestPer the nbval docs (opens in new window).
Step 2 - Strict mode (default)
pytest --nbval my_notebook.ipynbRe-executes every cell; fails if any output differs from stored.
Step 3 - Lax mode (failure-only)
pytest --nbval-lax my_notebook.ipynb"Collects notebooks and runs them, failing if there is an error" - skips output comparison unless cells bear the #NBVAL_CHECK_OUTPUT marker per the nbval docs (opens in new window). Use as the default for tutorials where output is incidental and execution is what matters.
Step 4 - Per-cell controls
Add comments at cell start:
| Marker | Effect |
|---|---|
# NBVAL_SKIP | Cell not executed during testing |
# NBVAL_IGNORE_OUTPUT | Cell runs; output diff ignored |
# NBVAL_CHECK_OUTPUT | Force output checking (lax mode) |
# NBVAL_RAISES_EXCEPTION | Validate that the cell raises |
Cell tags (in notebook metadata, lowercase-with-dashes: nbval-skip, nbval-ignore-output, etc.) are equivalent and recommended for non-Python kernels.
Step 5 - Sanitize dynamic outputs
For timestamps, UUIDs, memory addresses:
pytest --nbval my_notebook.ipynb --sanitize-with sanitize.cfgsanitize.cfg:
[regex1]
regex: \d{1,2}/\d{1,2}/\d{2,4}
replace: DATE-STAMP
[regex2]
regex: 0x[0-9a-fA-F]+
replace: MEMORY-ADDR
[regex3]
regex: \d+\.\d+(?:e-?\d+)?
replace: NUMBERTune carefully - over-sanitizing makes nbval miss real regressions.
Step 6 - Test discovery
# Whole notebooks/ directory
pytest --nbval notebooks/
# Filter by name
pytest --nbval notebooks/ -k "tutorial"
# Single notebook + verbose
pytest --nbval --verbose notebooks/intro.ipynbStep 7 - CI integration
# GitHub Actions
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install
run: |
pip install -r requirements.txt
pip install nbval pytest
- name: Run notebook tests (lax)
run: pytest --nbval-lax notebooks/ --sanitize-with sanitize.cfgFor tutorial repos, lax mode + sanitize is usually right.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Use strict mode for tutorial notebooks | Every random seed change fails CI | Use --nbval-lax (Step 3) |
Skip cells liberally with # NBVAL_SKIP | Coverage shrinks; notebook becomes untested | Use # NBVAL_IGNORE_OUTPUT instead - still verifies execution |
| Sanitize all numeric output | Real regressions hidden | Targeted regexes (Step 5) |
| Run nbval against notebooks that mutate disk/state | Tests become flaky | Use ephemeral working dirs; monkeypatch.chdir(tmp_path) |
No requirements.txt pinning | "Works on author's machine"; CI fails on minor lib bumps | Pin notebook deps separately from prod deps |
Limitations
References
Deep detail for notebook-ci-pipeline-author: the full assembled GitHub Actions workflow and the complete testbook test file. The SKILL.md spine holds the per-stage snippets and integration decisions; this bundle holds the two longest blocks so the spine stays focused.
Complete workflow
Steps 1-7 of the skill assemble into one workflow file. Paste this to .github/workflows/notebook-ci.yml and adjust the notebook path, papermill parameters, and test path to match the repo. Order matters: nbstripout verify -> pip cache -> papermill -> nbval-lax -> testbook -> nbconvert HTML -> artifact upload.
name: Notebook CI
on:
push:
paths:
- 'notebooks/**'
- 'tests/**'
- 'requirements.txt'
pull_request:
paths:
- 'notebooks/**'
jobs:
notebook-ci:
runs-on: ubuntu-latest
env:
EXECUTED_NB: artifacts/analysis-executed.ipynb
steps:
- uses: actions/checkout@v4
- name: Verify notebooks are stripped
uses: kynan/nbstripout@main
with:
paths: '**/*.ipynb'
- uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Execute notebook (papermill)
run: |
mkdir -p artifacts
papermill notebooks/analysis.ipynb \
$EXECUTED_NB \
-p seed 42 \
-p n_samples 1000
- name: Output regression (nbval-lax)
run: |
pytest --nbval-lax $EXECUTED_NB \
--sanitize-with sanitize.cfg \
-v
- name: Unit tests (testbook)
run: pytest tests/test_notebook_functions.py -v
- name: Convert to HTML
if: always()
run: |
jupyter nbconvert --to html \
--template lab \
--embed-images \
$EXECUTED_NB \
--output artifacts/analysis-report.html
- name: Upload artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: notebook-ci-${{ github.run_id }}
path: |
artifacts/analysis-executed.ipynb
artifacts/analysis-report.html
if-no-files-found: warn
retention-days: 14Full testbook test file
tests/test_notebook_functions.py - the module-scoped fixture executes the kernel once per pytest session; each test resolves a notebook function with tb.ref() and asserts on its return value:
import pytest
from testbook import testbook
@pytest.fixture(scope="module")
def tb():
with testbook("notebooks/analysis.ipynb", execute=True) as tb:
yield tb
def test_clean_data_drops_nulls(tb):
clean_data = tb.ref("clean_data")
result = clean_data(tb.ref("pd").DataFrame({"a": [1, None, 3]}))
assert len(result) == 2
def test_model_output_shape(tb):
predict = tb.ref("predict")
assert predict(tb.ref("test_input")).shape == (1,)The execution stage of the pipeline (SKILL.md Step 3) in tool depth: the parameters cell tag, CLI + Python API, parameter flags, sweeps, and regression-test wiring.
Papermill executes notebooks programmatically with injected parameters, producing an output notebook with results. Per the Papermill execute docs (opens in new window), it pairs naturally with regression testing: run a parameterized notebook in CI, assert on outputs.
When to use
How to use
Step 1 - Install
pip install papermillPer the Papermill execute docs (opens in new window).
Step 2 - Tag the parameters cell
In your notebook, tag one cell with parameters:
# Cell tagged "parameters"
alpha = 0.5
ratio = 0.2
input_path = "data/sales.parquet"Papermill replaces these with injected values at execution time (adds an injected-parameters cell after the tagged cell).
Step 3 - Python API execution
import papermill as pm
pm.execute_notebook(
'path/to/input.ipynb',
'path/to/output.ipynb',
parameters=dict(alpha=0.6, ratio=0.1)
)Per the Papermill execute docs (opens in new window).
Step 4 - CLI execution
# Local in/out
papermill local/input.ipynb local/output.ipynb -p alpha 0.6 -p ratio 0.1
# S3 output
papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1Parameter flags per the Papermill execute docs (opens in new window):
| Flag | Meaning |
|---|---|
-p NAME VAL | Simple parameter (auto-typed) |
-r NAME VAL | Raw string (preserve as string) |
-f file.yaml | Parameters from YAML file |
-y "key: val" | Inline YAML (supports lists, dicts) |
-b base64yaml | Base64-encoded YAML |
Step 5 - Use as regression test
import json
import papermill as pm
import nbformat
def test_analysis_with_known_inputs(tmp_path):
out_path = tmp_path / "out.ipynb"
pm.execute_notebook(
'analysis.ipynb',
str(out_path),
parameters=dict(seed=42, n_samples=1000),
)
nb = nbformat.read(str(out_path), as_version=4)
final_cell = nb.cells[-1]
output_text = final_cell.outputs[0]['text']
result = json.loads(output_text)
assert abs(result['mean'] - 0.5) < 0.01
assert result['n'] == 1000The output notebook is artifact-friendly - attach to CI runs for review when assertions fail.
Step 6 - Parameter sweeps in CI
# GitHub Actions matrix sweep
strategy:
matrix:
seed: [42, 123, 7]
n_samples: [100, 1000]
steps:
- run: |
papermill analysis.ipynb out-${{ matrix.seed }}-${{ matrix.n_samples }}.ipynb \
-p seed ${{ matrix.seed }} \
-p n_samples ${{ matrix.n_samples }}
- uses: actions/upload-artifact@v4
with:
name: papermill-output-${{ matrix.seed }}-${{ matrix.n_samples }}
path: out-${{ matrix.seed }}-${{ matrix.n_samples }}.ipynbStep 7 - Pair with nbval / testbook
| Tool | Strength | Pair with papermill how |
|---|---|---|
| nbval | Full-notebook output regression | Run papermill first (parameter inject) → run nbval on output |
| testbook | Function-level unit tests | testbook can use papermill's executor under the hood - see testbook configuration for execute_kwargs |
Papermill is the engine; nbval and testbook are the assertion layers. Use all three for production notebook QA.
Step 8 - TQDM progress descriptions
Add comments at cell start:
#papermill_description=load_data
df = load_dataset()
#papermill_description=train_model
model.fit(df)Per the Papermill execute docs (opens in new window): integrates with TQDM for meaningful CI progress indicators.
Worked example
An analyst ships analysis.ipynb that samples a distribution and prints a JSON summary in its last cell. To gate it in CI:
On a green run the assertions pass and out.ipynb uploads as a CI artifact. When a refactor shifts the sampled mean, the assertion fails, the job goes red, and the attached output notebook shows the offending cell for review.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
Forget the parameters cell tag | Parameters never inject; notebook runs with defaults | Tag the cell explicitly (Step 2) |
Mix -p and -r types incorrectly | -p version 1.0 becomes float 1.0; loses leading zeros etc. | Use -r for strings (Step 4) |
| Run papermill against side-effect notebooks (writes to prod DB) | Papermill is non-transactional; partial failures leave bad state | Use ephemeral workdirs / staging credentials in test runs |
| Ignore the output notebook (only check exit code) | Subtle errors visible only in cell outputs | Save + inspect output notebook (Step 5); upload as artifact (Step 6) |
| Skip seed parameterization | Tests flake on stochastic models | Always -p seed N for reproducible runs |