How to Run Automated Tests in GitHub Actions
Run automated tests in GitHub Actions: triggers, caching, matrix builds, sharding, and failure artifacts, with copy-ready pytest, Jest, and Playwright jobs.
"Works on my machine" is a fine thing to say. Merging without green CI is not. Teams that skip wiring tests into their pipeline discover failures in staging or production. Martin Fowler put it directly: "The key to fixing problems quickly is finding them quickly." GitHub's 2025 Octoverse report counted 11.5 billion GitHub Actions minutes running tests last year, a 35% increase year-over-year. The 2024 Stack Overflow Developer Survey found 68.6% of professional developers have CI/CD available but only 56.3% report automated testing.
This guide closes that gap with a test workflow that gates every pull request, built one step at a time: triggers, dependency caching, a matrix, sharding, and failure artifacts. The workflow structure doesn't depend on the test framework, so each step shows the pattern once and the copy-ready jobs near the end give a complete file for pytest, Jest, and Playwright. Playwright-specific tuning (Docker images, worker counts, screenshot and video capture) lives in Playwright in GitHub Actions: Docker, Failure Capture, and Worker Parallelism.
GitHub Actions facts worth knowing before setup
A few numbers to have in mind before you touch YAML (as of October 2026):
Prerequisites
You need a repo whose tests run from a single command (pytest, npm test, or npx playwright test) and a committed lockfile: package-lock.json for Node.js projects, a pinned requirements.txt for Python. Caching keys off that file, so a missing lockfile means a cold install on every run. No prior GitHub Actions experience is required. If you don't have a suite yet, the Playwright TypeScript setup guide and the pytest fixtures guide are good starting points.
Step 1: the minimal workflow file
Create .github/workflows/tests.yml. Everything else builds on this skeleton:
# .github/workflows/tests.yml
name: Tests
on:
workflow_dispatch: # manual trigger - useful for testing the workflow itself
jobs:
test:
runs-on: ubuntu-latest # Ubuntu 24.04
steps:
- name: Checkout code
uses: actions/checkout@v7It does nothing yet, but it's a valid workflow that you can trigger manually from the Actions tab. Build from here rather than pasting a 60-line workflow and debugging everything at once.
Step 2: triggers that match how teams work
The on: block controls when the workflow runs. Most teams want four triggers:
on:
push:
branches: [main] # run on every push to main
pull_request: # gate every PR - this is the one that stops broken merges
workflow_dispatch: # manual run from the Actions tab
schedule:
- cron: '0 2 * * *' # nightly at 02:00 UTC - catches dependency drift
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true # cancel the old run when a new commit is pushed to the same PRThe concurrency block is worth adding early. The workflow syntax docs define it as: "Use concurrency to ensure that only a single job or workflow using the same concurrency group will run at a time." Without it, pushing two commits quickly to a PR queues two full runs. The cancel-in-progress: true flag drops the older one the moment the newer run starts.
The pull_request trigger is what actually gates merges. Without it, a red test suite doesn't block anything: developers just click through the warning. To make the gate binding, mark the test job as a required status check in the branch protection rules.
Step 3: language setup and dependency caching
Cold installs on every run add minutes. Both setup actions have caching built in: point them at your package manager and they key the cache on the lockfile. These fragments go under the steps: key of your job.
For a Python project:
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: '3.13'
cache: 'pip' # keyed on requirements files
- name: Install dependencies
run: pip install -r requirements.txtFor a Node.js project (Jest, Playwright, or anything else run through npm):
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
cache: 'npm' # caches ~/.npm, keyed on package-lock.json
- name: Install dependencies
run: npm ciThe caching docs note two limits worth knowing: "By default, the limit is 10 GB per repository" and "GitHub will remove any cache entries that have not been accessed in over 7 days." Caches stay warm on active repos; on repos with infrequent pushes, expect a cold install every couple of weeks.
Browser binaries are the one dependency not worth caching by hand. Playwright's own CI guide says "Caching browser binaries is not recommended," because restoring the cache takes about as long as downloading the browsers, and the Linux system libraries they need can't be cached anyway (Playwright docs: CI). Install them with npx playwright install --with-deps on every run.
Step 4: running tests and keeping the evidence
Run the suite with the same command you use locally, and have it write a machine-readable report next to the console output. JUnit XML is the format almost every CI dashboard and test-reporting tool reads:
- name: Run tests
run: pytest --junit-xml=reports/junit.xml # or: npx jest --ci / npx playwright test
- name: Upload test results
if: failure() # only upload when tests fail - saves artifact storage
uses: actions/upload-artifact@v7
with:
name: test-results
path: reports/
retention-days: 30 # default is 90 days; 30 is plenty for a CI artifactThe if: failure() condition is the important decision here. Uploading artifacts on every run costs storage and creates noise. Upload on failure only, and set a sensible retention-days. The GitHub docs note: "By default, the artifacts and log files generated by workflows are retained for 90 days", and private repos can extend the period up to 400 days. Thirty days is enough for debugging a CI failure.
The report flag differs by framework. pytest writes JUnit XML with --junit-xml=path. Jest needs a reporter package such as jest-junit, added with --reporters="default" --reporters="jest-junit" (Jest CLI docs), and its --ci flag makes a missing snapshot fail the test instead of being written silently. Playwright's own trace files in test-results/ are the richer artifact: download the zip and run npx playwright show-trace trace.zip to replay every step the runner took.
Step 5: a matrix across versions, systems, or browsers
A single job on one runtime is a reasonable start; it misses version-specific and browser-specific bugs. The strategy.matrix block multiplies one job definition across configurations:
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.12', '3.13'] # or node-version, or browser
fail-fast: false # don't cancel 3.12 if 3.13 fails - you want both results
max-parallel: 2 # set lower on private repos to control minute consumption
steps:
# ... checkout and install steps from above ...
- uses: actions/setup-python@v7
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'For a Playwright suite, the same block takes a browser: [chromium, firefox, webkit] list and the test command becomes npx playwright test --project=${{ matrix.browser }}. The values must match the project names in playwright.config.ts exactly.
Set fail-fast: false for test matrices: if one configuration fails, you still want to see whether the others passed. With fail-fast: true (the default), a single failure cancels all in-progress jobs and you lose the other configurations' signal.
For billing awareness on private repos: three parallel jobs each taking 5min is 15 minutes per PR run. At 100 PRs/month that's 1,500 of the 2,000 free minutes. The matrix docs note: "A matrix strategy lets you use variables in a single job definition to automatically create multiple job runs that are based on the combinations of the variables." The limits page adds: "A job matrix can generate a maximum of 256 jobs per workflow run."
Step 6: parallelism and sharding for faster feedback
A matrix runs the whole suite once per configuration. Parallelism and sharding split one suite so it finishes sooner, and each framework does it differently.
pytest parallelizes inside a single job with pytest-xdist: pytest -n auto starts one worker per CPU. On a 4-CPU public runner, that's often enough before you need more machines.
Jest and Playwright both split a suite across jobs with a --shard flag in index/count form. Jest's CLI docs describe it as "The test suite shard to execute," and a matrix supplies the index:
strategy:
matrix:
shard: [1, 2, 3, 4]
steps:
# ... setup steps ...
- name: Run tests (shard ${{ matrix.shard }} of 4)
run: npx jest --ci --shard=${{ matrix.shard }}/4
# Playwright: npx playwright test --shard=${{ matrix.shard }}/4 --reporter=blobPlaywright shards write blob reports that a final job merges into one HTML report with npx playwright merge-reports; the Docker and workers post covers how sharding interacts with the workers setting inside each job.
One honest note on sharding: it adds a job's worth of setup time and artifact transfer to every shard. If the suite finishes in a few minutes on a single job, sharding makes the workflow more complex for little gain. Reach for it when the single-job runtime starts to slow down reviews.
Copy-ready test jobs for pytest, Jest, and Playwright
Each of these is a complete .github/workflows/tests.yml with the triggers, concurrency, caching, and failure artifacts from the steps above. Pick the one that matches your suite.
pytest
name: Tests
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
pytest:
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
matrix:
python-version: ['3.12', '3.13']
fail-fast: false
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- run: pip install -r requirements.txt # include pytest and pytest-xdist
- run: pytest -n auto --junit-xml=reports/junit.xml
- if: failure()
uses: actions/upload-artifact@v7
with:
name: pytest-results-${{ matrix.python-version }}
path: reports/
retention-days: 30Jest
name: Tests
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
jest:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: '24'
cache: 'npm'
- run: npm ci
- run: npx jest --ci --coverage
- if: failure()
uses: actions/upload-artifact@v7
with:
name: jest-coverage
path: coverage/
retention-days: 30Playwright
name: Tests
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
playwright:
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
matrix:
browser: [chromium, firefox, webkit] # must match playwright.config.ts projects
fail-fast: false
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: '24'
cache: 'npm'
- run: npm ci
- run: npx playwright install --with-deps ${{ matrix.browser }}
- run: npx playwright test --project=${{ matrix.browser }}
- if: failure()
uses: actions/upload-artifact@v7
with:
name: playwright-traces-${{ matrix.browser }}
path: test-results/
retention-days: 30This Playwright job is deliberately plain. Pinning the official Docker image, choosing which screenshots, videos, and traces to keep, and setting workers for a hosted runner are covered in Playwright in GitHub Actions: Docker, Failure Capture, and Worker Parallelism.
Testland's github-actions-test-jobs skill for Claude Code generates these workflows for a given repo, including OS and runtime matrices, service containers such as PostgreSQL and Redis, and JUnit artifact upload. Once a workflow exists, the ci-pipeline-health-critic agent reviews it for missing concurrency cancellation, retries that hide flakiness, and absent report uploads.
Troubleshooting common setup issues
Cache always misses
The usual causes are a lockfile that isn't committed (the cache key is a hash of it, and npm ci refuses to run without package-lock.json), a lockfile that changes on every run because a step rewrites it, or a matrix where jobs on different operating systems overwrite each other's entries. The setup actions include the OS in their cache key; a hand-written actions/cache step needs runner.os in its key for the same reason.
Job killed after 6 hours
The GitHub docs state the hard limit plainly: "Each job in a workflow can run for up to 6 hours of execution time." For long suites that approach this, a silent kill is the worst outcome because the run shows as cancelled with no test results. Add timeout-minutes at the job level, set to a little more than your normal runtime. When the suite consistently runs into that timeout, that's a sharding signal, not a reason to raise it.
Burning through minutes on private repos
The math is straightforward: three matrix jobs at 5min each is 15 minutes per PR run. At 100 PRs/month that's 1,500 of the 2,000 free minutes on GitHub Free, leaving little room for pushes to main, nightly runs, and workflow_dispatch calls. Two options work well together: run one configuration on pull requests and the full matrix on the nightly schedule, or set up self-hosted runners, which aren't billed for minutes.
Artifacts bigger than expected
Three mistakes produce oversized artifacts: uploading a whole output directory when only the reports or traces matter, forgetting retention-days and letting artifacts pile up at the 90-day default, and a path glob that accidentally includes node_modules or a virtual environment. Scope the path: to the report directory and nothing else.
Frequently asked questions
How do you run only smoke tests on pull requests?
Tag the smoke tests and filter on the tag. For pytest, use a marker and run pytest -m smoke (see pytest markers). For Playwright, tag tests with @smoke and run npx playwright test --grep @smoke. For Jest, put the tag in the test name and run npx jest -t @smoke, which runs only tests whose full name matches. Run the full suite on the nightly schedule. This keeps PR feedback fast without giving up coverage.
Do self-hosted runners work for browser tests?
Yes, with one extra responsibility: you install the browser dependencies. On a fresh Ubuntu runner that means running npx playwright install --with-deps during the workflow, or pre-installing browsers in the runner image. Self-hosted runners are the right call when you're hitting the private-repo minute limits or need hardware that GitHub's hosted runners don't provide.
What's the difference between fail-fast and continue-on-error?
fail-fast is a matrix-level setting. When fail-fast: true (the default), a failure in any matrix job cancels all other in-progress jobs. Set it to false to let all jobs run to completion regardless of failures.
continue-on-error is a step-level setting. When true, a failing step doesn't fail the job. Use it for non-blocking checks like coverage thresholds that you want visible but don't want blocking merges. For a test matrix, fail-fast: false at strategy level is almost always the right choice.
How do you view a Playwright trace from a failed CI run?
Download the artifact from the GitHub Actions run page (under the "Artifacts" section at the bottom of the run summary). Unzip it locally. Then run:
npx playwright show-trace path/to/trace.zipThe Trace Viewer opens in your browser and shows a timeline of every action the test took, screenshots at each step, network requests, console logs, and the DOM snapshot at the point of failure. It runs entirely locally from the trace file.
What this CI foundation enables next
Tests now gate merges. That's the baseline. The next problem shifts from "do tests run in CI?" to "do the tests actually tell you something useful?" A suite that's 40% flaky passes CI by chance and fails trust. Two posts that address this directly: How to Fix Flaky Tests: A Four-Step Triage covers finding and fixing unstable tests, and regression suite organization for web apps covers structuring a growing suite so it stays maintainable.
GitHub Actions usage for testing grew 35% year-over-year, and the public runner specs keep improving. CI capacity is getting cheaper per test run. The bottleneck shifts to test design.