TestRail - vendor specifics

Reference detail for test-management-sync (opens in new window). TestRail (Gurock / Idera) is the standalone (non-Jira) test management tool; the body's worked example targets it. This page carries the full vendor detail: auth, case-ID mapping, run lifecycle, per-result fields, CI wiring, and untested-case handling.

Authentication

TestRail uses HTTP Basic auth with email + API key (preferred over password - the API key is per-user, revocable):

# Generated in TestRail: My Settings → API Keys
TESTRAIL_API_KEY=<generated>
TESTRAIL_USER=test-runner@example.com
TESTRAIL_HOST=https://yourcompany.testrail.io

All requests use:

Authorization: Basic <base64(email:api_key)>
Content-Type: application/json

The base API URL is ${TESTRAIL_HOST}/index.php?/api/v2. Every endpoint is appended after ?/api/v2.

Map test names to TestRail case IDs

Two common patterns:

Pattern A - Embed case ID in test name

def test_C1234_can_add_to_cart():
    ...

A regex extracts C1234 (the TestRail case ID) at sync time.

Pattern B - Annotation / metadata

@Test
@TestRailCase(id = 1234)
void canAddToCart() { ... }

Or in JS:

test('can add to cart [C1234]', async () => {
  // ...
});

Pattern A is the lowest-friction; Pattern B is cleaner when the test framework supports custom annotations. Either way, the sync script needs a way to find the case ID from the test result.

Open a Test Run for the build

# scripts/testrail_sync.py
import base64, json, requests
from os import environ as env

API = f"{env['TESTRAIL_HOST']}/index.php?/api/v2"
AUTH = base64.b64encode(f"{env['TESTRAIL_USER']}:{env['TESTRAIL_API_KEY']}".encode()).decode()
HEADERS = {'Authorization': f'Basic {AUTH}', 'Content-Type': 'application/json'}

def open_run(project_id, suite_id, name, case_ids):
    r = requests.post(
        f'{API}/add_run/{project_id}',
        headers=HEADERS,
        json={
            'suite_id': suite_id,
            'name': name,
            'include_all': False,
            'case_ids': case_ids,
        },
    )
    r.raise_for_status()
    return r.json()['id']  # Run ID

include_all: False + case_ids: [...] opens a run scoped to the exact cases the automated suite covers. Without this, a 5,000-case project produces a 5,000-row Test Run with thousands of empty cells.

Batch results back

The well-known status ID convention for stock TestRail installations:

Custom status IDs (added by the project admin) follow 6+. Read the get_statuses endpoint at sync-script init to confirm - don't hard-code.

Verify before batching: call get_statuses and assert every status_id in your map appears in the returned set. If one is missing (a renamed or custom status), fix the map and re-run rather than posting - TestRail accepts an unknown status_id and writes the result to the wrong status silently.

def add_results(run_id, results):
    """results = [{'case_id': 1234, 'status_id': 1, 'comment': '...', 'elapsed': '12s'}]"""
    r = requests.post(
        f'{API}/add_results_for_cases/{run_id}',
        headers=HEADERS,
        json={'results': results},
    )
    r.raise_for_status()
    return r.json()

Use add_results_for_cases (batch), not add_result_for_case (per-case). A 200-test run is one POST instead of 200 POSTs; TestRail's rate limit (180 req/min on shared cloud) makes per-case posting flaky.

Per-result fields

Fields accepted on each entry in the add_results_for_cases results array:

Close the run

After all results are in:

def close_run(run_id):
    requests.post(f'{API}/close_run/{run_id}', headers=HEADERS)

Closed runs are read-only - no further results can be added. Useful for release-stamp runs; skip for runs that get re-run.

Wire into a CI pipeline

- name: Run tests
  run: npm test -- --reporters=jest-junit
  env:
    JEST_JUNIT_OUTPUT_FILE: junit.xml

- name: Sync to TestRail
  if: always()
  env:
    TESTRAIL_HOST: ${{ secrets.TESTRAIL_HOST }}
    TESTRAIL_USER: ${{ secrets.TESTRAIL_USER }}
    TESTRAIL_API_KEY: ${{ secrets.TESTRAIL_API_KEY }}
    TESTRAIL_PROJECT_ID: '42'
    TESTRAIL_SUITE_ID: '7'
    BUILD_VERSION: ${{ github.sha }}
  run: python scripts/testrail_sync.py junit.xml

The sync script:

1. Parses junit.xml (see junit-xml-analysis).
2. Extracts case IDs from test names.
3. Opens a run named <branch> · <sha-short>.
4. Batches results.
5. Optionally closes the run - typically only on main.

Handling untested case IDs

Tests that have no TestRail case ID (case removed; new test; intentional sync-skip) need explicit handling:

unmapped = [t for t in tests if extract_case_id(t['name']) is None]
if unmapped:
    print(f"Warning: {len(unmapped)} tests have no TestRail case ID:")
    for t in unmapped:
        print(f"  - {t['name']}")

Don't silently drop unmapped tests - they're candidates for either new TestRail cases or naming-pattern fixes.

TestRail-specific anti-patterns

Limitations

• Test cases must already exist in TestRail. Pre-create cases manually or via add_case before the first sync; the sync script doesn't create cases on the fly.
• Per-step results require a different endpoint. custom_step_results is per-installation; the project admin must enable a custom field for steps before the API accepts step-level data.
• Rate limits. Cloud installations cap at 180 req/min per IP. Plan batching accordingly.
• TestRail Cloud auth is per-user, not per-app. OAuth / SSO is not supported for API access; the API key is the only mechanism.
• No first-party JUnit XML import. TestRail's own JUnit importer is a separate (deprecated) tool; the sync-script approach here is the modern path.

References

• TestRail official documentation portal: https://support.testrail.com/hc/en-us/categories/7080117421716 (categorized KB; per-endpoint articles).
• TestRail API Reference: https://support.testrail.com/hc/en-us/sections/7077986539540 (Results / Runs / Cases / Statuses endpoints).
• Per-language client libraries: testrail (Python), testrail-java-client (Java), testrail-api (JS) - reference implementations of the API shapes above.

Xray for Jira - vendor specifics

Reference detail for test-management-sync (opens in new window). Xray exposes test cases / executions as Jira issue types (Test, Test Set, Test Plan, Test Execution, Pre-Condition) - automated test sync writes results into Test Execution issues.

Xray comes in two flavors:

This page covers the Cloud flow as the primary path; Server notes are inline.

The official documentation is at docs.getxray.app. At the time of authoring (2026-05-05), the Cloud import-results page was 403 to automated WebFetch; the URL is the canonical reference for real-browser navigation. The well-known endpoints + payload shapes below are documented in the official xray-junit-extensions GitHub repo (xray-junit-ext (opens in new window)) and the xray-postman-collections public collections, both first-party Xray-App tools.

How to use

1. Determine the flavor - Xray Cloud vs Server / DC - since it sets the auth model and import host (table above).
2. Authenticate: Cloud exchanges client_id + client_secret for a 24h JWT; Server / DC uses a Jira PAT or Basic auth (Step 1).
3. Pick the import endpoint that matches your runner's output format (/junit, /cucumber, /testng, /nunit, /robot, or generic JSON) (Step 2).
4. Pin each test method to an existing Test issue with @XrayTest(key=...) (and @Requirement for coverage) so renames don't orphan issues (Step 3).
5. Emit Xray-extended output - JVM via xray-junit-extensions (Step 5), JavaScript via the official Playwright reporter (below).
6. POST the results with projectKey=..., adding testExecKey=... to update an existing Test Execution instead of creating a new one (Step 7).
7. Wire it as an if: always() CI step (End-to-end CI shape below).

Step 1 - Authenticate (Cloud)

Per the xray-junit-ext (opens in new window) reference, Cloud auth is a two-step flow:

# 1. Exchange credentials for a JWT
JWT=$(curl -X POST 'https://xray.cloud.getxray.app/api/v2/authenticate' \
  -H 'Content-Type: application/json' \
  -d '{"client_id": "'"$XRAY_CLIENT_ID"'", "client_secret": "'"$XRAY_CLIENT_SECRET"'"}' \
  | tr -d '"')

# 2. Use the JWT in subsequent requests
curl -X POST 'https://xray.cloud.getxray.app/api/v2/import/execution/junit' \
  -H "Authorization: Bearer $JWT" \
  -H 'Content-Type: application/xml' \
  --data-binary @junit.xml

The JWT has a 24-hour validity (well-documented across Xray API clients); refresh per CI run.

For Xray Server / DC, use a Jira PAT or Basic auth instead; the JWT step is skipped.

Step 2 - Pick the import endpoint per format

Each endpoint accepts the format's native output and parses it server-side; no per-test pre-mapping is needed.

For the generic JSON endpoint, payload shape:

{
  "info": {
    "summary": "CI run for ABC-123",
    "description": "Automated regression run",
    "user": "ci-runner",
    "version": "1.4.5",
    "revision": "abc1234",
    "testPlanKey": "PROJ-100",
    "testEnvironments": ["staging", "chrome"]
  },
  "tests": [
    {
      "testKey": "PROJ-1234",
      "start": "2026-05-05T14:00:00Z",
      "finish": "2026-05-05T14:00:12Z",
      "comment": "Test passed cleanly",
      "status": "PASSED"
    }
  ]
}

Step 3 - Map test methods to Xray Test issues

Per xray-junit-ext (opens in new window), the JUnit 5/6 extension provides two annotations:

@XrayTest

"enforce mapping of result to specific, existing Test identified by issue key, using the key attribute" (xray-junit-ext (opens in new window))

@Test
@XrayTest(key = "CALC-1000")
public void canAddNumbers() { /* ... */ }

Without @XrayTest, the extension auto-creates a Test issue per JUnit method on first run (auto-provisioning). Pinning with key prevents drift across renames.

@Requirement

"identify the covered requirement(s) ... it's possible to identify one covered issue or more" (xray-junit-ext (opens in new window))

@Test
@Requirement("CALC-1234")
public void canAddNumbers() { /* ... */ }

This populates the Jira-side coverage link from the test back to the requirement issue.

Step 4 - XrayTestReporter for evidence

Per xray-junit-ext (opens in new window), the XrayTestReporterParameterResolver extension injects an XrayTestReporter into test methods:

"Add comments to Test Runs / Define Test Run custom field values / Attach evidence files" (xray-junit-ext (opens in new window))

@Test
@ExtendWith(XrayTestReporterParameterResolver.class)
@XrayTest(key = "CALC-1000")
public void canAddNumbers(XrayTestReporter reporter) {
    // ... test logic ...
    reporter.addComment("Calculator returned correct sum");
    reporter.addEvidence("screenshot.png");
}

Evidence files are attached to the Test Run inside the Test Execution issue - useful for failure debugging from Jira.

Step 5 - Configure the extension

Per xray-junit-ext (opens in new window), the extension reads xray-junit-extensions.properties for output config:

# xray-junit-extensions.properties (place on the test classpath)
report_filename=TEST-results
report_directory=target/xray-reports
add_timestamp_to_report_filename=false

The output is JUnit XML augmented with Xray-specific metadata; pass this enriched XML to the import endpoint (Step 2).

Step 6 - End-to-end CI shape

# .github/workflows/xray-sync.yml
- name: Run tests with Xray-aware JUnit reporter
  run: ./mvnw -B verify
  # Produces target/xray-reports/TEST-results.xml

- name: Get Xray JWT
  id: xray_auth
  env:
    XRAY_CLIENT_ID: ${{ secrets.XRAY_CLIENT_ID }}
    XRAY_CLIENT_SECRET: ${{ secrets.XRAY_CLIENT_SECRET }}
  run: |
    JWT=$(curl -s -X POST 'https://xray.cloud.getxray.app/api/v2/authenticate' \
      -H 'Content-Type: application/json' \
      -d '{"client_id":"'"$XRAY_CLIENT_ID"'","client_secret":"'"$XRAY_CLIENT_SECRET"'"}' \
      | tr -d '"')
    echo "::add-mask::$JWT"
    echo "jwt=$JWT" >> "$GITHUB_OUTPUT"

- name: Import to Xray
  if: always()
  run: |
    curl -X POST 'https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=CALC' \
      -H "Authorization: Bearer ${{ steps.xray_auth.outputs.jwt }}" \
      -H 'Content-Type: application/xml' \
      --data-binary @target/xray-reports/TEST-results.xml

projectKey (the Jira project key) is the critical query param; without it the import fails or lands in the wrong project.

Verify after the POST: assert a 2xx whose body carries the Test Execution issue key the import created or updated. A 4xx (bad projectKey, expired JWT, or malformed XML) or a body with no key means nothing landed - fix that cause and re-run; retry once on a 5xx (transient), and never treat a non-2xx as success.

Non-JVM teams: Playwright reporter

Per the Xray-App GitHub org, the playwright-junit-reporter (opens in new window) project ships a Playwright reporter that emits Xray-compatible JUnit XML:

// playwright.config.ts
reporter: [
  ['list'],
  ['@xray-app/playwright-junit-reporter', {
    outputFile: 'target/xray-reports/results.xml',
  }],
],

Then the same import endpoint consumes the output.

Step 7 - Test Execution issue lifecycle

By default, each import creates a new Test Execution issue. For "update an existing execution per build" (e.g. one execution per release branch), pass testExecKey=PROJ-XYZ in the query string:

POST /api/v2/import/execution/junit?projectKey=CALC&testExecKey=CALC-9999

Pattern:

• PR runs: new Test Execution per push (lots of issues; auto-archive via Jira workflow after PR merge).
• Release runs: one Test Execution per release branch; updated on every push (uses testExecKey).

Xray-specific anti-patterns

Limitations

• Cloud and Server have different endpoints + auth. Don't share scripts; maintain per-flavor variants.
• Auto-provisioning creates issues in the project. Heavy auto-provisioning can balloon Jira project size; pre-create Test issues via the Xray UI or xray-postman-collections for a pre-seeded suite.
• No first-party Cypress reporter in the Xray-App org; the community-maintained cypress-xray-junit-reporter is the de-facto choice but isn't officially supported.
• Documentation domain is JS-rendered + auth-gated. Per the source-fetch failure documented here (2026-05-05), the primary canonical references for the import API require real browser navigation; the GitHub repos under https://github.com/Xray-App/ are the most reliable programmatically-fetchable sources.

References

• xray-junit-ext (opens in new window) - official xray-junit-extensions repo: @XrayTest(key=...), @Requirement(...), XrayTestReporter injection, xray-junit-extensions.properties config.
• https://github.com/Xray-App/xray-postman-collections - official Postman collections for every Xray Cloud public API endpoint (including /api/v2/authenticate and /api/v2/import/execution/*).
• https://github.com/Xray-App/xray-maven-plugin - Maven-side integration with the same import + auth shape.
• https://github.com/Xray-App/playwright-junit-reporter - official Playwright reporter for Xray-compatible JUnit XML.
• https://docs.getxray.app/ - canonical doc portal (auth/region-gated; consult in a real browser).

Zephyr Scale - vendor specifics

Reference detail for test-management-sync (opens in new window). "Zephyr" disambiguates into three Jira test-management products that are not API-compatible - picking the right one is step zero:

This page covers Zephyr Scale Cloud as the primary path - it's the most-deployed Zephyr variant in 2026 and the one new projects pick. Notes for Squad / Enterprise are inline.

The official documentation is at support.smartbear.com/zephyr-scale-cloud/. At the time of authoring (2026-05-05), the documentation site was behind WebFetch limits (auth/region-gated content); the URL is the canonical reference for real-browser navigation. Patterns below are the stable shapes documented across the SmartBear KB and per-language clients (zephyr-scale-python-client, the Postman collection SmartBear ships, and the mgechev/zephyr-scale-cloud-cli community client).

Authenticate (Zephyr Scale Cloud)

Zephyr Scale Cloud uses a long-lived API token (generated via "API Access Tokens" in the Zephyr Scale settings) sent as a Bearer token:

ZEPHYR_TOKEN=<long-lived-token>

curl -H "Authorization: Bearer $ZEPHYR_TOKEN" \
  'https://api.zephyrscale.smartbear.com/v2/healthcheck'

Unlike Xray Cloud, no JWT exchange step - the token is used directly.

The token is per-account, not per-project - guard it with the same care as a Jira admin credential.

If the team uses Zephyr Squad, the endpoints + auth differ significantly - see the Squad-specific REST API docs and the distinct prod-api.zephyr4jiracloud.com host.

Map test methods to Zephyr Test Cases

Two patterns mirror the TestRail / Xray approach.

Pattern A - Embed Test Case key in test name

def test_TC1234_can_add_to_cart():
    ...

test('can add to cart [TC1234]', async () => { /* ... */ });

A regex extracts TC1234 (the Zephyr Scale Test Case key) at sync time.

Pattern B - JUnit metadata via custom adapter

For Java / TestNG:

@Test
@TestCaseKey("PROJ-T1234")
public void canAddToCart() { /* ... */ }

The @TestCaseKey annotation is provided by community adapters (no first-party SmartBear annotation library at the time of writing); a small custom JUnit extension reads the annotation and emits a Zephyr-compatible JSON file alongside the JUnit XML.

Open a Test Cycle for the build

# scripts/zephyr_sync.py
import os, requests

BASE = 'https://api.zephyrscale.smartbear.com/v2'
HEADERS = {
    'Authorization': f"Bearer {os.environ['ZEPHYR_TOKEN']}",
    'Content-Type': 'application/json',
}
PROJECT_KEY = os.environ['JIRA_PROJECT_KEY']    # e.g. "CALC"

def open_cycle(name, version=None):
    r = requests.post(f'{BASE}/testcycles', headers=HEADERS, json={
        'projectKey': PROJECT_KEY,
        'name': name,                          # e.g. "Build #1234"
        'plannedStartDate': iso_now(),
        'description': f'Automated cycle for {os.environ.get("BUILD_VERSION", "")}',
        'jiraProjectVersion': version,         # optional Jira version ID
    })
    r.raise_for_status()
    return r.json()['key']                      # e.g. "CALC-R42"

The returned key (e.g. CALC-R42) is the Test Cycle's identifier; results land inside it.

Post execution results

Per the documented Zephyr Scale Cloud /testexecutions endpoint shape (consistent across SmartBear KB versions):

def post_execution(cycle_key, test_case_key, status, comment=None,
                   actual_end_date=None, execution_time=None):
    r = requests.post(f'{BASE}/testexecutions', headers=HEADERS, json={
        'projectKey': PROJECT_KEY,
        'testCycleKey': cycle_key,
        'testCaseKey': test_case_key,           # e.g. "CALC-T1234"
        'statusName': status,                   # 'Pass' | 'Fail' | 'Blocked' | 'Not Executed'
        'comment': comment,
        'actualEndDate': actual_end_date,       # ISO-8601
        'executionTime': execution_time,        # milliseconds
    })
    r.raise_for_status()
    return r.json()

statusName accepts the Zephyr-installed status names. For projects with custom statuses, query /statuses?projectKey=...&statusType=TEST_EXECUTION at script init to confirm the available names - don't hard-code beyond the four built-ins (Pass, Fail, Blocked, Not Executed).

Batch multiple results

The /testexecutions endpoint is per-execution. For batched POSTs, the documented /automations/executions endpoint accepts a payload that wraps multiple results - the exact shape is variant per Zephyr Scale version. The conservative pattern is to retry per-execution with bounded concurrency:

from concurrent.futures import ThreadPoolExecutor

def post_all(cycle_key, results, max_concurrent=5):
    with ThreadPoolExecutor(max_workers=max_concurrent) as ex:
        list(ex.map(lambda r: post_execution(cycle_key, **r), results))

max_concurrent=5 keeps under the rate limit (60 req/min on most plans) for typical run sizes.

Wire into CI

- name: Run tests
  run: npm test -- --reporters=jest-junit

- name: Sync to Zephyr Scale
  if: always()
  env:
    ZEPHYR_TOKEN: ${{ secrets.ZEPHYR_TOKEN }}
    JIRA_PROJECT_KEY: 'CALC'
    BUILD_VERSION: ${{ github.sha }}
  run: python scripts/zephyr_sync.py junit.xml

The script:

1. Parses junit.xml (junit-xml-analysis).
2. Extracts Test Case keys.
3. Opens a Test Cycle.
4. Posts executions with bounded concurrency (above).

Folder + label organization

Zephyr Scale Test Cases live in folders. Two patterns:

• Per-feature folder: Checkout/, Cart/, Auth/ - automated tests in those folders sync to Test Cases there.
• Per-tier folder: Smoke/, Regression/, Edge cases/ - automated tests carry a tier label that the sync script translates to folder.

The folder structure is created via the Zephyr UI; the sync script references existing Test Case keys and doesn't create folders on the fly.

JUnit XML import (alternative path)

Zephyr Scale also accepts a JUnit XML file via the /automations/executions/junit endpoint with a multipart body. This is simpler than the per-execution sync but loses per-test metadata (no comment, no execution time per case beyond what JUnit XML carries):

curl -X POST "https://api.zephyrscale.smartbear.com/v2/automations/executions/junit?projectKey=$JIRA_PROJECT_KEY&autoCreateTestCases=true" \
  -H "Authorization: Bearer $ZEPHYR_TOKEN" \
  -F "file=@junit.xml"

Per-execution POST is preferred when comment / evidence matters; this JUnit XML import is the lightweight default.

Worked example

A Jest suite syncing one build to project CALC:

1. Tests carry the Test Case key in the name (Pattern A): test('can add to cart [CALC-T1234]', ...).
2. Export the token and run tests to JUnit XML:

export ZEPHYR_TOKEN=...             # from Zephyr Scale > API Access Tokens
export JIRA_PROJECT_KEY=CALC
npm test -- --reporters=jest-junit

1. The sync script opens a cycle, then posts one execution:

cycle = open_cycle("Build #1234")           # returns e.g. "CALC-R42"
post_execution(cycle, "CALC-T1234", "Pass",
               comment="green on CI", execution_time=1240)

The execution lands inside cycle CALC-R42; open it in Jira to see the Pass recorded against CALC-T1234.

Zephyr-specific anti-patterns

Limitations

• Three Zephyr products with different APIs. Squad and Scale diverged years ago; Enterprise is its own thing. Check which variant the team has before pattern-matching tutorials.
• No first-party adapter library across all languages. SmartBear ships Postman collections + Java reference clients; Python / JS / Ruby teams use community-maintained adapters with varying maintenance status.
• Folder + Test Case management is a UI workflow. Programmatic Test Case creation exists but is fragile across versions; the sync-to-existing-cases pattern is more durable.
• Documentation site is auth/region-gated. Per the source-fetch failure documented above (2026-05-05), the canonical references require real-browser navigation; per-language client repos are the most reliable programmatic source.

References

• https://support.smartbear.com/zephyr-scale-cloud/ - canonical Zephyr Scale Cloud documentation portal (auth/region-gated; consult in a real browser).
• https://support.smartbear.com/zephyr-scale-cloud/api-docs/ - REST API reference for Scale Cloud.
• https://support.smartbear.com/zephyr-squad-cloud/ - Squad Cloud reference (different product, different API).