Testland
Browse all skills & agents

unreal-automation-system

Author and run Unreal Engine's Automation Test Framework - Epic's C++ test framework for UE 4.x / 5.x, documented at dev.epicgames.com/documentation/en-us/unreal-engine. Covers the five test categories Epic defines (Unit / Feature / Smoke / Content Stress / Screenshot Comparison), the IMPLEMENT_SIMPLE_AUTOMATION_TEST and IMPLEMENT_COMPLEX_AUTOMATION_TEST macros, the BDD-style Automation Spec API (DEFINE_SPEC / BEGIN_DEFINE_SPEC / Describe / It / BeforeEach / LatentIt / xIt), latent commands (ADD_LATENT_AUTOMATION_COMMAND), the Automation Driver for UI input simulation (IAutomationDriverModule::Get().CreateDriver(), By::Id / By::Path locators), running via Session Frontend (Window > Test Automation) and command line (-ExecCmds="Automation RunTests …"), and CI integration. Use when the unit under test is C++ Unreal code that needs the UE runtime, editor, or UMG UI surface.

Install with skills.sh (any agent)

npx skills add testland/qa --skill unreal-automation-system
View source

unreal-automation-system

Overview

This skill wraps the C++ Automation Test Framework (UE 4.x / 5.x) plus the two most commonly composed sub-systems. Per Epic's Automation Test Framework documentation (opens in new window), the framework spans five test categories (Unit, Feature, Smoke, Content Stress, Screenshot Comparison) and supports multiple authoring styles (traditional, BDD Spec, UI Driver, Functional, Python / Blueprint).

The full authoring code (macros, Spec, Driver) lives in references/authoring-macros-and-apis.md; running, report parsing, and CI wiring live in references/running-and-reporting.md. This file is the decision surface.

Composes with:

  • game-test-categories-reference for the canonical six categories Unreal's five categories map to.
  • platform-cert-overview-reference for cert-gated requirements automation tests should cover.
  • multiplayer-state-machine-coverage for replication / dedicated-server state coverage authored as automation tests.

When to use

  • Unit under test is C++ code in an Unreal Engine project (UE 4.x or UE 5.x) that needs the engine runtime, editor APIs, or UMG UI surface.
  • You want CI-runnable tests via Unreal's command-line Automation entry point.
  • You need BDD-style readable specs (Automation Spec) or scripted UI input simulation (Automation Driver) on top of plain assertion tests.

For Python / Blueprint editor tests outside C++, see Editor Automation in Unreal Engine (opens in new window). For end-to-end pipeline / build-farm orchestration, use Gauntlet (opens in new window) on top of the framework this skill covers.

How to use

  1. Pick the category from the table below (Unit / Feature / Smoke / Content Stress / Screenshot Comparison) and the matching EAutomationTestFlags filter + application-context mask.
  2. Pick the authoring style - plain RunTest macro, data-driven complex macro, BDD Spec, or Automation Driver for UI - and lift the pattern from references/authoring-macros-and-apis.md.
  3. Name the test path as dot-separated segments (MyGame.Health.Damage_…); the tree keys the Session Frontend.
  4. Run in-editor via Session Frontend to iterate, then run headless via UnrealEditor-Cmd.exe … -ExecCmds="Automation RunTests …" for CI.
  5. Parse the -ReportOutputPath JSON - treat any "state": "Fail" as a failed build; full schema in references/running-and-reporting.md.
  6. Split the CI matrix - SmokeFilter on PRs (sub-second), ProductFilter | StressFilter plus screenshot comparison nightly.

Test categories and flags

Per the Automation Test Framework page (opens in new window), Epic's five categories are:

CategoryPurpose
Unit"API level verification tests."
Feature"System-level tests that verify such things as PIE, in-game stats, and changing resolution."
SmokeTests that "complete within 1 second" and run automatically.
Content Stress"More thorough testing of a particular system to avoid crashes."
Screenshot ComparisonFor comparing renders "between versions or builds".

Each test declares flags from EAutomationTestFlags that mix:

  • Filter - SmokeFilter, EngineFilter, ProductFilter, PerfFilter, StressFilter, NegativeFilter.
  • Application context - EditorContext, ClientContext, ServerContext, CommandletContext, plus the convenience mask ApplicationContextMask.

Typical combo for a product-level test runnable in editor / client / server contexts: EAutomationTestFlags::ProductFilter | EAutomationTestFlags::ApplicationContextMask.

Authoring styles

Pick the style that matches the unit under test, then lift the full pattern from references/authoring-macros-and-apis.md:

StyleMacro / APIUse when
Simple testIMPLEMENT_SIMPLE_AUTOMATION_TEST + RunTestOne assertion body over plain C++
Data-drivenIMPLEMENT_COMPLEX_AUTOMATION_TEST + GetTestsOne sub-test per enumerated row (assets, configs)
Multi-frameADD_LATENT_AUTOMATION_COMMANDTest must yield to the tick loop across frames
BDD SpecDEFINE_SPEC / BEGIN_DEFINE_SPEC + Describe / It / LatentItReadable specs; .spec.cpp files; async via FDoneDelegate
UI DriverIAutomationDriverModule::Get().CreateDriver() + By::IdSimulate cursor / click / type on UMG; runs off the GameThread

Running

Iterate in-editor via Window → Test Automation (Session Frontend), then run headless for CI:

UnrealEditor-Cmd.exe MyGame.uproject \
    -ExecCmds="Automation RunTests MyGame.Inventory; Quit" \
    -unattended -nopause -testexit="Automation Test Queue Empty" \
    -ReportOutputPath="artifacts/automation" \
    -log

Command variants, the -ReportOutputPath JSON schema, Gauntlet, and a full GitHub Actions job are in references/running-and-reporting.md.

Worked example

Goal: cover an inventory system's stacking rule as a CI-gating BDD spec, plus a menu-close UI check.

  1. Category + flags. Stacking logic is pure C++ product logic, so ProductFilter | ApplicationContextMask; it runs in well under a second, so a second SmokeFilter copy joins the PR job.
  2. Authoring. Write FInventorySpec with DEFINE_SPEC and a Describe("AddItem") block holding two It("should …") cases - one asserting count increases by the stack amount, one asserting an over-cap add is rejected (pattern in references/authoring-macros-and-apis.md). Save it as InventorySpec.spec.cpp.
  3. UI check. Add FMenuDriverSpec with the Driver enabled in BeforeEach on TaskGraphMainThread, the It running on EAsyncExecution::ThreadPool, By::Id("SubmitButton") clicked, and Disable() in AfterEach.
  4. Run headless. UnrealEditor-Cmd.exe MyGame.uproject -ExecCmds="Automation RunTests MyGame.Inventory; Quit" -unattended -nopause -ReportOutputPath="artifacts/automation" -log.
  5. Gate. CI reads artifacts/automation/index.json; the over-cap It shows "state": "Fail" when the rule regresses, so the job fails the build. Nightly re-runs the same specs under ProductFilter | StressFilter.

Anti-patterns

Anti-patternWhy it failsFix
All tests as SmokeFilterSmoke tests "complete within 1 second" per framework docs (opens in new window) - long tests break the smoke contractUse ProductFilter for tests > 1 s
Running Automation Driver on GameThreadAPI "cannot run on the GameThread" per Driver docs (opens in new window)Use EAsyncExecution::ThreadPool on the driver-using It
Spec without It descriptions starting with "should"Runner output reads poorlyPer Spec docs (opens in new window), start descriptions with "should"
Trusting By::Path locators"Brittle" per Driver docs (opens in new window)Prefer By::Id with tagged metadata
Sleep-style waits in latent commandsFlaky under CI loadUse FDoneDelegate (LatentIt) or custom IAutomationLatentCommand::Update() polling
Spec test file without .spec.cpp extensionBuild system may not pick it upPer Spec docs (opens in new window), use .spec.cpp suffix and no "Test" in filename
Cloning non-thread-safe shared pointers in async ItCrash under threadpoolPer Driver docs (opens in new window), cache them on the test class
No BeforeEach / AfterEach cleanup of IAutomationDriverModuleDriver state leaks between specsPair Enable() / Disable() calls in BeforeEach / AfterEach per Driver docs (opens in new window)

Limitations

  • C++ build required. Tests live in C++ modules; pure-content Blueprint projects need a code module added to use this framework. (Pure Blueprint projects can use Blueprint Functional Tests in-level - see Epic's docs on Functional Testing.)
  • No common exit code definition comparable to Unity's caveat - parse the -ReportOutputPath JSON or scrape the log for LogAutomationController Fail lines.
  • Spec parameterised tests are loop-generated per Spec docs (opens in new window); there is no [TestCase] analogue from NUnit. The framework is NUnit-inspired but not NUnit-derived (unlike Unity's UTF).
  • Screenshot comparison baseline storage + tolerance configuration is engine-version-specific - consult per-version Epic docs.
  • Editor-context tests cannot run in dedicated-server only builds - declare EditorContext on those tests; client / server tests need their own flag set.
  • Documentation source. The dev.epicgames.com docs (opens in new window) are the public mirror; deeper detail (full macro implementations, precise JSON report schema) lives in the engine source under Engine/Source/Runtime/AutomationController/ and Engine/Source/Developer/AutomationMessages/ - partners with engine source access should consult those for authoritative details.

References

Authoring macros and APIs

View source (opens in new window)

Authoring macros and APIs

Deep reference for unreal-automation-system SKILL.md - the full macro, Automation Spec, and Automation Driver code. Consult after picking the test category and flags in the spine, when writing the actual test body.

Simple automation test

Per Automation Test Framework documentation (opens in new window), the canonical macro pair is IMPLEMENT_SIMPLE_AUTOMATION_TEST + RunTest:

#include "Misc/AutomationTest.h"

IMPLEMENT_SIMPLE_AUTOMATION_TEST(
    FHealthComponentDamageTest,
    "MyGame.Health.Damage_DeductsCorrectAmount",
    EAutomationTestFlags::ProductFilter |
        EAutomationTestFlags::ApplicationContextMask)

bool FHealthComponentDamageTest::RunTest(const FString& Parameters)
{
    UHealthComponent* Health = NewObject<UHealthComponent>();
    Health->Initialize(/* MaxHealth */ 100.f);

    Health->ApplyDamage(35.f);

    TestEqual(TEXT("Current HP after 35 damage"),
              Health->GetCurrent(),
              65.f);
    return true;
}

Naming convention: the second macro argument ("MyGame.Health.Damage_…") is the test path shown in the Session Frontend. Dot-separated segments build a tree.

Complex automation test (data-driven)

IMPLEMENT_COMPLEX_AUTOMATION_TEST generates a sub-test per row returned from GetTests:

IMPLEMENT_COMPLEX_AUTOMATION_TEST(
    FAllAssetsLoadCleanlyTest,
    "MyGame.Assets.LoadCleanly",
    EAutomationTestFlags::ProductFilter |
        EAutomationTestFlags::ApplicationContextMask)

void FAllAssetsLoadCleanlyTest::GetTests(
    TArray<FString>& OutBeautifiedNames,
    TArray<FString>& OutTestCommands) const
{
    // Enumerate every .uasset under /Game/Characters
    // and emit one sub-test per asset path.
}

bool FAllAssetsLoadCleanlyTest::RunTest(const FString& Parameters)
{
    // Parameters is the row from GetTests.
    UObject* Loaded = StaticLoadObject(UObject::StaticClass(),
                                       nullptr, *Parameters);
    return TestNotNull(TEXT("Asset loaded"), Loaded);
}

Latent commands

For tests that must span multiple frames, use ADD_LATENT_AUTOMATION_COMMAND to chain commands that yield back to the engine tick loop:

ADD_LATENT_AUTOMATION_COMMAND(
    FEngineWaitLatentCommand(/* Seconds */ 2.0f));

Custom latent commands derive from IAutomationLatentCommand and override Update() (returns true when complete).

Automation Spec (BDD style)

Per the Automation Spec documentation (opens in new window), specs are "built following the Behavior Driven Design (BDD) methodology" and use Describe / It / BeforeEach / AfterEach instead of one RunTest body. Spec files use .spec.cpp extension.

Simple spec:

DEFINE_SPEC(
    FInventorySpec,
    "MyGame.Inventory",
    EAutomationTestFlags::ProductFilter |
        EAutomationTestFlags::ApplicationContextMask)

void FInventorySpec::Define()
{
    Describe("AddItem", [this]()
    {
        It("should increase count by the stack amount", [this]()
        {
            FInventory Inv;
            Inv.AddItem(EItem::Potion, /* Count */ 3);
            TestEqual(TEXT("Potion count"),
                      Inv.GetCount(EItem::Potion),
                      3);
        });

        It("should reject items past max stack", [this]()
        {
            FInventory Inv;
            const bool bOk = Inv.AddItem(EItem::Potion, 999);
            TestFalse(TEXT("Over-cap add rejected"), bOk);
        });
    });
}

Per the same docs, It() descriptions should "start with 'should'" so the runner output reads as full sentences ("Inventory AddItem should increase count by the stack amount").

For specs with shared state, use BEGIN_DEFINE_SPEC / END_DEFINE_SPEC:

BEGIN_DEFINE_SPEC(
    FBackendSpec,
    "MyGame.Backend",
    EAutomationTestFlags::ProductFilter |
        EAutomationTestFlags::ApplicationContextMask)
    TSharedPtr<FMyBackendClient> Client;
END_DEFINE_SPEC(FBackendSpec)

void FBackendSpec::Define()
{
    BeforeEach([this]()
    {
        Client = MakeShared<FMyBackendClient>();
    });

    LatentIt("should return items asynchronously",
        [this](const FDoneDelegate& Done)
    {
        Client->QueryItemsAsync([Done](const TArray<FItem>& Items)
        {
            // assert on Items here, then signal completion
            Done.Execute();
        });
    });
}

LatentIt (per the Automation Spec docs (opens in new window)) gives the test a FDoneDelegate it must invoke when the async work has finished - Unreal's analogue of a Promise / future-based test.

Disabling: prefix the spec function with x (xIt(…), xDescribe(…)) per the same docs.

Automation Driver (UI input simulation)

Per the Automation Driver documentation (opens in new window), Automation Driver "enabling programmers to simulate user input … cursor movement, clicks, pressing, typing, scrolling, drag-and-drop, and more". It pairs with Automation Spec; the synchronous Driver API cannot run on the GameThread so the test must execute on a ThreadPool context.

Pattern (paraphrased from the same page):

BEGIN_DEFINE_SPEC(FMenuDriverSpec,
    "MyGame.Menu.Driver",
    EAutomationTestFlags::ProductFilter |
        EAutomationTestFlags::ApplicationContextMask)
    FAutomationDriverPtr Driver;
END_DEFINE_SPEC(FMenuDriverSpec)

void FMenuDriverSpec::Define()
{
    BeforeEach(EAsyncExecution::TaskGraphMainThread, [this]()
    {
        IAutomationDriverModule::Get().Enable();
        Driver = IAutomationDriverModule::Get().CreateDriver();
    });

    It("Submit button should close the menu",
       EAsyncExecution::ThreadPool, [this]()
    {
        FDriverElementRef Submit = Driver->FindElement(
            By::Id("SubmitButton"));
        Submit->Click();
        // assert menu state changed …
    });

    AfterEach(EAsyncExecution::TaskGraphMainThread, [this]()
    {
        IAutomationDriverModule::Get().Disable();
    });
}

Locator hierarchy (per the Automation Driver page (opens in new window)):

LocatorWhenCaveat
By::Id("SubmitButton")Most reliableRequires explicit metadata tagging on the widget
By::Path("…")Powerful"Brittle" per the docs - depends on widget hierarchy
By::Cursor()Returns widget under cursorMainly for hover tests
By::Delegate(…)Custom lambda discoveryPower-user fallback

FindElement returns a FDriverElementRef; actions like Click(), Type(), TypeChord() are exposed on the element. "All Automation Driver actions automatically wait the configured ImplicitWait timespan for any dependent scenarios" per the same page.

Running and reporting

Deep reference for unreal-automation-system SKILL.md - how to execute the authored tests (editor and command line), how to parse the JSON report, and how to wire the run into CI. Consult after the tests are written.

Session Frontend (Editor)

Per the Automation Test Framework documentation (opens in new window), open Window → Test Automation (or equivalent Session Frontend) inside the Editor. Tests appear as a tree keyed on the test path (the dot-separated string from the test macro). Click Run Tests for selected leaves; results show pass / fail

  • log output inline.

Command line

Run tests from the command line by launching the editor / commandlet with -ExecCmds:

UnrealEditor-Cmd.exe MyGame.uproject \
    -ExecCmds="Automation RunTests MyGame.Inventory; Quit" \
    -unattended -nopause -testexit="Automation Test Queue Empty" \
    -ReportOutputPath="artifacts/automation" \
    -log

Variants:

  • Automation RunTests <Filter> - run tests matching the path prefix.
  • Automation RunAll - run every registered test.
  • Automation Quit - quit when queue drains.

On Windows the editor binary is UnrealEditor-Cmd.exe; on macOS / Linux the launcher script is UnrealEditor with the same flags.

Gauntlet for build-farm runs

For full build-and-test pipelines (deploy a packaged build to a target devkit, run automation, collect artifacts), the next layer up is Gauntlet (Unreal's automation harness referenced on the Automation Test Framework page (opens in new window)). Out of scope for this skill - gauntlet wraps the same Automation RunTests command-line surface internally.

Parsing results

The -ReportOutputPath directory contains a JSON index.json plus per-test JSON / HTML detail. Top-level shape:

{
  "devices": [{
    "deviceName": "WindowsEditor",
    "instance": "WindowsEditor"
  }],
  "reportCreatedOn": "2026-05-21T10:33:00Z",
  "succeeded": 38,
  "failed": 1,
  "succeededWithWarnings": 2,
  "notRun": 0,
  "totalDuration": 142.7,
  "tests": [
    {
      "fullTestPath": "MyGame.Inventory.AddItem should increase count by the stack amount",
      "testDisplayName": "should increase count by the stack amount",
      "state": "Success",
      "entries": [/* per-step log entries */]
    },
    {
      "fullTestPath": "MyGame.Inventory.AddItem should reject items past max stack",
      "state": "Fail",
      "entries": [/* assertion failure detail */]
    }
  ]
}

(Field names paraphrased from observed Unreal output; the precise schema ships with the engine version under Engine/Source/Runtime/AutomationController/. For CI gating, treat any "state": "Fail" as a failed build.)

There is no public common exit code; parse the JSON or scrape the log for LogAutomationController: Display: Test ...: Result: Fail lines.

CI integration

GitHub Actions example (paraphrased from common Unreal CI patterns; Unreal CI itself is documented inside the editor SDK):

jobs:
  unreal-automation:
    runs-on: windows-2022
    steps:
      - uses: actions/checkout@v4
        with:
          lfs: true
      - name: Run automation tests
        shell: pwsh
        run: |
          $UE = "C:\Program Files\Epic Games\UE_5.4\Engine\Binaries\Win64\UnrealEditor-Cmd.exe"
          & $UE "$PWD\MyGame.uproject" `
            -ExecCmds="Automation RunTests MyGame; Quit" `
            -unattended -nopause `
            -testexit="Automation Test Queue Empty" `
            -ReportOutputPath="$PWD\artifacts\automation" `
            -log
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: automation-report
          path: artifacts/automation

CI pipelines typically split the test run into:

  • PR job - EAutomationTestFlags::SmokeFilter only ("complete within 1 second" per the framework docs (opens in new window)), for sub-minute PR feedback.
  • Nightly job - ProductFilter | StressFilter plus screenshot comparison.

For a sketch of how internal QA categories (game-test-categories-reference) map to Unreal's flags, see the category reference.

Related skills

game-perf-profiling

Profiles game builds against frame-time, memory, GPU draw-call, and GC-spike budgets using Unity Profiler + Profile Analyzer + Performance Testing package and Unreal Insights + stat commands. Establishes pass/fail thresholds (16.6 ms at 60 fps, 33.3 ms at 30 fps), writes automated performance regression tests that run in CI, and emits a structured budget report per SKU. Use when a title must hit a declared frame-time or memory budget before a milestone gate or platform-cert submission, or when a recent change needs a performance regression check.

game-test-categories-reference

Pure-reference catalog of the testing categories that apply to a video-game build before it ships. Defines the six canonical buckets the industry tests against - functional / compliance / compatibility / performance / localization / accessibility - plus the multiplayer and content-rating sub-axes. Cross-references each bucket to the platform-holder vocabulary that drives it (Microsoft Xbox Requirements / XR test cases, Sony TRC, Nintendo Lotcheck, Steam Direct review). Use as the taxonomy lookup when planning a game test pass, scoping QA effort, mapping platform-cert findings back to internal test categories, preparing a submission checklist, reviewing first-party certification requirements, or triaging cert testing failures against internal categories.

gameplay-recording-replay

Build a deterministic gameplay record/replay test artefact for Unity, Unreal, or Godot - record a player session, save it to disk, replay it bit-for-bit, and assert that the resulting game state matches the original. Covers Unity Input System's InputEventTrace API (Enable / Disable / WriteTo / ReadFrom / Replay) for input-level capture, Unreal's Replay System (DemoRec / DemoPlay / DemoStop console commands plus DemoNetDriver + NetworkReplayStreamer, default storage at %LOCALAPPDATA%/{Project}/Saved/Demos) for replication-stream capture, and Godot's community-pattern deterministic-RNG + input-script replay since Godot ships no first-party replay system. Use when authoring a regression-test artefact for player-recorded sessions, building a netcode replay for spectator / esports, or producing reproducible bug repros for cert teams.

godot-gut-tests

Author and run GUT (Godot Unit Test) - the community-canonical GDScript test framework at github.com/bitwes/Gut and gut.readthedocs.io. Covers install (Godot Asset Library or manual `addons/gut/` copy + plugin enable), GUT panel inside the editor, writing tests that extend GutTest with `test_` prefix methods, the assertion family (assert_eq / assert_almost_eq / assert_true / assert_signal_emitted), lifecycle hooks (before_each / after_each / before_all / after_all), inner classes for grouping, parameterized tests via `params=[...]`, doubles / stubs / spies, async / coroutine tests, the command-line runner (`-d -s addons/gut/gut_cmdln.gd -gdir=res://test -gjunit_xml_file=... -gexit`), JUnit XML export, and CI integration. Godot 4.x uses GUT 9.x (current main branch supports 4.6.x; godot_4_7 branch for 4.7.x); Godot 3.x uses GUT 7.x. Use when the unit under test is GDScript code in a Godot project.

multiplayer-state-machine-coverage

Build a coverage matrix for a networked-game state machine that exercises connect / authority-handoff / disconnect / reconnect / host-migration paths across Unity Netcode for GameObjects, Unreal Engine replication, and Mirror Networking. Workflow: enumerate the engine's connection states + ownership states + replicated-property update rules, cross them against latency / loss / out-of-order packet injection, encode each combination as a test fixture, and emit a go / no-go gate. Use before submitting a multiplayer title to platform cert - Microsoft's cert guide lists 'Multiplayer does not work as expected' as one of the most common Hold reasons, and Xbox XR-067 (MPSD session state) is failed by uncovered state-machine paths.

platform-cert-overview-reference

Pure-reference catalog of the four platform-holder certification regimes a multi-platform title submits to before release: Microsoft Xbox Requirements (XR) / Xbox certification on learn.microsoft.com, Sony Technical Requirements Checklist (TRC) on the gated PlayStation DevNet portal, Nintendo Lotcheck on the gated Nintendo Developer Portal, and Steam Direct review on partner.steamgames.com. Documents the submission workflow, severity / pass-fail vocabulary, test-bench configurations, and known SLAs for each platform. Cites public sources inline; cites gated NDA portals by stable ID per PLUGIN_AUTHORING.md Step 4 fallback. Use when planning a cert calendar, mapping internal QA findings to the platform's vocabulary, or sequencing submissions across platforms.

unity-test-framework

Author and run the Unity game-engine Test Framework (`com.unity.test-framework`, currently v1.8). Distinct from the ThrowTheSwitch Unity C testing library at throwtheswitch.org/unity - the two tools share only a name. Covers package install via Package Manager, the EditMode vs PlayMode split, the [Test] / [UnityTest] / [SetUp] / [TearDown] / [UnityPlatform] attributes, assembly-definition setup (Editor folder vs asmdef with `includePlatforms` / `optionalUnityReferences: [TestAssemblies]`), Test Runner window, command-line batch invocation with `-runTests` / `-testPlatform` / `-testResults` / `-testFilter` / `-testCategory`, NUnit 3.5 assertion API, and CI integration. Use when the unit under test is C# Unity code that needs to exercise the Unity runtime or editor.