Test Automation Framework

The Test Automation Framework for Entgra replaces brittle record-and-playback scripts and single-engineer Playwright suites with a git-native test automation platform — a Playwright execution engine, a state-graph-verified LLM test generator, and a version-locked release model for shipping shared test infrastructure to independent product teams, all without a private npm registry and without ever trusting an AI's word for what a browser actually did.

The Test Automation Framework for Entgra is a pnpm-workspace monorepo built as a control plane for test automation — a React dashboard, a Fastify platform API of roughly 70 routes, a standalone execution worker, and four shared packages — that treats test catalogs, environments, auth strategies, evidence, and its own release cycle as independently evolving, first-class concerns. It runs the full lifecycle end to end: bootstrapping a consumer repo straight from a checksummed release tarball, syncing a git-native YAML test catalog, generating or hand-authoring Playwright tests, resolving and executing runs against a specific product version and environment, capturing policy-driven evidence, and self-healing failed runs from that evidence. The core engineering challenge was letting an LLM draft tests without ever trusting its account of what a UI does — every step it proposes has to be independently proven against a live DOM before a single line of test code gets written.

• Evidence-Verified AI Test Generation: Drives a bounded discovery loop — up to 5 actions per step and 20 total — where Claude proposes one action at a time against a live, authenticated browser, and every "step completed" claim is independently re-verified against the real DOM before it's accepted, with rejections fed back into the loop instead of failing the run outright.

• Ranked, Live-Reconstructed Selectors: Scores candidate locators by stability — test id, role plus accessible name, label, placeholder, name, id, visible text, structural CSS path as a last resort — and re-walks that ranked list against the live page at execution time, since a selector that looked unique in a static snapshot doesn't always resolve to one element under Playwright's live semantics.

• Idempotent Run Orchestration: Every worker result carries a caller-supplied idempotency key checked against a receipts table before any mutation, guaranteeing exactly-once side effects and ensuring a late result can never resurrect an already-cancelled or terminal run.

• Registry-Free Framework Distribution: Ships shared test infrastructure as a SHA-256-checksummed, signed tarball into an isolated version-locked directory rather than through a private npm registry, with a CLI that refuses to operate against a lock file claiming a newer schema version than it understands.

• Honestly-Scoped Evidence Redaction: Runs every text artifact through exact-match and generic secret-pattern redaction, but records each artifact's actual guarantee rather than overselling it — logs are redacted, traces are best-effort, and screenshots or video are explicitly flagged as not guaranteed clean.

• Catalog Sync Safety Guard: Refuses to reconcile the test catalog at all if a sync would archive more than 25% of the active catalog in one pass, treating a mass deletion as evidence of a broken scan rather than an intentional edit.

I designed the dashboard's UI/UX myself, built around operating and tracing runs rather than just displaying them — scheduling recurring runs, gating them on environment health, cancelling mid-flight, and drilling from a run down through its cases, steps, and evidence without losing context.