A shared, verified specification, not a smarter prompt, is what makes AI-generated tests trustworthy.

TL;DR

• Two engineers running the same prompt through the same AI test tool got two different test suites, a problem covered in detail in a previous piece on why AI test generation fails.
• The root cause is that a prompt is not a specification. It’s a guess the AI makes fresh each time, worded however the prompt happened to be worded.
• A shared, human-verified specification removes the guesswork: the same spec produces the same tests, every time, because the ambiguity never enters the process.
• This is the mechanism behind Atlas, an AI test design and automation studio built as a companion product upstream of Sahi Pro: Atlas builds the verified spec and generates deterministic tests from it, Sahi Pro executes them.
• Atlas is industry-agnostic, but the case for it sharpens fastest where a wrong test is genuinely expensive: lending, insurance, healthcare, and back-office fintech.
• Atlas runs on the customer’s own LLM infrastructure and API keys, including AWS Bedrock with Claude Sonnet models, so test design data never leaves the customer’s own environment by default.

A team runs the same prompt through the same AI test generation tool twice, a day apart, against the same application. They get two different test suites. Neither is wrong. Neither is the same. This is the specific failure covered in an earlier piece on why prompt-to-test AI tools break down under CI/CD scrutiny: the tool is generating tests fresh from the prompt each time, and a prompt underspecifies intended behavior enough that the AI fills the gaps differently on different runs.

Here’s what the same team would get instead, with a shared, verified specification in place: the exact same prompt, run through a spec that a human already reviewed and confirmed correct, produces the exact same tests, every single time. Not because the AI got smarter. Because the ambiguity that caused the variance was resolved once, on paper, before any test got generated.

That’s the mechanism this piece is about, and it’s the reason a specific product exists to make it practical at scale.

What a shared spec actually is

A shared specification is a fixed, reviewed description of exactly what an application should do: for these inputs, these expected outputs, these edge cases explicitly in scope or explicitly excluded. It’s the artifact a previous piece in this series described as missing from most testing workflows: the step between “someone understands what a feature should do” and “a test exists that verifies it,” made explicit and put in writing instead of living implicitly in whoever happens to write the test that day.

Once that spec exists and has been verified by someone with the right domain knowledge, it becomes the single source of truth that any test generation, human or AI, draws from. The spec doesn’t change unless the underlying requirement changes. Neither, then, do the tests generated from it.

Why deterministic beats generative here

The distinction between prompt-to-test and spec-to-test isn’t about which produces better-written individual test cases. It’s about which produces the same test cases twice.

Deterministic, in this context, means the specification and the generation logic together define an exact output: run the same spec through the same generation process, get the same tests, every time. This matters for three concrete reasons a testing team runs into quickly. A CI/CD pipeline needs failures to mean “something in the application changed,” not “something in test generation changed between runs.” A compliance or audit review needs to point to an exact test artifact and reproduce it on demand, which a regenerated, variable test set can’t reliably provide. And a team scaling test coverage across a large application needs combination coverage, systematically generating tests across the meaningful combinations of inputs a spec defines, rather than hoping a prompt happens to think of the right edge cases.

Atlas is built around this distinction specifically. It doesn’t generate tests from a prompt. It builds a human-verified specification first, with the domain expert in the loop to confirm the spec is actually correct, and only then generates tests from that spec. The output is deterministic by construction: the same spec always produces the same tests, and when the spec changes, the tests regenerate to match it, on purpose, not by chance.

Where Atlas fits

Atlas sits upstream of Sahi Pro, not inside it and not as a feature of it. Atlas is where the test design happens: building and verifying the specification, then generating the deterministic test cases from it. Sahi Pro is where those tests actually run, against the real application, using the same relational, proximity-based execution approach that already makes Sahi Pro’s test suites resilient to UI changes. Atlas designs. Sahi Pro runs.

This split matters because it means test design gets the deterministic, spec-first treatment described above, while test execution keeps the stability Sahi Pro customers already rely on. Neither product tries to be the other.

Atlas is not built for one industry. Any team where dev velocity has outpaced test design, the exact gap described elsewhere in this series, is a fit. That said, the case sharpens fastest in specific environments: lending, insurance, healthcare, and back-office fintech, where a wrong test isn’t just an inconvenience, it’s a compliance exposure or a customer-facing financial error. In these environments, being able to point to a specific, human-verified specification and reproduce the exact tests generated from it isn’t a nice-to-have. It’s frequently the artifact an auditor or regulator actually wants to see.

Cost and scale, once the spec exists

Once a specification exists and is verified, generating tests from it scales in a way manual authorship never could. A single verified spec can drive systematic combination coverage across the meaningful input combinations it defines, rather than relying on whoever writes the tests to think of the right edge cases by hand. That’s a direct answer to the gap described earlier in this series: dev velocity outpaced test velocity because manual authorship has a fixed speed ceiling. A verified spec removes that ceiling for the test-generation step specifically, without removing the human judgment that decided what the spec should say in the first place.

The practical cost profile follows from this. Onboarding a new application into this workflow means writing and verifying one spec per feature area, not writing tests for every input combination by hand. Traceability comes for free as a byproduct, not as separate documentation effort: because tests are generated from a specific, versioned spec, a team can always point to which spec produced which tests, which matters as much for an internal audit as it does for an external one. None of this requires the reader’s AI spend or infrastructure budget to grow in proportion to test volume, since the compute-heavy part, generating deterministic tests from an already-verified spec, is a fixed cost per spec rather than a cost that scales with every new test case.

Data residency: your LLM, your infrastructure

A natural question for any team evaluating an AI-driven test design tool, especially in regulated industries, is where the data goes. Atlas is built to run on the customer’s own LLM infrastructure and their own API keys, rather than requiring data to pass through a shared, vendor-hosted model. That includes the option to run on AWS Bedrock using Claude Sonnet models, contained entirely within the customer’s own cloud environment.

For a lending or healthcare team evaluating this, the practical implication is straightforward: application specifications, which can contain sensitive business logic or data-handling details, stay inside infrastructure the customer already controls and has already cleared for compliance purposes, rather than introducing a new third-party data path that has to be separately reviewed and approved.

Closing the loop on this series

The pattern across this series has been consistent: dev velocity outpaced test velocity, prompt-to-test AI tools tried to close that gap and introduced a non-determinism problem instead, and most teams have no structural way for the people who understand their domain best to close testing gaps directly. Atlas addresses the mechanism underneath all three: a verified, shared specification, generated into deterministic tests, reviewable by the domain experts who actually understand what’s being tested, running on infrastructure the customer already trusts.

See how Atlas builds a verified spec for your application, and what deterministic test generation looks like against your own use case. Request a walkthrough with the Atlas team to see the mechanism against a scenario from your own application, rather than a generic demo.

Sources

• Prior pieces in this series: “Your Team Ships Five Times Faster. Nobody Told Testing.” and “Two Engineers. Same Prompt. Same AI Tool. Different Tests.”
• Amazon Web Services, AWS Bedrock documentation on model hosting and customer data boundaries
• ISTQB (International Software Testing Qualifications Board), material on test design and specification-based testing