Amazon says no human wrote or reviewed the code for its new cloud testing tool

Amazon released AWS Emulator on Nov. 11, a local tool that imitates AWS services so developers can test cloud applications on their own machines. The company says autonomous AI agents wrote and reviewed its implementation.

In Amazon's engineering account, engineer Matthew Goldberg says his team neither wrote nor reviewed the emulator code. He also describes the substantial engineering work needed to turn an experiment into a product, including the guardrails and checks that determine which changes can ship.

Amazon describes three teams of agents: one manages fidelity tests, another finds differences between the emulator and real AWS, and a third fixes individual issues. Changes face checks for compilation, coding standards, security and regressions before merging, according to the company.

The test agents work from public AWS API models and verify their scenarios against the real cloud. The public evaluation suite says every fidelity test was checked against at least two sandboxed AWS accounts.

AWS calls the degree of agreement with its cloud services “fidelity.” A test passes when the emulator matches AWS for every call in that workflow, including status codes, resulting resources, side effects and errors, according to its fidelity documentation. Returning success where AWS returns an error fails that test.

The boundary is coverage. The evaluation repository explicitly says behavior without a fidelity test is unverified. Passing a scenario establishes a match for that scenario; it does not establish that every way an application uses AWS will work.