It is possible now to reliably and automatically detect and fix every bug which users are encountering on the web. This clearly hasn’t happened yet. Our mission at Replay is to bring it about.
A few ingredients are needed:
- A modern coding agent
- Observability to see every bug
- Observability to root cause each bug
- A harness wiring these together
The right observability is key to this process, as it is generally to the AI powered SDLC.
Agents need context to thrive, and observability is the context layer for agents to know how the app they are building actually behaves. When they don’t actually know what is going on, they start guessing, make mistakes, and things go downhill quickly. When they do know, they can confirm theories about the app’s behavior and stay on track.
Better observability drives better results.
Perfect Observability
Replay’s core technology is a time travel debugger for Chromium and Node.js. Forked versions of these runtimes capture non-deterministic inputs to the runtimes with low overhead, allowing the runtime to be replayed deterministically. Everything it did while recording can be recovered and unpacked into what is essentially a database, allowing agents to see every statement that was executed, the state at every point, every control and data dependency, and so on, with no loss of detail.
A time travel debugger provides perfect observability: every question about how the app behaved can be answered. With one in hand, agents never need to guess. They can figure out exactly what happened, work their way back to the actual root cause of a problem, and develop the right fix.
Replay QA
We launched Replay QA a few months ago as a drop in tool for finding and explaining every bug in an app, building on our debugger. QA explores and tests apps on its own or with guidance, and with new self healing capabilities can create recordings by reproducing user sessions from partial and redacted data. It processes the recordings it creates to spot a wide variety of bugs (from broken features to glitches, performance issues, security holes and so on) and for every bug identifies the root cause and supports its analysis with evidence drawn from the recording.
QA can also fix itself. Its entire backend is recorded, and when problems are detected (failures to spot a bug or root cause it accurately) those problems are debugged, fixed, and converted to a durable test case by a software factory.
As an experiment we ran QA on 40 apps that have won contests or otherwise been promoted by major app building tools (Lovable, Replit, Base44, Bolt.new, etc). We had Astra confirm and deduplicate findings and ended up with over 900 bugs, with the distribution below.
Example: dead button
For comparison we ran other QA tools on some of these apps. Below is a bug report from QA Wolf on one of the apps, CallVance. A download button doesn’t work. QA Wolf identifies and generates STR for the bug, but does not explain at all what is causing it.
An agent addressing this bug report has to read the code and guess what the root cause is. If it guesses wrong it will have to go through several repair cycles and might end up stuck and unable to fix the problem.
These problems are not specific to CallVance or QA Wolf. The products of app building tools are loaded with bugs, and QA’ing them without using a time travel debugger can only give limited information about the causes. Surface level observations from watching screen recordings or looking at network and console logs can sometimes suggest a cause, but can’t identify it definitively, and the agent is always left guessing.
Below is a report from Replay QA for the same dead button.
Replay QA identifies the exact problem: a missing onClick handler on the React component. QA identified the rendering function associated with the button, and produced evidence showing that function and the missing handler. An agent reviewing this report can cross check the root cause against the evidence, and fix the bug itself without guessing.
Example: layout shift
The technique above applies just as well to more complex bugs, including ones that are difficult to spot without recording based analysis. Below is a layout shift in the same CallVance app. While the app loads, a skeleton changes size and other controls on the page shift around on their own, impacting usability.
Below is the root cause from the report.
This report is more extensive and describes the root cause of the layout shift in reference to all the contributing factors, including the different React commits, the network requests made in between, and variable values in the component render functions. This detailed report provides an agent exactly what it needs to fix the layout shift properly.
What’s Next
Replay QA is generally available and free to try today. It’s easy to set up integrations including self healing user session analysis, github and agent connections, and scheduled runs to keep a close watch on your app according to your needs. We are continually improving QA and are working on incorporating backend recording generation and analysis more deeply for full stack time travel debugging.
QA is also available to use via API to generate recordings and analyze them for bugs. Contact us if you’re interested in integrating its capabilities directly into a product you’re building.