Meetup Summary: What Did You Test? Rethinking Test Coverage with AI

Solution Street recently hosted a NOVA SART technical meetup at our Herndon, Virginia office featuring a thought-provoking presentation on a deceptively simple question: “What did you test?” Presented by Dave Piacente, a longtime software testing and test automation expert, the session explored a common challenge facing modern development teams: even when automated tests are passing and coverage numbers look good, teams can still struggle to explain what behaviors their tests actually exercised. Dave shared his experience with test automation, Selenium, and enterprise development teams before demonstrating an emerging AI-assisted approach for turning test activity into a clearer picture of what was actually tested.

From “Does It Pass?” to “What Did We Test?”

For years, software teams have focused on questions such as:

• How does the organization generate value?
• How is the system actually being used?
• How much of the code or functionality is covered by tests?

These are important questions, but Dave argued that they don’t necessarily tell us what was actually tested.

A test suite can report 90% coverage and have a completely green CI (Continuous Integration) pipeline, yet an organization can still release a serious defect. The problem is that traditional measures of coverage tend to tell us whether certain code or features were exercised, but not whether the tests addressed the behaviors that matter most to customers and the business.

That distinction led to the central question of the presentation:

What did you test?

The question sounds simple, but answering it in a useful and repeatable way can be surprisingly difficult.

Start With the Value the Software Is Supposed to Deliver

One of the first ideas Dave explored was the importance of understanding the larger purpose of a system.

Instead of beginning with a list of features and asking whether each feature was tested, teams should first understand the goals of the organization, product, or system. From there, they can work backward:

Who are the customers? What problems are we solving for them? What value are we providing? And do our tests actually exercise the behaviors that deliver that value?

This perspective becomes especially important in large organizations, where an individual development or QA team may be several layers removed from the ultimate customer.

The goal isn’t necessarily to understand the entire enterprise. Instead, teams can identify the largest meaningful area within their sphere of influence (perhaps a division, product line, or set of applications) and ask how that part of the organization creates value.

That context can then help identify gaps between what the software is intended to accomplish and what the test suite actually exercises.

An Evolution of Test Automation

Dave also walked through the evolution of automated testing.

Teams have moved from the familiar “works on my machine” approach toward automated tests running continuously in CI environments. From there came an increased focus on test coverage and increasingly sophisticated automation.

Now, AI is entering the picture, with the possibility of AI-generated tests and AI-assisted verification.

But Dave’s point was that simply adding more automation, or even AI, doesn’t automatically solve the underlying problem.

You can have extensive automation, a high coverage percentage, and a green CI pipeline while still having a major blind spot.

In other words, coverage alone doesn’t necessarily equal protection.

Capturing What Happened in the Browser

Dave’s proposed solution starts by capturing something that already exists: the events generated by the browser while an end-to-end test is running.

His open-source project uses a browser extension to observe actions such as:

• Navigation between pages
• User interactions
• Elements being clicked or interacted with
• Text entered into fields
• Relevant element attributes
• Page state
• JavaScript alerts and errors
• Other observable browser events

The goal isn’t initially to determine whether the test passed or failed. Instead, the system captures evidence of what actually happened during the test.

This approach is intentionally lightweight. Rather than requiring integrations or SDKs for every possible testing framework, the browser extension can sit alongside the existing automation and collect events without requiring extensive changes to the test code.

The captured information is stored and can be processed after the test completes, making it well suited to a continuous integration environment.

Turning Test Events Into Meaning With an LLM

The next step is where AI enters the picture.

Raw browser events aren’t particularly useful to a human. A sequence such as clicking a button, entering an email address, navigating to a billing page, and selecting a cancellation option doesn’t immediately communicate the business behavior being tested.

An LLM can take those events and turn them into a concise description of the behavior.

For example, rather than describing every individual interaction, the system might determine that the test represents:

A returning customer cancels their subscription.

This distinction is important because automated tests often contain common subflows that aren’t really the behavior being tested. Logging in, for example, may be required before nearly every test but isn’t necessarily the purpose of those tests.

The system therefore attempts to identify and remove these repeated subflows so that the resulting descriptions focus on the meaningful behavior.

From Test Descriptions to Meaningful Coverage

Once the system has interpreted the tests, the resulting behaviors can be compared with expected behaviors.

For example, a team might want to know:

• Was the “forgot password” behavior tested?
• Was searching with right-to-left language input tested?
• Was viewing the dashboard tested?
• Was account creation tested?

The system can compare those expected behaviors against the behaviors observed during the automated tests and categorize them as:

• Covered
• Partially covered
• Not observed

Importantly, the presentation demonstrated a combination of AI and deterministic logic.

The LLM is used to interpret the raw evidence and create meaningful descriptions of the observed behavior. A deterministic comparison layer then compares those interpreted flows against the expected behaviors.

This creates a potentially more useful picture of coverage: not simply “what percentage of the code ran?”, but “which behaviors did our tests actually exercise?”

A Live Demonstration

Dave demonstrated the concept using a sample web application and a suite of 11 automated tests implemented in both Playwright and Selenium.

The system captured the browser activity generated by the tests and then processed the resulting event streams through an LLM.

The resulting interface provided several layers of information, including:

• A human-readable description of each test’s behavior
• The reasoning behind that description
• An LLM confidence score
• The interpreted flow
• The underlying evidence
• Repeated behavior across tests
• Raw event data

The system then used those interpreted behaviors to produce a higher-level summary of the test suite.

In the demonstration, 11 tests ultimately represented six unique behaviors. Expected behaviors could then be entered and compared against those observed behaviors to determine whether they were covered, partially covered, or not observed.

The resulting information could also be exported as a report or as structured JSON for further processing.

Why This Could Matter for Enterprise QA Teams

One of the most interesting parts of the meetup was the discussion around escaped defects.

Dave described conversations with large enterprises where a production defect can trigger a significant manual investigation. A team may have to assemble a “tiger team,” examine the release, review test data, and manually determine what was tested and how the defect managed to escape.

That’s a particularly compelling use case for this approach.

Rather than starting an investigation with a large collection of test artifacts and asking someone to reconstruct what happened, a system that has already captured and interpreted test behavior could provide an immediate starting point:

Here is what the tests actually exercised.

That doesn’t necessarily explain why a defect escaped, nor does it replace traditional assertions or other forms of testing. But it could help teams quickly identify what was and wasn’t covered and focus their investigation where it matters.

Beyond Automated Tests

The discussion also raised an intriguing possibility: applying the same concept to real-world application usage.

If browser events can tell us what an automated test did, similar event data could potentially tell us what an actual customer did.

That could open the door to applications such as:

• Production bug investigation
• Customer issue triage
• Understanding real-world user journeys
• Identifying gaps between expected and actual usage
• Using production behavior to inform future test coverage

In this model, the technology begins to look less like traditional test reporting and more like a form of application observability focused on behavior and intent.

An Idea That’s Still Evolving

Dave was clear that the project is currently a reference implementation and thought experiment rather than a production-ready blueprint. There are important questions still to answer.

For example, the current approach focuses primarily on observable browser interactions. It does not yet capture everything that might be relevant to a test, such as all page content, data that wasn’t directly interacted with, assertions, or the ultimate pass/fail result.

There are also technical considerations around complex web applications, browser differences, domain-specific terminology, and how to interpret behaviors consistently across organizations.

Those limitations are part of the value of the project, however. By making the approach open source, Dave is inviting others to experiment with it, challenge its assumptions, and help determine where the concept can provide real value.

The Bigger Question

The most important takeaway from the meetup may not have been the particular tool or implementation Dave demonstrated. It was the shift in perspective.

Modern QA teams have increasingly sophisticated automation, CI pipelines, coverage metrics, AI tools, and observability platforms. But all of those tools still need to answer a fundamental question:

Are we testing the things that matter?

As software becomes more complex, and as AI accelerates the pace at which software and tests are created, that question may become even more important.

“What did you test?” provides a simple way to bring the conversation back to behavior, customer value, and business outcomes. Instead of measuring testing solely by the amount of code exercised or the number of tests that passed, teams can begin asking whether their tests provide meaningful evidence that the software does what it is supposed to do.

That is a question worth asking, especially before the next production defect forces us to ask it the hard way.

About NOVA SART

Solution Street is proud to host technical meetups through NOVA SART, bringing together software professionals from across Northern Virginia and the broader technology community to share ideas, experiences, and practical approaches to software development, testing, and engineering.

We appreciate everyone who joined us in Herndon for “What Did You Test?” and especially Dave for sharing his work and engaging the group in a lively discussion about the future of software testing.  Watch the Meetup recording to hear the conversation and explore the ideas discussed during the session.

Interested in joining us for future NOVA SART events? Join our NOVA SART Meetup group to stay up to date on upcoming meetups, presentations, and opportunities to connect with other technology professionals in the Northern Virginia community.