Should engineering organizations have quality assurance (QA)? A lot of engineering leaders say they should not. I review the topic, and provide some suggestions for QA leaders, and some nuance on the problems with the way QA and engineering work together. 🌶️ ahead!

QA should not exist

In some engineering leadership circles, the answer is “of course not”. QA is largely seen as an old fashioned practice: “Engineering should own quality”.

I tend to be skeptical of this type of engineering hubris. But there are some good arguments for this:

• QA slows things down. There is too much back and forth when you’re passing work in a gated process. QA finds an issue, dev fixes it. Then QA rechecks it and finds other issues. This continues forever, greatly slowing down engineering velocity.
• Throw it over the wall. There is a moral hazard in having other people responsible for whether your work is correct. You’re creating bad behavior for your engineers by not having them responsible for their quality.
• Incentives are poor for QA. If your job is to find issues, you’re going to find issues. This automatically slows down engineering, regardless of whether the issues are important or not.

QA is often compared to Operations. Having engineers write code and operators put it in production and care for it has similar problems: handoffs and incentive problems.

In a lot of modern and startup environments, engineering leaders view that as a harmful practice. And they’re often dismissive of the value of QA as a result.

QA should definitely exist

Other leaders are firmly in support of QA.

They argue:

• Testing is a skill. Looking for issues is a real skill, and having an adversarial approach does result in finding deeper issues. Engineers sometimes exhibit an arrogance that they can do everyone else’s job, but the truth is this is a real skill, and not one that is easy for developers to pick up.
• Automed tests are valuable. Automated tests, when embedded in your workflows, are incredibly valuable. And they can be done by people who are less expensive than engineers, making their work high leverage.
• Some situations absolutely require QA. There are some high stakes situations where a skilled expert can be worth their weight in gold. They can save the company from considerable risk and ensure a better product experience.

I find this attitude more often in traditional companies, but I see it in a lot of places.

Perhaps the strongest argument nowadays for QA is that with AI, automated verification is a leverage maximizer.

The testing pyramid

Before I weigh in further, I’d like to make sure you’re familiar with the testing pyramid. The basic ideas is that there is a natural pyramid of value with testing. The base is unit tests, which are fast and deterministic. Then above that you have integration tests, and then above that, slower UI tests. You want to have lots of fast unit tests. And fewer integration tests, and fewer still UI tests.

Everyone agrees that the engineers should be responsible for unit tests and integration tests. And the UI tests are up for debate. In a lot of organizations, that’s where the split is: QA owns end to end UI testing, and is testing it from the end user perspective. Other organizations share ownership of the end to end tests.

My (slightly) more nuanced take?

I polled my engineering leadership peers, and 100% of a sample of 10 people said, “You mostly should not have QA on software teams, with some edge cases and exceptions.”

I mostly agree with that, but I do have some nuance I’d like to add, and some ideas on ways that QA might be made more high leverage. And I have some advice for QA leaders to make sure they’re as valuable as possible in their companies.

Don’t start with QA

If I am building out an organization from scratch, I wouldn’t start out with QA on a team. Everyone on the team would know that engineering owns quality. Engineering would do unit, integration, and UI tests as a regular part of their work. CI/CD and test automation would be built in from the beginning. Most teams should not start with QA unless they have some very specific reason for doing so.

Don’t have a handoff

I agree that structurally, you don’t want to have a handoff between QA and engineering, if you have QA. It’s much better to have QA “shift left” and be a part of the team itself, doing the work more continually with the engineering team. So my first suggestion is that if you have QA, you embed them on a team and aim to get their work into the same cadence as the engineering team.

Have engineering own quality

If you do have QA, I believe it’s best to still have engineering “own” quality. Metaphorically, I like to treat QA as the quality experts, similar to SREs who are reliability experts. SREs should not be doing all the reliability work, just like QA should not be doing all the quality work. They are experts who can help the team be better at quality, and make sure the team is doing quality work.

Require automation

You almost always want to focus on automated testing. Any manual testing should be for highly unusual situations. I’ve seen it be useful for brittle architectures, or for basically probing for quality issues by a highly experienced QA person.

So if I was building a team from the ground up, and evolving it over time, I’d lean hard on automation and the team being customer focused, and try to solve those problems before reaching for QA.

When would I reach for QA?

There are a few times where QA is high leverage. I’ve seen brittle architectures that are hard to test in an automated way. (Again, I would try to automate first, but this could be a backup plan while we fix the architecture).

I’ve also seen very skilled QA people be deployed as a strategic roaming asset within the company, to focus on the most sensitive projects. I remember working with a couple of people in the past who were just amazing at making sure projects were successful and were shipping with quality. Having roaming QA can be a good pattern, because it can only operate within a larger engineering culture of engineering being responsible for the quality of their own work.

I also think there is a lot of untapped potential with QA. The space is changing rapidly with new AI based tools, and I’ll talk below about some potential ways you might tap QA for high leverage benefits.

What about when there is already a QA team?

I often join teams that have an existing QA team. What do I do when I join a situation like that?

Sometimes it’s not the top priority

First of all, it’s often not my top priority. If there are other higher priorities, I’ll focus on them first. But I can usually improve things a bit with just a few nudges, that don’t require a lot of extensive work. I’ll often meet with the Head of QA and talk these over as goals:

Shift QA left

• Get rid of handoff between dev and QA.
• Have QA members embedded on the team and attend meetings.
• Try to get the workflows of the QA people onto pull requests (PRs) rather than later in the process.
• Emphasize that engineering owns quality, but QA are quality experts. Engineers can do QA tasks as well. Often QA can help with test plans that engineers can be involved in doing. Engineers can also write test plans and have QA review. Less process gates and more working together.
• Share concept of testing pyramid, and make sure engineers are writing unit tests, and that PR review requires unit tests. Ironically, I find that most companies that have had QA for a while also don’t have a good unit test suite. I’m not sure why that is, but I think it’s because engineering thinks QA is there as a safety net.
• One challenge is with QA teams in different time zones, or where they are offsite and the team is on-site.

Focus on automated testing

• Emphasize that testing needs to be automated. Ask the leader to recommend training or replacing team members to move us to mostly automated tests.
• Make sure the test suite is fast. Make sure we have statistics on how quickly it runs, report on it frequently. Get the QA team to make it faster. Often this means separating parts of the test suite by criticality so that a portion of it can run fast enough, and other parts can run later.
• Make sure the tests are a part of the developer workflow, not only seen by QA.

Metrics

• Unit test coverage
• Unit and UI test run times.

Evaluate the situation

Once the burning fires are put out, improving the way QA operates rises towards the top of the list. But it depends on the company. I’ve seen companies where the problem was so egregious the best solution was to let go of QA. I’ve seen situations where it was working fine, or fine enough that I didn’t care to intervene because I had other priorities.

So my actions at this point depend a lot on the situation. I might invest less in QA. I might pare down QA gradually. I might let QA go completely. Or I might explore ways to make QA more high leverage. I think the last point is the most interesting, because at least in my circles, I haven’t heard people talk about it much. So let’s talk about that.

Introducing the Automated Verification Engineer (AVE)

If I were a QA leader, here’s how I would see the world:

1. I’m in a role that is sometimes treated as old-fashioned.
2. AI is rapidly increasing the leverage of automated verification. Why? If you can improve a coding agent’s ability to detect failures automatically by 50%¹, you’re speeding up the company something like 2x, because a much larger percentage of its changes will be able to be accepted. Automated verification means your work is speeding up the company AND improving quality, instead of slowing it down.
3. Automated verification is something my team can help with.
4. I’m going to need to market this differently than QA. I would probably create a new term like “Automated Verification Engineer” (AVE) and a new set of responsibilities. That way I can redefine expectations of the work my team does.

Then, I’d embark on a somewhat experimental approach to redefine how we work together. Here’s an example of how it might look, an Automated Verification Engineer.

¹ (This number is just a guestimate, not a real mathematical calculation. I think the value actually would increase nonlinearly as you got more automation, because of the increased confidence you haven’t broken something.)

What is an Automated Verification Engineer?

• I’d view the role as a sort of cross between a Developer Experience Engineer and Quality Assurance.
• The AVE would be focused on making sure the developer pipeline gets great feedback early on to customer problems introduced by code.
• The AVE’s tests are mostly for AI agents. The aim is to help the agents realize when their hallucinations and bad changes aren’t valid.
• All tests are automated, deterministic, non-flaky, and as fast as possible.

Metrics for an AVE:

1. The goal is fast feedback for the engineering team. So I’d use “time to feedback” as one of the most important metrics for success.
2. You could instrument your test suite, and report on the number of tests that fail in an automated testing situation. Each one of those tests was highly valuable (with some caveats).

What is the role of an AVE Leader?

1. Part of your job is to help engineers understand they’re responsible for quality. But also help the organization understand how automated verification engineers work, and how it’s different than QA.

2. I think you need firsthand understanding of how coding tools work. So I’d encourage you to play with them every week.

3. You also need to level up your skills in the testing and deployment lifecycle. I’d partner with the engineering teams responsible for it, study how it works, and volunteer to pair and do work there. You should be able to change the set of things being testing in each PR, or move the tests to run in paralllel.

4. In the age of AI agent development, I believe you should probably be doing a lot more experimentation. Evaluating different tools, looking for things that change the way you do your work.

5. One idea, that I’m not sure is a good idea or terrible idea, is to get more involved in the signals coming back on quality. For example, an AVE Leader could be involved in the bug triage, and work with support on issues on things that have been released. This would give good signals on what is missing from the quality process. Move from being transactional (reviewing work), to try and make the quality better.

6. Another idea is to work with design, to get a sense of what types of design issues can be automatically tested for. Usability and quality are linked.

7. Every month, try to make your tests faster, employing patterns like 1) testing only what has changed, 2) testing locally as you work, 3) breaking tests in parallel, 4) automatically disabling or deleting or ticketing or breaking the build for flaky tests. You should always have projects in flight to be testing out various schemes of speeding up tests.

8. The company should have a “testing pyramid” approach, with unit tests at the bottom as a foundation, then integration and UI tests above that. There are other tests to consider as well, but generally you want more unit tests than UI tests.

Some of these responsibilities might already be covered well in your organization, and this is just a sketch of how it could work, not a job description. So modify it for your company.

Automated Verification Engineer is experimental

I’ll add that this is all experimental. It might even be a bad idea. But if I were a quality leader, I’d be doing some sort of experiments along these lines.

Disrespecting QA?

I’m more nervous about this post than most, because I think it risks making people feel like their role isn’t important. I’d like to emphasize that it isn’t about the people themselves, but the role that is the problem. In the same way that Operations was a problematic role in many organizations, I believe QA is not often set up well. I think a combination of respect for the people involved, and some experimentation in the role might yield some big results in the age of agent based AI development. But the bottom line is that working more continuously, more within the same workstreams, typically yields much better results than passing things on in a transactional, assembly line process.

It would be a lot easier to say “no, just don’t do QA”, and honestly for a lot of places that makes sense. Or to say, “yes, let’s just keep QA the way it is”. But a lot of the challenge is to experiment with the workflows, and see if you can make QA a high leverage point for the company. If you do manage to make automated verification exceptional, you’re taking your role from a cost center to a point of differentiation for the company.

If you have any feedback on this post, let me know. I imagine my thinking will evolve, and I’d like to hear your perspective.

Also, if you do end up experimenting with these ideas, let me know. I’d like to see what worked and what didn’t!

Thank you

Bruce Vaughn reviewed the post and provided feedback.

Image by John from Pixabay