AI wrote your tests and they all pass. Do they prove anything? Unit vs integration tests

Cipher here. I read software the way an attacker does: I look for the check that was never really checked. Green test runs make me nervous when I know who wrote the tests. If an AI wrote the code and then wrote the tests for that code, who tested the AI?

What do unit, integration and end-to-end tests each catch?

A unit test runs one small piece alone and catches wrong logic. An integration test runs a few pieces together, or your code plus a real database, and catches wiring bugs. An end-to-end test runs the whole product like a user and catches broken flows, but it is slow and its failures are hard to trace.

Google's testing blog puts the trade-off in three words: a good feedback loop is "fast", "reliable" and "isolates failures". Unit tests have all three. In the same post, one tenth of a second "is considered slow for unit tests".

Unit tests have one blind spot, and Google names it: "even if the units work well in isolation, you do not know if they work well together." An integration test "takes a small group of units, often two units, and tests their behavior as a whole."

Martin Fowler's longer article is more specific about integration tests. They test your application "with all the parts that live outside of your application": databases, filesystems, network calls to other services. His example is plain. Start a database, connect your app, run a function that writes data, then read the data back and check it.

He also gives one rule worth keeping: "Write integration tests for all pieces of code where you either serialize or deserialize data." That covers saving a record, reading an API response and parsing a file. In an AI-built app, those are the places where two pieces most often disagree about a field name.

How many of each should you write?

Google suggests 70% unit, 20% integration and 10% end-to-end as "a good first guess", and says "the exact mix will be different for each team." The shape is a pyramid: many small fast tests at the bottom, a few slow broad ones on top.

Fowler's original pyramid note says the same: "many more low-level UnitTests than high level BroadStackTests running through a GUI." He adds a fair caveat in a footnote. The pyramid assumes broad tests are slow and brittle. "If my high level tests are fast, reliable, and cheap to modify - then lower-level tests aren't needed."

Google also names two shapes to avoid. The ice cream cone has mostly end-to-end tests and few unit tests. The hourglass has many unit tests and many end-to-end tests with almost nothing in the middle. The hourglass is the one I would watch for in AI-written suites: lots of tiny tests, plus a flow test or two, and no test that touches the database.

One more habit from Fowler helps. If a high-level test finds a bug, "you also have a missing or incorrect unit test." Write a unit test that reproduces the bug before you fix it, and the bug stays dead.

Why do AI-written tests that mirror the code prove little?

A test written by reading the code under test usually expects what the code does, not what it should do. If the code has a mistake, the test copies the mistake and passes. This is my reasoning, not a measurement: I did not run an AI tool, and I found no study that measures it.

Here is a small illustration. The rule is "free shipping at $50 or more". The code has an off-by-one:

function shippingFee(subtotal) {
  if (subtotal > 50) return 0; // bug: the rule says 50 or more
  return 5.99;
}

A test that mirrors the code computes its expected value from the same logic:

test("shippingFee", () => {
  for (const subtotal of [10, 50, 100]) {
    expect(shippingFee(subtotal)).toBe(subtotal > 50 ? 0 : 5.99);
  }
});

It runs every line. It passes. It also cannot fail, because the "expected" value is the code. A test from the requirement writes the answer by hand:

test("free shipping starts at exactly $50", () => {
  expect(shippingFee(49.99)).toBe(5.99);
  expect(shippingFee(50)).toBe(0); // fails against the buggy code
});

The second test fails against the buggy code, which is its job. The difference is where the expected number came from: the code, or the rule.

This is not a new idea. Google's "Test Behavior, Not Implementation" says tests should focus on the public API and not on implementation details. Fowler's practical article says to test observable behavior: "if I enter values x and y, will the result be z?", not whether the method calls one class and then another. Tests tied to the implementation break on every refactor and, as the example shows, cannot disagree with it.

For the reader, the fix is simple. Ask the AI to write tests from your written rules, with example values you chose. Tell it not to read the implementation first. Then read the expected values yourself. If you cannot read code, here is how to check what an AI changed.

What does coverage actually tell you?

Coverage tells you which lines your tests ran. It does not tell you whether any test checked the result. Fowler calls it "a useful tool for finding untested parts of a codebase" and "of little use as a numeric statement of how good your tests are."

Google's coverage post is blunter: "Code coverage does not guarantee that the covered lines or branches have been tested correctly, it just guarantees that they have been executed by a test." PIT's docs say traditional coverage "measures only which code is executed by your tests" and "does not check that your tests are actually able to detect faults."

The extreme case is a test with no assertions. Fowler calls it AssertionFreeTesting, and warns that "high coverage numbers are too easy to reach with low quality testing." An AI told to raise coverage to 100% can do exactly that, honestly, with tests that check little.

A low number is still useful. Google says "a low code coverage number does guarantee that large areas of the product are going completely untested." So use coverage to find gaps, not to prove quality. On targets, Google's guideline is 60% "acceptable", 75% "commendable" and 90% "exemplary", and it says there is no ideal number. Fowler would "be suspicious of anything like 100%".

The number is a hint. Fowler's own test for "enough testing" has no percentage in it: you rarely get bugs in production, and you are rarely afraid to change code.

What is mutation testing, and what is a mutation score?

Mutation testing plants small bugs in your code, one at a time, and runs your tests against each. A mutation score is the percentage of planted bugs your tests caught. It measures whether your tests would notice a real change, which coverage cannot.

Stryker's docs say it directly: bugs, or mutants, "are automatically inserted into your production code. Your tests are run for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived." PIT describes the same loop and says the percentage of mutants killed gauges test quality.

The score has a formula, from Stryker's FAQ: % Mutation score = # Total detected / # Total mutants * 100, where detected means killed plus timed out. Survived mutants and mutants on code with no tests count against you. Mutants that caused an error are left out.

Stryker's own example is an age check, return user.age >= 18. It generates mutants such as > 18, < 18, return false and return true. Here is the arithmetic for two tests, age 30 allowed and age 10 refused. This is my own hand calculation, not a tool run:

The tests run every line, so coverage reads 100%. Three mutants are caught. The > 18 mutant survives, because no test uses exactly 18. That is 3 of 4, or 75%. Add a test for age 18 and the score is 4 of 4.

Notice what survived: a boundary. Age gates, rate limits, price thresholds and permission checks all live on boundaries, and they are where an off-by-one hurts most. A surviving mutant is a bug an attacker would be glad to find first.

Does mutation testing find real bugs?

Google tested this on its own code. It reported that in around 70% of cases a bug was coupled with a mutant, meaning a mutant touched the same code as the real bug. That is one company's codebase, not a law. Google also says mutants "are simpler than bugs found in the wild".

The post gives more detail. The experiment ran 33 million test suites. In more than 90% of cases, either all mutants on a line were killed or none was, so Google's tool reports at most one mutant per line. Early on, developers rated about 80% of the reported mutants "not useful". Tuning cut that to around 15%.

Two cautions come with this. First, some mutants change nothing a test could see. Google's example is removing a cache lookup, which gives "functionally equivalent code". PIT warns about the same thing and skips mutants on lines that call common logging frameworks. A surviving mutant is a question, not a verdict.

Second, mutation testing is slow, because it reruns tests many times. Stryker's FAQ opens with how to speed it up. Run it on the code that matters, such as money, access control and limits, not the whole app.

Google's own tool runs on code changes during review. Google's engineer also reports that projects using it "get more tests over time". The idea has reached AI tooling too. Meta's paper describes a system that generates mutants, then asks an LLM to write tests that kill them. In its abstract, engineers accepted 73% of the tests in two test-a-thons. I read only the abstract, so I cite it for what Meta says it built, nothing more.

How do you use this on an AI-built app?

Write the rules in plain sentences, have the tests come from those rules, add one integration test where data crosses a boundary, and run mutation testing on the few modules that guard money or access. Use coverage only to find code no test touches.

Stryker covers JavaScript and TypeScript, C# and Scala. PIT covers Java and the JVM. For other languages, search for a mutation tool for your stack and check its docs. Do not take my word that one exists.

If you build with a tool like Claude Code, the same rule applies: the tests are only as independent as their source. For the security side, see the 6 security checks before you share a vibe-coded app. If your tests touch a database, what a query is and what an API is are the pieces an integration test connects.