Trust Is Produced, Not Felt
Most teams treat trust as something that settles over a codebase — a feeling that builds up after enough green pipelines and enough uneventful releases. That is not trust. That is familiarity, and it feels identical right up to the morning it doesn't.
Last week I gave you four questions for telling a real quality gate from a decorative one. Can it block? Does it fail for reasons a human can argue with? Is it independent of the thing it is judging? Does someone own the consequence when it passes something it should not have?
Those questions work. Ask them honestly about your own pipeline and you will find at least one gate that has been waving traffic through for a year. But there is a harder problem sitting underneath them, and it is the one I want to spend this post on. Gates catch things that fail. The failure modes that do the most damage at machine speed do not fail. They accumulate. Every check stays green while the ground the checks are standing on quietly stops being true.
Trust is a product, not a mood
Most teams treat trust as something that settles over a codebase — a feeling that builds up after enough green pipelines and enough uneventful releases. That is not trust. That is familiarity, and it feels identical right up to the morning it doesn't.
Trust is a manufactured product. It has inputs, a production process, and a shelf life. If you are not producing it deliberately, you are not accumulating it.
This used to be easy to miss, because trust arrived free as a byproduct. When a human wrote every line, the writing itself produced understanding: somebody in the building knew why the code looked the way it did, what had been tried and rejected, which constraint the odd-looking branch was there to protect. Nobody scheduled that work. It fell out of the work that was already happening.
It does not fall out of the work anymore. When generation runs ahead of comprehension, understanding stops being a byproduct and becomes a line item. You either pay for it or you go without it, and going without it stays invisible for a surprisingly long time. Here are the four ways it goes missing.
1. Plausibility drift
The first failure mode is the one the other three grow out of. A model's output is optimised to look like a correct answer. Looking like a correct answer is also most of what a reviewer actually checks under time pressure. Those two facts meet in your pull request queue.
Individually, each plausible-but-slightly-wrong artifact is cheap. A helper that handles the common case and quietly mishandles the empty one. A config that is right for staging and wrong for the region you launch in next quarter. Nothing breaks. It goes in.
The cost is not the artifact. It is that each accepted artifact moves the baseline. The next generation is conditioned on the last, review calibrates to whatever has been passing, and "how we do it here" is now partly composed of decisions nobody actually made.
Drift is not a bug you find. It is a direction you eventually notice you have been travelling in. The counter-measure is provenance. For any consequential change you should be able to answer three questions: what was this generated from, what was it checked against, and who accepted it. Not to assign blame — to make drift visible while correcting it is still cheap.
2. Surface inconsistency
The second is what drift looks like once it has scale. Individually correct components can be collectively incoherent. Two modules handle errors differently. Three services disagree about whether a timestamp is UTC. The same domain concept is a customer in one place, an account in another and a party in a third, and every one of them is defensible inside its own file.
No test fails, because no test asserts that the system agrees with itself. Consistency was never the responsibility of any individual change. It was maintained by a small number of people holding the whole shape in their heads — a mechanism that does not survive contact with parallel generation.
The counter-measure is making the shape explicit. Interface contracts, naming and error conventions, and the decisions behind them have to live somewhere the generator reads, not somewhere the team remembers. An unwritten convention stops being a convention the moment most of the code is written by something that cannot pick it up in the hallway.
3. Silent regressions
The third is the one that pages you six weeks late. A silent regression is a behaviour change no test noticed, because no test ever encoded that behaviour. This is not a coverage problem in the metric sense. Coverage counts lines executed, not commitments honoured. Plenty of systems sit at 85% coverage with most of their real obligations undocumented: the ordering something downstream depends on, the empty-list response a client special-cases, the retry that has to stay idempotent.
Those obligations lived in someone's head. A refactor that is locally clean and globally destructive walks straight through a green suite. The counter-measure is differential evidence. For a change of any consequence, "the tests pass" is the weakest claim available. The stronger claim is: here is what this system did before, here is what it does now, and here is the difference in behaviour — not just the difference in source. Anything that changed and should not have is the finding.
4. Documentation rot
The fourth used to be a hygiene problem. It is now a supply chain problem. Documentation has always drifted from code. What changed is who reads it. Docs, decision records, comments and READMEs are context — the material a model is handed to ground its work. A stale document is no longer a mild inconvenience for a new joiner who was going to ask a colleague anyway. It is an input that confidently produces wrong output at scale, and that output looks plausible, which puts us back at failure mode one.
The counter-measure is treating context as production infrastructure. If a document is load-bearing for generation, it needs an owner, a review trigger tied to the code it describes, and a way to be marked untrusted when it goes stale. A document nobody is responsible for should not be in the context window.
What the four have in common
None of them fail. That is the entire point. Gates evaluate individual changes against stated criteria. Each of these is a property of the system over time. So a perfectly designed gate — blocking, articulate, independent, owned — can sit on top of a foundation that all four are eroding from underneath, and report green the whole way down.
The four diagnostics from last week are necessary. They are not sufficient. The question to add is this: what is my pipeline actually producing as evidence, and would that evidence survive a hostile reading in six months?
If the honest answer is "green checkmarks", you are not producing trust. You are producing the feeling of it.
This is the argument I develop at length in my chapter, "Quality at Machine Speed: AI Orchestration, and the Future of Trust in Software Quality", in Erik Seversen's The AI Transformation: Thriving Within Civilization's Next Big Disruption.
Next week closes the series. I will walk through what is actually inside the chapter, and introduce the other contributors worth your time — because a multi-author book lives or dies on whether the chapters next to yours are any good, and these are.
This is part 4 of a five-part series on AI orchestration and the future of trust in software quality. Earlier parts: the announcement, The Speed Trap, and A Gate That Works, and a Gate That Only Looks Like One. Subscribe to get the last one as it lands.
