There's a standard story about algorithmic fairness. A company trains a model. A reporter runs a demographic analysis. The model is biased. A press release about responsible AI is issued. Internal committees form. The model is adjusted.
This story positions fairness as an ethics problem — something that lives in the space between legal, PR, and product. A question for committees, not compilers.
I think this framing is wrong, and it's making both the ethics and the engineering worse.
Once an organization decides that their model should be fair, they're immediately confronted with a set of questions that are purely technical:
These aren't policy gaps. They're specification errors. The kind of errors that show up in code reviews, not ethics reviews.
Here's a framing I've found useful: fairness metrics are interface contracts between a model and the world it operates in.
When you ship a loan model with a stated demographic parity constraint of ≤5% disparity, you've made a contractual claim. Like any API contract, it can be:
This is exactly how we think about correctness in other systems. A sorting algorithm that works on the test cases but breaks on empty arrays isn't "mostly correct." A fair model that holds on average demographics but fails on rural zip codes isn't "mostly fair."
Over the past year working on FairLint DL, I've started to see a pattern in what separates teams that actually ship equitable systems from teams that just have an AI Ethics page.
They instrument fairness like they instrument latency. P99 latency isn't the only number in the monitoring dashboard — neither should aggregate accuracy be. Demographic breakdowns of error rates, per-slice calibration curves, temporal drift by group — these belong in your observability stack, not in a quarterly audit doc.
They version their fairness contracts. When the model changes, someone checks whether the demographic parity constraint still holds. This sounds obvious. Almost nobody does it.
They treat counterfactual generation as a debugging tool. If you want to understand why a model is unfair — not just that it is — you need to be able to ask: "What would this prediction have been if only the protected attribute changed?" That requires the ability to generate plausible counterfactuals, which is a technical capability, not a policy one.
The reason we reach for ethics committees and not engineers when fairness breaks is partly cultural — we've built a story where bias is a values failure. But bias that makes it to production is usually a systems failure: a missing test, a monitoring gap, an implicit assumption in a feature engineering pipeline.
I'm not saying ethics doesn't matter. It matters enormously — the choice of which fairness definition to optimize for is fundamentally normative. But once you've made that choice, executing on it is engineering. And treating it as engineering means it gets the same rigor: code review, continuous integration, regression testing, on-call rotations.
The committee can decide what "fair" means. The engineers have to build it.
Working on tooling for this exact problem? FairLint DL is a VS Code extension that runs real-time fairness audits on your ML pipeline — counterfactual generation included.