n.V6-3.04 | When The Machine Is Confidently Wrong
01| Some of what the platform decides will be decided by a model. Which projects get surfaced to which buyers. Which work gets flagged for quality review. What a contribution was worth. These are not recommendations about films. They determine whether a person in a poor neighbourhood gets paid this month.
02| Three failure modes, and none of them announce themselves.
03| Drift. A model is accurate against the world it was trained on. The world moves. A model that was right nine times in ten at launch can be right seven times in ten a year later, and nothing in the interface will look different. The output stays confident. Confidence is not a signal — it is a rendering choice.
04| Bias. A model trained on past outcomes learns past patterns, including the ones we would never write into a rule. If certain Maveriqs, or certain barrios, or certain kinds of project get systematically scored lower, the system will be doing quietly and at scale precisely what the whole platform exists to fight. Saying we oppose this is worthless without measurement. If we are not monitoring outcomes across groups, we are not against bias; we are just unaware of ours.
05| Opacity. A model that cannot explain itself is fine for a suggestion and unacceptable for a refusal. "The system rejected your project" is not a sentence anyone can argue with, learn from, or appeal. It is the algorithmic version of a closed window at a government office, and our users have had enough of those.
06| The line. Any decision that costs a Maveriq money or standing carries a human name and a reason a person can read. The machine can propose, sort, rank, draft and flag. It does not deliver the no. When it is used in a decision, we keep the record of which model, which version, which inputs, so the decision can be reconstructed later — because the request to reconstruct one will come, and "we don't know why it did that" is not an answer we are willing to give a person whose income depended on it.
07| What it costs us. Everything above caps automation, and automation is the mechanism that lets this platform scale at low cost. Every decision routed through a person is slower and more expensive than one that is not, and this rule puts human beings in the loop at exactly the highest-volume points. The ambition of running a large platform with a small staff and the ambition of never letting a machine silently deny somebody are in direct tension, and I am not going to pretend the tension resolves neatly.
08| It gets settled the same way as the rest of the volume: friendliness and trust beat maximum technical power. We will automate hard everywhere the downside of a wrong answer is an inconvenience, and we will keep a person in the chair everywhere the downside is somebody's rent. If that means our cost curve improves more slowly than a purer tech company's, that is the price of the users we chose.
