Training a single model is no longer an art. A weekend, a few tutorials, a bit of GPU time. The problem starts afterwards.
Why so many in the first place
We keep around 50 micro-models up to date. Not because we collect models. But because one big model for everything is, in practice, worse than many small ones for one task each.
One model maps companies to categories. One recognises objects. One pre-sorts documents. One checks whether an answer fits the context. Each is small, fast, cheap to run, and good at exactly one thing.
The advantage: when one gets worse, you immediately know which. The disadvantage: you have fifty things that can get worse.
The calculation nobody does
Compute is not the problem. We ran the numbers.
600 to 1,200 GPU hours a year, a few thousand dollars. Manageable. Nobody argues about that.
But here comes the part nobody calculates: every model needs fresh data every month. Every model needs an evaluation after training. With fifty models that is a hundred datasets and a hundred evaluations per month.
One question, a hundred times a month
And every one of those hundred evaluations asks the same question: is the new version really better than the old one?
That sounds simple. It isn't. "Better" means: better on the cases that matter to us. Not better on average and worse on the rare categories. Not better on the easy examples and worse on the hard ones. And not better because a different test set happened to be in there.
We've seen it. After a few months nobody looks closely anymore. The models get retrained, the numbers look roughly the same, move on. Until one day a model in production produces nonsense and nobody knows since when.
What has to be automated
So we stopped doing it by hand. Not because we're lazy, but because humans are the wrong tool for this kind of work.
The test set is fixed and only changed deliberately. The comparison runs automatically, per category, not just on average. A new version only goes to production if it hasn't got worse on the critical cases. And every decision is recorded: which data, which numbers, which result.
The human only looks where the pipeline reports something unusual. That's maybe five cases a month instead of a hundred. And on those five, they really look.
That is exactly what we built NeoCoder for. First for ourselves. Because otherwise we would not have kept the fifty models going.
What this means for others
Many companies are at model number one or two today. Manual work still functions there. Fact is: anyone who uses AI seriously ends up at ten or twenty models faster than they think.
And then the question is no longer whether you can train. It's whether you can afford to look a hundred times a month. My answer: no. Nobody can.
One model is a project. Fifty models are an operation. And an operation without automation is only a matter of time until something important gets missed.
Written by me. The thoughts, the values, the learnings, the mistakes: all mine. Grammar and spelling are corrected by our own twin model, trained on my texts. Sometimes a stumble stays in. That is mine too.