Ok, a question I hear often: a model is a black box, isn't it? You write something in at the front, you read what comes out at the back. For a model behind an interface that is true. For an open model on your own machine it never was. Whoever has the file can look inside. The only question is how to find anything among a few billion numbers: what is inside such a model that nobody told us about? We are investigating that right now with a deliberately odd test question: was this model trained on a game? And if so, which one? This is an attempt, not a result. But the why is simple: we want to make a model better without training it.
The question
Anyone who downloads an open model, for example one from the Qwen family, gets a file with a few billion numbers and a description that says little about the training data. That is the case with almost all of them. The researchers who studied this systematically put it dryly: the models are deployed everywhere, what they were trained on is rarely disclosed.
For us this is not an academic question. We build agents on these models and train them further. If I do not know what the model has already learned, I train blind: either things it has long been able to do, or against something that sits deep inside. The game is only the touchstone. A game has clear rules and a clear state. If there is anywhere you can cleanly demonstrate what a model has learned, it is there.
Two ways in: listening and X-raying
There are, simply put, two ways.
From outside, listening. You give the model a text and measure how surprised it is. A model that saw a text in training finds hardly a word of it improbable. A model that does not know it stumbles over a few outliers. A 2023 paper turned that into a method that needs no knowledge of the training data, and used it among other things to detect copyrighted books in models. All you need are the probabilities the model outputs.
From inside, X-raying. Here it gets more precise, and one term gets mixed up in everyday talk, in our own conversations too, so once cleanly: a model's weights are fixed after training. What moves with every request are the activations: what happens inside the model while it reads and answers. With an open model you can read those out. And then you can train a small probe on them that answers a single question: is a particular piece of information contained in these numbers, yes or no?
The model that had a board in its head
The paper that opened the subject up for language models is from 2022. Researchers trained a language model on nothing but move sequences of the board game Othello. No board, no rules, only text: move, move, move. The task was to predict the next legal move. Then they looked into the activations with a probe. And found the state of the board there. From pure move sequences the model had built itself an internal representation of the playing field that nobody had given it.
They did not stop at looking. They intervened in the activations, changed the internal board state, and the model then proposed moves that fitted the changed board. That is the difference between a hunch and a proof: if you turn the dial at that spot and the behaviour follows, you have found the right spot. A second paper a year later showed that this representation is even simpler than thought: the model does not think in black and white, but in "my colour" and "the opponent's".
And then chess, 2024, and this is where it becomes practical for us. A model trained only on real games, only on the next character. Again the board was found inside. But in addition something nobody had expected: the model internally estimated how good the players are, because that let it predict the next move better. The researcher read out that direction, amplified it in the model at query time, and the win rate rose by up to 2.6 times. Without a single new training step.
What we are trying
Our attempt turns the research procedure around. There they knew what the model was trained on and looked for the trace. We do not know the training and look for whether there is a trace.
And we do not do it by hand. A second model plays the examiner. This is how we proceed:
Hypotheses. The examiner draws up a list: which games are candidates, and how would you recognise each? Notation, typical openings, moves only someone who knows the board would make.
Test cases. For each hypothesis, positions that exist nowhere on the web. This is the most important step, more on it in a moment.
Measure. The model under examination gets the positions, and we record what it does, how confident it is, and what happens inside.
Counter-test. The same measurement with a game we invented ourselves. No model can know that one. What comes out there is the baseline.
Evaluate. The examiner writes a finding with evidence. A human reads it. The examiner proposes, it does not decide.
Where it deceives
Everyone who evaluates models knows the biggest trap, and here it is especially treacherous. A model can know a game because it has read about it. Chess books, forums, annotated games, all of that is on the web. That is not the same as having been trained on the game. A model that has read a thousand chess books can talk about openings like a connoisseur and still, on move twenty, move a piece that is no longer on the board.
Hence the invented positions. Memorised material does not help there. Whoever correctly carries the board state forward in a position that never existed has represented the game inside. Whoever only talks nicely about it has read about it.
Second trap: a probe sometimes finds something that it learned itself, not the model. Hence the counter-test with the invented game, and hence, when it gets serious, the intervention as in the Othello paper: turn the dial at that spot and see whether the behaviour follows. Third trap: the examiner is itself a model. It can be wrong, and it can be wrong with conviction. I wrote about exactly that last week, when our nightly council mistook agreement for stubbornness.
What it is for: optimising without training
Not for the games. For what comes after. If the method works on a game, where you can check it cleanly, then it also works for the questions that really occupy us in NeoCoder. Does this model really know our framework, or does it only talk about it? Has it represented our architecture rules inside, or does it keep them only as long as they are in the prompt? Where is the gap between what it knows and what it needs?
The real reason is this: training is expensive. Not only in dollars. A run needs clean data, days of curation, a check afterwards, and in the end a new model that has to be measured against the old one again. I have written enough in recent weeks about how much work goes into a single training pair. Hence the question that drives us: how much can we get out of a model before we touch it at all?
Choose the fitting model. For a task you take the one that already contains the most, not the biggest. That costs a measurement, not a run.
Tailor the query to the model. If you know in what form a model learned something, you ask it in that form. The same task, put differently, hits the spot inside where the knowledge sits. That is more than prompt tinkering: you do not try, you measure.
Amplify what is already there. The chess paper shows it: the playing strength was in the model, it only had to be turned up at query time. That is what I would try next, if the first step holds.
And only then train, and only the gap. What is still missing after the first three steps is the real gap. That is what we train, not the whole field. That is the difference between a run overnight and one over the weekend.
And there is one place where we already use the technique: in the nightly deliberations among the agents. Last week I described how a council argues over our learning material at night, an advocate, an examiner, and a referee at a stalemate. Each of these roles runs on a model, and each role demands something different: the examiner should find weaknesses, the advocate should make the answer as strong as it is. If we know what a model brings along inside, we can give each agent its own profile and sharpen it night after night. Which model suits which role? In what form does the case have to be put to it so that it shows its best? Where is it blind? We no longer answer that by feel, but from the measurement. And the council delivers new cases every night against which we check whether the profile has improved.
The flip side
What you can read, you can check. That is the good news, and a 2025 paper took it to the extreme: researchers deliberately trained a hidden objective into a model and set four teams on it who knew nothing about it. Three found it. So model auditing is possible.
But what you can amplify, someone else can change. With an open model nobody sees from outside whether the activations are being turned while it answers. That is the same lesson as the Cursor incident two weeks ago, only one layer deeper: not only the environment around the model belongs to the attack surface, the path through the model does too. That is why everything runs locally with us, with verified files, and why the X-raying belongs in the same workshop as the building.
Where we stand today
Plainly: on the game question we do not yet have a result that I would write down here as a finding. We have a method, a counter-test, and first measurements that I cannot yet interpret. Whether a game trace can be found as cleanly in a large general-purpose model as in the small models of the research, I do not know. It may well be that the answer is: not with our means. Then I will write that down too.
What remains
A model is no longer a black box once you have the file on your own machine. You can listen in, you can look in, and research has shown that there are things in there nobody wrote in. We are trying to make a tool out of that: not to find games, but to know what a model can do before we teach it anything.
The question was: did this model learn a game, and which one? We do not have the answer yet. But the question behind it is the right one: what is already in there? Whoever knows that chooses the fitting model, asks it the way it learned, amplifies what is already there, and in the end trains only the gap. Optimising without training. And whoever does not know trains blind. That is the expensive way.
Sources
Othello. Li, Hopkins, Bau, Viégas, Pfister, Wattenberg, Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task, 2022. Internal representation of the board, demonstrated by intervention. Also Nanda, Lee, Wattenberg, Emergent Linear Representations in World Models of Self-Supervised Sequence Models, 2023: "my colour" versus "opponent's colour".
Chess. Karvonen, Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models, 2024. Board state and player skill represented inside; amplifying the skill direction raises the win rate by up to 2.6 times.
Detecting training data from outside. Shi et al., Detecting Pretraining Data from Large Language Models, 2023. The Min-K% Prob method, applied among other things to copyrighted books.
Model auditing. Marks et al., Auditing language models for hidden objectives, 2025. Four teams, three find the hidden objective.
Own articles. Simply put: frontier AI (how a model learns), At night the platform deliberates with itself, The model wasn't hacked, Who owns the code.
Status and caveat. 13 November 2026. This is an attempt with an open outcome. The cited papers examine small models trained specifically on a game; whether that carries over to large general-purpose models is not established and is part of our question. Evidence that a model can do something is not evidence of what it was trained on. I will add results here, including negative ones.
Written by me. The thoughts, the values, the learnings, the mistakes: all mine. Grammar and spelling are corrected by a twin model of my own, trained on my texts. Sometimes that turns into a mix. Of course I read everything again, but nobody is infallible. Whatever stays in is mine too.