Let me be honest: the problem sounds boring, and it blocks more projects than you would think. A piece of software runs at a large organisation, a public authority, a bank, an insurer. It has to be tested, automated, fast, with real cases. And nobody is allowed to see the real cases.
The problem
At large government organisations, and not only there, you cannot simply put customer data in plain text onto a test system. That is not overcaution, that is the law, and rightly so. So the test team works with data it does not have. It makes up cases, guesses what an address looks like in production, how an amount is formatted, whether a status is written in upper or lower case.
And then the software goes live, and in production the status field says "paid" instead of "PAID", the amount says 89.504 instead of 89.50, the IBAN has spaces in it, and the date is missing in every tenth record. None of that was in the test. Because nobody in the test knew production.
Synthetic is clean. Too clean.
The usual answer is synthetic data. You generate customers that do not exist, with addresses that do not exist, and test with those. That solves the data protection problem. It does not solve the testing problem.
Fact is: synthetic data is generated clean, without errors. Production data has errors. Exactly the errors you would need to find with test cases are not in the synthetic data, because nobody writes them in on purpose. So you test against a world that is tidier than the real one, and then you are surprised in the real one.
The idea: compare hashes, not values
In July 2025 I opened a notebook and started. The idea is simple: you send every field of a record through a one-way function, a hash. "PAID" becomes a long random number. So does 89.50. You do that with the record from production and with the corresponding record from test. Then you only compare the random numbers.
Nobody sees a value. But everyone sees: the status field is different. The date field is missing in test. The amount field deviates. And that is enough for a tester to build a case that hits production. Automated, per record, a thousand times over, without a human ever reading a customer number.
In the first attempt I turned that into images. Each field a pixel, the colour from the hash. Production one image, test one image, and a third with the difference. That looked good. You saw immediately where it pinched. And a language model turned the difference into a sentence for the Jira ticket: "attribute date deleted, amount modified, status upper and lower case changed."
The mistake in the first prototype
And now the part I least like to tell. I looked at the prototype again a few weeks later, with some distance. Three things were wrong. One of them was exactly the mistake the whole thing was supposed to prevent.
The pixel difference says nothing. The hash of 89.50 and the hash of 89.51 are two completely different random numbers. The "total difference 1197" the notebook printed was a random value. Another field would have given 40 or 3000. For the statement you only need: hash equal or hash not equal, per field. The image may stay, as a display. As a measurement it was nonsense.
The diff for the language model came from plain text. I built the hashes, built the images, and then for the model made a diff from the original JSONs. That contained 89.50, PAID and the date, and those went to the API. The prompt did say "do not show values". But the values were already out by then. Exactly what the prototype was supposed to prevent, it did itself in the last step. As I said, looked at with some distance.
SHA-256 without a key is not anonymisation for small value ranges. Status has maybe five possible values. Whoever knows them hashes all five and knows immediately what is in production. Same for date, postcode, yes/no, canton. You need a secret key that lives only in the production system (HMAC). Then nobody can recompute it, not even the one who has the test data.
That is why I leave prototypes lying and look at them again. On the day you build them you do not see them.
What the second attempt does differently
The second attempt hashes with a key, compares only equal or not equal, and adds one thing that makes the argument "production data has errors" really carry: before hashing, it normalises in stages.
Stage 0, raw. The value as it is. Equal? Then the field is identical.
Stage 1, spelling. Trimmed and lower-cased. Equal now? Then the field differs only in upper and lower case or in spaces at the edges. "PAID" versus "paid".
Stage 2, inner spaces. All spaces removed. Equal now? Then it is an IBAN with and without grouping, a phone number with and without gaps.
Stage 3, rounding. As a number rounded to two decimals. Equal now? Then it is 89.504 versus 89.50, a rounding problem, not an amount problem.
Otherwise: really different. And in addition, also without a value: type differs, looks like a date on only one side, empty on one side.
That is the difference from the first attempt. Before, the tester knew: status is different. Now he knows: status differs only in upper and lower case. That is a different test case, a different error class, and he still has not seen a value. The errors that lie dormant in production come out as a class, not as customer data.
What the language model may see
The ticket text now comes without a model first. A table: field, finding, hint. That is usually enough. If you still want the model, because it phrases a sentence for the tester, it gets exclusively this table. Field names, finding classes, hints. No values, no hashes, no original JSONs. And at the end of the notebook there is a print that shows what left the machine. That is not a nicety. That is the test for the test.
Plus the rule I have applied everywhere since: if a prototype anonymises data, then the last line has to prove that nothing else went out. Not the prompt. The code.
One person may look: the person of trust
Up to here the tester knows that status is a spelling problem. He does not know whether that is a software bug, a data error in production, or an expectation that was simply wrong in the test. Only someone who may see the real value knows that. And that person exists in every organisation: one person, often from the business side, with access to production. The person of trust.
The person of trust is not an add-on. They are the part that makes the whole thing hold. This is how it runs: every finding carries a token that can only be resolved to the record inside the production boundary. The test run, or NeoCoder writing the tests, sees field, finding class, token. Nothing more. When a finding class is new, a sample goes to the person of trust, two or three tokens per field and class, not a thousand. They resolve it in production, see the real value, and answer not with the value but with a class and a rule.
Test expectation wrong. "Status is to be compared case-insensitively." The rule goes back, the test is adjusted, a hundred findings disappear.
Data error in production, not a bug. "Amount may only have two decimals, three come from the legacy system." That goes to data maintenance, not into a ticket. And it is the answer to the question I ask every CTO: does anyone know the data quality? Here it is, in numbers, without anyone having seen a customer.
Real bug. "Date is missing in production in around ten percent, the software does not catch that, and the test must not invent a date." That becomes a ticket. With token, without value.
Every resolution is logged: who, when, which token, why. That is what the data protection officer wants to see, and it is cheap if you build it in from the start. And the answer of the person of trust is training material in itself: the rule applies per field and class, not per record. Three samples become one rule for a thousand.
Synthetic, but with the errors from prod
And now back to the synthetic data that was too clean at the beginning. Because you still want it. More data than production provides. Cases that are rare. And the option to mix, if you want to.
The trick is not to generate it clean, but from a profile of production. The profile measures per field, without values: how often the field is missing, how often it is empty, how the lengths are distributed, how often upper case, how often lower, how often spaces at the edges, how often spaces inside, how many decimals, which power of ten. That is statistics, not customer data. And from that you generate records that have exactly these error rates. If you want, you turn the rates up, one and a half times, double, and you have a stress test that is worse than production.
The mix is marked. Every record knows whether it comes from production or is made up. For the made-up ones NeoCoder may look at the value itself, there is nothing to protect. For the real ones it asks the person of trust. And in the run you see both side by side: 213 times IBAN with spaces, 87 of them from production and 126 synthetic. The synthetic ones confirm that the test hits the class. The real ones say that the class really occurs in production.
And the check at the end, again: what left the room? Fingerprints, tokens, profile, findings. Zero plain-text values. On the first run it was two. I had exported minimum and maximum of the amount in the profile. Those are real values. Now it is only the power of ten. The same mistake as in the first prototype, one level further. You make it again and again. That is why the check is in the code.
Word for word: the test case in Xray
So far, so good. But "amount, rounding only, 47 records" is not a ticket. A tester wants to know word for word where the error is, what to do, and what they should see. And only a system that knows the whole system can do that. We wrote it once before, in Code is getting cheap: you never get away from knowing the whole system. Here is the proof, in an unremarkable place.
Because the field "date" is more than the field. Which module writes it. Which writer. Which epic, which story, which sprint, which pull request touched it last. Whether it comes from a form or from an event that passes through three systems. That is the system map, and with us NeoCoder has it, because it knows the epics, all sprints and all tasks that were done. Finding plus system map becomes a test case in words.
I looked at what exists for this. For Jira, Xray is the usual choice, alongside Zephyr and TestRail. Xray has Test, Precondition, Test Set, Test Plan and Test Execution as issue types of their own, a manual test has steps with action, data and expected result, and results are imported via REST as JSON: per test a status and a comment, per step a status and an actual result, plus evidence as attachments. That is exactly what the prototype now produces: per finding class a test issue with steps and an execution result, ready for import. Without a value, with tokens.
The ticket text tells the tester: field date is missing in production, the test expects it. Suspected cause: the software does not catch a missing field. Location: order-flow, written by the event OrderClosed to InvoiceCreated, epic EP-15, story ST-150, last changed in PR 430 in sprint 15. Rule: date is missing in prod in around ten percent, the software has to catch that. That is word for word. With that someone goes and finds the spot in a quarter of an hour instead of a day.
Functional or end-to-end?
And then the question that saves the most time if you answer it before the ticket: can I hand this straight to a developer, or does it first need a tester who reproduces it and describes what happens?
The testing world has distinguished this for decades, component test, integration test, system test, acceptance test, and in practice it comes down to two questions. Is the error in one module, at one field, and deterministically reproducible from the fingerprint diff? Then it is functional. Test and token go straight to the developer, a tester in between would only repeat what the diff already says. Or does the field come into being across several modules or systems, through an event, an order of things, a state? Then it is end-to-end. There a human first has to reproduce the flow and describe what happens, because the diff only says that something is missing at the end, not where along the way.
Functional. Spelling, spaces, rounding, one module. In the run: amount from billing-core, email from customer-master. Straight to the developer, with test case and token. No tester.
End-to-end. A field that comes into being across order-service, event-bus and billing-core. In the run: date. First a tester who reproduces the flow, then the developer, with the description. NeoCoder sets the first step of the test case accordingly: "reproduce the flow, note order and timestamps."
Unclear. Few records, really different, no rule. That is end-to-end too, for a different reason: you do not know what happened. You must not drop that into a developer's backlog. Someone looks at it.
NeoCoder makes the decision from the system map before a human sees the ticket. It is in the title, in square brackets, and in the assignment. That is not much code. It is the information that in most organisations is in no ticket and that every tester first has to work out at the start.
And the GDPR?
I am not a lawyer, and in the end the client's data protection officer signs this off. But three things have to be kept apart, and the design implements them.
The keyed fingerprints are pseudonymisation, not anonymisation. As long as the key exists, they are personal data for whoever holds it. That is not a disadvantage: pseudonymisation is exactly the measure the GDPR recommends, and for whoever does not hold the key, the test system, NeoCoder, the model, the data is practically not attributable. I would still record it internally as pseudonymised, not anonymous. Honest is safer.
The profile and the finding classes are statistics. "Ten percent without a date" has no personal reference if the group is large enough. With five records even a statistic would be a value, so a minimum size belongs in there. The synthetic data derived from it is anonymous, there was never a person behind it. And the language model gets field names and classes, so no personal data at all.
The person of trust is the only place where a human sees a value: inside production, for a documented purpose, logged. That is purpose limitation, data minimisation and accountability in one. What is still missing before a client signs off is paper and operations, not code: the legal basis for the access of the person of trust, an entry in the register of processing activities, and the question of where the key lives and who rotates it.
What stays
The prototype is three modules with a few hundred lines: comparison, person of trust, synthetics, Xray. It does not solve the problem for everyone, but it shows that it works: compare production against test, automated, fast, with the real errors from production, without a human or a model ever seeing a customer value. For many organisations that is the difference between "we test with what we make up" and "we test with what really happens".
And the mistake in the first attempt stays as a reminder. I built a system that protects data, and in the last step I sent the data out. Not because I did not know. But because at the end of the day I quickly wanted to build the nice part, the sentence for the ticket. The nice part is always the one where you stop checking.
Synthetic data is clean, and clean is not what you have to test. Keyed hashes and comparison in stages show how production differs from test without showing a value. A person of trust says what the difference means, with a rule, not with the value. Synthetic data by profile brings the production errors into the test, without production. And the model only gets the table. The last line in the code proves it. Not the prompt.
Written by me. The thoughts, the values, the learnings, the mistakes: all mine. Grammar and spelling are corrected by our own twin model, trained on my texts. Sometimes a stumble stays in. That is mine too.