Baxnet Ideas · Technical explainer
The Training Examples Teach More Than Their Labels
TL;DR: An LLM generated fictional candidate wording and style variants during a development exercise. Their lineage has to remain visible when training and testing a topic classifier.
When people hear that a classifier learns from examples, the phrase can make the material sound more solid than it is. A label is only one part of an example. Its wording, length, punctuation, setting, and relationship to nearby examples can all become clues for the model.
That mattered when we began exploring topic training data for Mimoto. A limited observed basis helped us identify kinds of language a topic might contain. It came from a limited set of available chats, and it was still narrow; it did not give a complete picture of how people speak. The aim was to develop candidate experiments, not to turn a small set of observed material into a claim about everyone.
The article does not show those messages. Every example below is invented. A source message, a protected test, or a private row is not needed to explain the design problem.
More wording does not mean more people
For the same development exercise, we used an LLM to generate fictional candidate wording and style variants for training material. A fictional seed such as “The boiler engineer is coming at eight” might lead to invented variants such as “Boiler engineer here at eight” or “The engineer’s coming at eight to look at the boiler.”
Those variants may cover wording a hand-built phrase list misses. They may help expose whether the topic definition survives a change in tone or syntax. More generated versions do not add independent people or sources to the observed basis. Treating descendants as a large new population would produce a more impressive count and a weaker account of what it means.
The work therefore has two separate jobs. The development generator proposes material for review. The runtime classifier, if one is ever selected, has to make a topic decision on supplied text. A data-preparation tool can spend time proposing alternatives for review. A runtime classifier would need predictable behaviour on the text it is actually given. They should not be judged as if they carry the same workload, nor should one be used as proof that the other is ready.
Keep the review attached to the source family
A practical augmentation workflow would start with a narrow definition, not a prompt asking for more examples. It should state what the topic includes, what it excludes, and where the honest answer is unknown. It can then propose fictional variants to exercise what the limited observed basis revealed. Each proposed row would need a semantic check: does it still belong in the topic? Is its label defensible? Has a change in style quietly changed the subject?
The checks are more useful when they are concrete. A fictional home-maintenance topic can include “The tap is leaking again.” It can exclude “The software update leaked the new screen.” It can leave “Can you sort that leak?” unknown when no context says what kind of leak is being discussed. A generator can offer all three kinds of sentence. It cannot decide the boundary by itself.
That is why a review record matters. It should preserve the proposed label, the reason a row belongs or does not belong, and its relationship to the seed that prompted it. We should not say that every possible style is covered, that a generated variant proves real-world prevalence, or that a review has approved a product interpretation beyond the evidence it actually examined.
A near-copy belongs with its family
The most flattering evaluation can be one where the test answer is already hiding in the training material. If the training set contains “The boiler engineer is coming at eight” and the test set contains “Engineer at eight for the boiler,” a classifier may seem to generalise while it has effectively seen the same event twice.
That risk grows with generated variants. A seed and its descendants share more than a label. They can share the same event, phrasing pattern, and assumptions about what a positive example looks like. Scikit-learn’s guidance on common pitfalls makes the general point: information from outside a training split can create an overly optimistic evaluation. In this work, the design requirement is to keep an original seed and all of its descendants in the same partition, then keep related event, context, and template families together as well.
One related family goes to Train or Test.
Generated material needs its lineage retained. A variant can be useful for testing a definition, finding a missing negative, or making a reviewer ask a better question. It should not drift into both Train and Test merely because the wording looks different.
The resulting candidate may still fail. It may encounter a regional phrase, an incomplete reply, a joke, or a context that was not in the supplied window. That failure is useful information. It can show that a label is too loose, the observed basis is too narrow, or the experiment needs a different kind of difficult example.
The aim is not to create the largest possible dataset. It is to create a small, inspectable basis for asking whether a topic definition survives contact with unfamiliar wording. More rows are only helpful when their origin, limits, and relationship to the test remain visible.