that “teams are remarkably clever, and are sometimes smarter than the neatest folks in them.” He was writing about decision-making, however the identical precept applies to classification: get sufficient folks to explain the identical phenomenon and a taxonomy begins to emerge, even when no two folks phrase it the identical method. The problem is extracting that sign from the noise.
I had a number of thousand rows of free-text information and wanted to do precisely that. Every row was a brief natural-language annotation explaining why an automated security finding was irrelevant, which features to make use of for a repair, or what coding practices to comply with. One particular person wrote “that is take a look at code, not deployed anyplace.” One other wrote “non-production atmosphere, secure to disregard.” A 3rd wrote “solely runs in CI/CD pipeline throughout integration checks.” All three meant the identical factor, however no two shared greater than a phrase or two.
The taxonomy was in there. I simply wanted the proper software to extract it. Conventional clustering and key phrase matching couldn’t deal with the paraphrase variation, so I attempted one thing I hadn’t seen mentioned a lot: utilizing a domestically hosted LLM as a zero-shot classifier. This weblog publish explores the way it carried out, the way it works, and a few ideas for utilizing and deploying these methods your self.
Why conventional clustering struggles with quick free-text
Commonplace unsupervised clustering works by discovering mathematical proximity in some function house. For lengthy paperwork, that is normally effective. Sufficient sign exists in phrase frequencies or embedding vectors to kind coherent teams. However quick, semantically dense textual content breaks these assumptions in just a few particular methods.
Embedding similarity conflates totally different meanings. “This secret’s solely utilized in improvement” and “This API secret’s hardcoded for comfort” produce related embeddings as a result of the vocabulary overlaps. However one is a couple of non-production atmosphere and the opposite is about an intentional safety tradeoff. K-means or DBSCAN can’t distinguish them as a result of the vectors are too shut.
Subject fashions floor phrases, not ideas. Latent Dirichlet Allocation (LDA) and its variants discover phrase co-occurrence patterns. When your corpus consists of one-sentence annotations, the phrase co-occurrence sign is just too sparse to kind significant matters. You get clusters outlined by “take a look at” or “code” or “safety” somewhat than coherent themes.
Regex and key phrase matching can’t deal with paraphrase variation. You possibly can write guidelines to catch “take a look at code” and “non-production,” however you’d miss “solely used throughout CI,” “by no means deployed,” “development-only fixture,” and dozens of different phrasings that each one categorical the identical underlying thought.
The frequent thread: these strategies function on floor options (tokens, vectors, patterns) somewhat than semantic which means. For classification duties the place which means issues greater than vocabulary, you want one thing that understands language.
LLMs as zero-shot classifiers
The important thing perception is straightforward: as an alternative of asking an algorithm to find clusters, outline your candidate classes primarily based on area information and ask a language mannequin to categorise every entry.
This works as a result of LLMs course of semantic which means, not simply token patterns. “This secret’s solely utilized in improvement” and “Non-production atmosphere, secure to disregard” comprise virtually no overlapping phrases, however a language mannequin understands they categorical the identical thought. This isn’t simply instinct. Chae and Davidson (2025) in contrast 10 fashions throughout zero-shot, few-shot, and fine-tuned coaching regimes and located that enormous LLMs in zero-shot mode carried out competitively with fine-tuned BERT on stance detection duties. Wang et al. (2023) discovered LLMs outperformed state-of-the-art classification strategies on three of 4 benchmark datasets utilizing zero-shot prompting alone, no labeled coaching information required.
The setup has three elements:
- Candidate classes. A listing of mutually unique classes outlined from area information. In my case, I began with about 10 anticipated themes (take a look at code, enter validation, framework protections, non-production environments, and so forth.) and expanded to twenty candidates after reviewing a pattern.
- A classification immediate. Structured to return a class label and a quick purpose. Low temperature (0.1) for consistency. Quick max output (100 tokens) since we solely want a label, not an essay.
- A neighborhood LLM. I used Ollama to run fashions domestically. No API prices, no information leaving my machine, and quick sufficient for hundreds of classifications.
Right here’s the core of the classification immediate:
CLASSIFICATION_PROMPT = """
Classify this textual content into considered one of these themes:
{themes}
Textual content:
"{content material}"
Reply with ONLY the theme quantity and title, and a quick purpose.
Format: THEME_NUMBER. THEME_NAME | Motive
Classification:
"""And the Ollama name:
response = ollama.generate(
mannequin="gemma2",
immediate=immediate,
choices={
"temperature": 0.1, # Low temp for constant classification
"num_predict": 100, # Quick response, we simply want a label
}
)Two issues to notice. First, the temperature setting issues. At 0.7 or increased, the identical enter can produce totally different classifications throughout runs. At 0.1, the mannequin is almost deterministic, which helps easy classification. Second, limiting num_predict retains the mannequin from producing explanations you don’t want, which hurries up throughput considerably.
Constructing the pipeline
The total pipeline has three steps: preprocess, classify, analyze.
Preprocessing strips content material that provides tokens with out including classification sign. URLs, boilerplate phrases (“For extra data, see…”), and formatting artifacts all get eliminated. Widespread phrases get normalized (“false optimistic” turns into “FP,” “manufacturing” turns into “prod”) to cut back token variation. Deduplication by content material hash removes precise repeats. This step decreased my token price range by roughly 30% and made classification extra constant.
Classification runs every entry via the LLM with the candidate classes. For ~7,000 entries, this took about 45 minutes on a MacBook Professional utilizing Gemma 2 (9B parameters). I additionally examined Llama 3.2 (3B), which was quicker however barely much less exact on edge circumstances the place two classes have been shut. Gemma 2 dealt with ambiguous entries with noticeably higher judgment.
One sensible concern: lengthy runs can fail partway via. The pipeline saves checkpoints each 100 classifications, so you may resume from the place you left off.
Evaluation aggregates the outcomes and generates a distribution chart. Right here’s what the output regarded like:
The chart tells a transparent story. Over 1 / 4 of all entries described code that solely runs in non-production environments. One other 21.9% described circumstances the place a safety framework already handles the chance. These two classes alone account for half the dataset, which is the type of perception that’s onerous to extract from unstructured textual content every other method.
When this strategy isn’t the proper match
This system works finest in a selected area of interest: medium-scale datasets (a whole bunch to tens of hundreds of entries), semantically advanced textual content, and conditions the place you’ve gotten sufficient area information to outline candidate classes however no labeled coaching information.
It’s not the proper software when:
- your classes are keyword-defined (simply use regex),
- when you’ve gotten labeled coaching information (prepare a supervised classifier; it’ll be quicker and cheaper),
- if you want sub-second latency at scale (use embeddings and a nearest-neighbor lookup),
- or if you genuinely don’t know what classes exist. On this case, run exploratory subject modeling first to develop instinct, then swap to LLM classification as soon as you may outline classes.
The opposite constraint is throughput. Even on a quick machine, classifying one entry per fraction of a second means 7,000 entries takes near an hour. For datasets above 100,000 entries, you’ll need an API-hosted mannequin or a batching technique.
Different functions price attempting
The pipeline generalizes to any downside the place you’ve gotten unstructured textual content and wish structured classes.
Buyer suggestions. NPS responses, assist tickets, and survey open-ends all undergo from the identical downside: diverse phrasing for a finite set of underlying themes. “Your app crashes each time I open settings” and “Settings web page is damaged on iOS” are the identical class, however key phrase matching gained’t catch that.
Bug report triage. Free-text bug descriptions may be auto-categorized by part, root trigger, or severity. That is particularly helpful when the particular person submitting the bug doesn’t know which part is accountable.
Code intent classification. That is one I haven’t tried but however discover compelling: classifying code snippets, Semgrep rules, or configuration guidelines by objective (authentication, information entry, error dealing with, logging). The identical method applies. Outline the classes, write a classification immediate, run the corpus via a neighborhood mannequin.
Getting began
The pipeline is easy: outline your classes, write a classification immediate, and run your information via a neighborhood mannequin.
The toughest half isn’t the code. It’s defining classes which can be mutually unique and collectively exhaustive. My recommendation: begin with a pattern of 100 entries, classify them manually, discover which classes you retain reaching for, and use these as your candidate record. Then let the LLM scale the sample.
I used this system as a part of a larger analysis on how security teams remediate vulnerabilities. The classification outcomes helped floor which forms of safety context are most typical throughout organizations, and the chart above is among the outputs from that work. In case you’re within the safety angle, the complete report is offered at that hyperlink.
