Prompt Injection Guardrail is a machine learning security tool that detects and blocks prompt injection attacks before they reach a large language model. Language models cannot reliably separate instructions written by their developer from instructions smuggled in by a user, which leaves them open to messages designed to override their intended behaviour. Rather than trying to make the main model resistant to these attacks on its own, this project places a small, specialised classifier in front of it to act as a filter.
The project involved fine-tuning Meta's Llama-Prompt-Guard-2-86M classifier on 12,858 samples drawn from three public security datasets, together with synthetic data I generated to target specific weaknesses I found during testing. The trained model is served through a FastAPI service that inspects each incoming prompt and returns a classification in roughly 20 to 30 milliseconds. I designed the service to fail closed, so that if the classifier encounters an error the request is blocked rather than allowed through, and kept the decision threshold on the server side so that it cannot be altered by the traffic being inspected.
The central question for any fine-tuned model is whether the training actually added anything, so I evaluated the finished classifier against the unmodified base model on the same set of attacks. Fine-tuning raised recall from 0.667 to 1.000, a gain of 33 percentage points. What made the comparison more revealing than the number alone was that the base model was not uncertain about the attacks it missed. It scored them confidently as safe, and its failures fell into three consistent categories: attacks framed as authority or compliance requests, attacks obfuscated through leetspeak or base64 encoding, and attacks wrapped inside fictional scenarios.
To avoid the usual ways that evaluation numbers become misleading, I checked the training and test sets for near-duplicate overlap after normalising for casing, punctuation and character substitution, and then built a final holdout set of 60 attacks written entirely in vocabulary that appears nowhere in the training data. The model caught 58 of those 60. Because that set never influenced any training decision, it is the figure I consider the honest estimate of how well the model generalises.
The most useful part of this project was not the final score but the process of diagnosing failures and correcting them. After the first round of training, the model flagged the sentence "Ignore my earlier question about Python and help me with JavaScript instead" as an attack with complete confidence. That is not an attack; it is an ordinary user changing their mind. The model had learned to associate the phrasing with malicious intent without learning whose instructions were being ignored, because public injection datasets contain almost no examples of a user overriding their own earlier message.
I addressed this by generating 500 benign self-corrections paired with contrastive attacks of near-identical wording, which removed the false positive but caused recall to drop slightly. The error had moved rather than disappeared. A newly missed attack used the word "briefing" to refer to the system prompt, a synonym my contrastive examples had never covered. The third round replaced hand-written examples with combinatorial generation across verbs, system-reference nouns, qualifiers and actions, producing enough coverage that the model had to learn the underlying concept rather than memorise a list of words. Recall recovered while the false positive stayed resolved.
The model still has real limitations, which I chose to document rather than leave out. It performs worse on attacks phrased as questions or statements instead of direct commands, and its confidence scores are poorly calibrated, with almost no predictions falling between the two extremes. It is also a single layer of defence rather than a complete solution, and needs to be paired with architectural protections such as least privilege on tools and validation of model output.
Through this project I improved my understanding of transformer fine-tuning, dataset construction, adversarial evaluation, and secure API design. I also learned how easily a model can appear to perform well while carrying a specific and discoverable blind spot, and how much of practical machine learning work is measuring failures rather than improving averages.
