I’m a Master’s student at Kookmin University in Seoul, working on AI security, AI safety, and the policy questions between them. I write explanations of attacks and defenses and write-ups of my own research, with the code to rerun them.
Writing
Prompt-injection detectors, tested on prompts they should let through
A detector with F1 0.988 still blocked about a third of harmless, security-related prompts from public datasets. No threshold fixed it, and extra training only fixed the prompts we had written ourselves.
Indirect prompt injection, tested on an email agent
An email agent with a mock send tool and a fake secret, 360 poisoned inboxes, two OpenAI models, no leaks, and why that is less reassuring than it sounds.
Prompt injection, explained by running one
A twenty-line summarizer, a document with two extra sentences, four local models, and why this is a harder problem than SQL injection ever was.
Paper notes
Measuring the Wrong Thing: the jailbreaks that work score lower than the ones that fail
On Llama-3.1-8B, a harmfulness score that separates harmful from harmless prompts well ranks the jailbreaks that succeed below the ones that fail; the broader claims rest on thinner evidence.
XSTest: the over-refusal test holds up, its automatic grader less so
XSTest’s human-labelled results reproduce exactly from the released files, but the GPT-4 grader the paper offers as a cheaper substitute agrees with those labels far less than its tables suggest.
PIGuard and NotInject: what the over-defense result shows, and what it doesn’t
NotInject shows public prompt-injection guards blocking many harmless prompts that contain attack-like words, but PIGuard’s own gain is hard to read, because its released code picks the model using items from the test sets.
Publications
- PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift DOI · arXiv · Benchmark and code