By learning which attention heads should focus on instructions vs. user input, you can defend against prompt injection attacks with minimal overhead—HoS reduces attack success rates from 97% to 10% while preserving the model's ability to use tools and answer questions correctly.
Heads of Security (HoS) is a defense against prompt injection attacks that works by training small attention gates for each transformer head to separate trusted instructions from untrusted user data. Unlike methods that require full model retraining, HoS adds minimal parameters while maintaining model utility and resisting both black-box and white-box attacks far better than existing defenses.