For safe AI deployment, you need to estimate how often catastrophic rare events occur in agent behavior. This paper provides a practical method using importance sampling with learned weight perturbations, achieving massive efficiency gains over naive approaches.
This paper tackles estimating extremely rare event probabilities in AI agent trajectories—events so uncommon that standard Monte Carlo sampling is impractical. The authors develop a new importance sampling method that tweaks a language model's weights to generate more likely rare events, using gradient-based optimization to search efficiently.