Newton-based bilevel optimization can now handle non-unique solutions and singular Hessians by leveraging shared curvature between lower-level and multiplier updates, enabling faster convergence in practical ML tasks like hyperparameter tuning and data reweighting.
This paper develops Newton-based methods for bilevel optimization that work when lower-level solutions aren't unique and the Hessian is singular—situations where standard methods fail. The authors propose two new algorithms (Newton-based GALET and Newton-based BAMM) that exploit shared curvature structure and provide faster convergence guarantees.