Learning Proposals for Practical Energy-Based Regression

• Energy-based models (EBMs) have experienced a resurgence within machine learning in recent years, including as a promising alternative for probabilistic regression. However, energy-based regression requires a proposal distribution to be manually designed for training, and an initial estimate has to be provided at test-time. We address both of these issues by introducing a conceptually simple method to automatically learn an effective proposal distribution, which is parameterized by a separate network head. To this end, we derive a surprising result, leading to a unified training objective that jointly minimizes the KL divergence from the proposal to the EBM, and the negative log-likelihood of the EBM. At test-time, we can then employ importance sampling with the trained proposal to efficiently evaluate the learned EBM and produce standalone predictions. Furthermore, we utilize our derived training objective to learn mixture density networks (MDNs) with a jointly trained energy-based teacher, consistently outperforming conventional MDN training on four real-world regression tasks within computer vision. Code is available at https://github.com/fregu856/ebms_proposals.

Ämnesord

SAMHÄLLSVETENSKAP  -- Ekonomi och näringsliv -- Företagsekonomi (hsv//swe)

SOCIAL SCIENCES  -- Economics and Business -- Business Administration (hsv//eng)

Publikations- och innehållstyp

ref (ämneskategori)

kon (ämneskategori)