Out-of-Distribution Generalization via Risk Extrapolation

Out-of-Distribution Generalization via Risk Extrapolation

25 Feb 2021 | David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Prieur, Aaron Courville
Risk Extrapolation (REx) is a method for achieving out-of-distribution (OOD) generalization by optimizing for robustness to distributional shifts. The core idea is to reduce differences in risk across training domains, which helps models generalize better to new, unseen test domains. REx can handle both causal and anticausal relationships in input data, making it effective in scenarios where distributional shifts are extreme. The method is motivated as a form of robust optimization over a perturbation set of extrapolated domains (MM-REx), and a simpler variant is V-REx, which penalizes the variance of training risks. REx has been shown to outperform Invariant Risk Minimization (IRM) in settings involving covariate shift and requires invariant prediction. Theoretical analysis shows that equalizing risks can lead to the discovery of causal mechanisms. Experiments demonstrate that REx outperforms IRM on tasks such as Colored MNIST with covariate shift and domain generalization tasks. REx is particularly effective in scenarios involving both covariate and interventional shifts, while IRM struggles in such cases. The method is robust to distributional shifts and can provide better generalization in real-world settings.Risk Extrapolation (REx) is a method for achieving out-of-distribution (OOD) generalization by optimizing for robustness to distributional shifts. The core idea is to reduce differences in risk across training domains, which helps models generalize better to new, unseen test domains. REx can handle both causal and anticausal relationships in input data, making it effective in scenarios where distributional shifts are extreme. The method is motivated as a form of robust optimization over a perturbation set of extrapolated domains (MM-REx), and a simpler variant is V-REx, which penalizes the variance of training risks. REx has been shown to outperform Invariant Risk Minimization (IRM) in settings involving covariate shift and requires invariant prediction. Theoretical analysis shows that equalizing risks can lead to the discovery of causal mechanisms. Experiments demonstrate that REx outperforms IRM on tasks such as Colored MNIST with covariate shift and domain generalization tasks. REx is particularly effective in scenarios involving both covariate and interventional shifts, while IRM struggles in such cases. The method is robust to distributional shifts and can provide better generalization in real-world settings.
Reach us at info@study.space
[slides] Out-of-Distribution Generalization via Risk Extrapolation (REx) | StudySpace