This paper proposes CLEF, a causal debiasing framework based on counterfactual inference to address the context bias interference in context-aware emotion recognition (CAER). CLEF reveals that the harmful bias confounds model performance along the direct causal effect via the tailored causal graph, and accomplishes bias mitigation by subtracting the direct context effect from the total causal effect. Extensive experiments prove that CLEF brings favorable improvements to existing models. The framework is designed to be model-agnostic, allowing it to be integrated into existing methods to achieve consistent performance gains. CLEF introduces a non-invasive context branch to capture the adverse direct effect caused by context bias. During inference, it eliminates the direct context effect from the total causal effect by comparing factual and counterfactual outcomes, resulting in bias mitigation and robust prediction. The framework is evaluated on two large-scale CAER datasets, EMOTIC and CAER-S, showing significant improvements in emotion recognition accuracy. The results demonstrate that CLEF effectively reduces the impact of context bias, leading to more accurate and reliable emotion predictions.This paper proposes CLEF, a causal debiasing framework based on counterfactual inference to address the context bias interference in context-aware emotion recognition (CAER). CLEF reveals that the harmful bias confounds model performance along the direct causal effect via the tailored causal graph, and accomplishes bias mitigation by subtracting the direct context effect from the total causal effect. Extensive experiments prove that CLEF brings favorable improvements to existing models. The framework is designed to be model-agnostic, allowing it to be integrated into existing methods to achieve consistent performance gains. CLEF introduces a non-invasive context branch to capture the adverse direct effect caused by context bias. During inference, it eliminates the direct context effect from the total causal effect by comparing factual and counterfactual outcomes, resulting in bias mitigation and robust prediction. The framework is evaluated on two large-scale CAER datasets, EMOTIC and CAER-S, showing significant improvements in emotion recognition accuracy. The results demonstrate that CLEF effectively reduces the impact of context bias, leading to more accurate and reliable emotion predictions.