OrCo: Towards Better Generalization via Orthogonality and Contrast for Few-Shot Class-Incremental Learning

OrCo: Towards Better Generalization via Orthogonality and Contrast for Few-Shot Class-Incremental Learning

27 Mar 2024 | Noor Ahmed, Anna Kukleva, Bernt Schiele
OrCo is a novel framework designed to enhance the performance of Few-Shot Class-Incremental Learning (FSCIL) by addressing the challenges of catastrophic forgetting, overfitting, and intransigence. The framework is built on two core principles: features' orthogonality in the representation space and contrastive learning. During pretraining, the model is trained using a combination of supervised and self-supervised contrastive losses to improve feature separation and generalization. In subsequent incremental sessions, the OrCo loss is employed to align the model with generated pseudo-targets, maximizing margins between classes and preserving space for new data. The OrCo loss consists of three components: perturbed supervised contrastive loss (PSCL), cross-entropy loss (CE), and orthogonality loss (ORTH). PSCL enhances class separation by introducing perturbations to pseudo-targets, while CE aligns class features with assigned targets. ORTH enforces a geometric constraint on the feature space to mimic the distribution of pseudo-targets. The framework has been evaluated on three benchmark datasets: mini-ImageNet, CIFAR100, and CUB200, achieving state-of-the-art performance. The results demonstrate that OrCo effectively addresses the challenges of FSCIL by improving generalization, reducing overfitting, and mitigating catastrophic forgetting. The framework's use of orthogonality and contrastive learning ensures that the model can adapt to new classes while preserving previously learned knowledge.OrCo is a novel framework designed to enhance the performance of Few-Shot Class-Incremental Learning (FSCIL) by addressing the challenges of catastrophic forgetting, overfitting, and intransigence. The framework is built on two core principles: features' orthogonality in the representation space and contrastive learning. During pretraining, the model is trained using a combination of supervised and self-supervised contrastive losses to improve feature separation and generalization. In subsequent incremental sessions, the OrCo loss is employed to align the model with generated pseudo-targets, maximizing margins between classes and preserving space for new data. The OrCo loss consists of three components: perturbed supervised contrastive loss (PSCL), cross-entropy loss (CE), and orthogonality loss (ORTH). PSCL enhances class separation by introducing perturbations to pseudo-targets, while CE aligns class features with assigned targets. ORTH enforces a geometric constraint on the feature space to mimic the distribution of pseudo-targets. The framework has been evaluated on three benchmark datasets: mini-ImageNet, CIFAR100, and CUB200, achieving state-of-the-art performance. The results demonstrate that OrCo effectively addresses the challenges of FSCIL by improving generalization, reducing overfitting, and mitigating catastrophic forgetting. The framework's use of orthogonality and contrastive learning ensures that the model can adapt to new classes while preserving previously learned knowledge.
Reach us at info@study.space
[slides and audio] OrCo%3A Towards Better Generalization via Orthogonality and Contrast for Few-Shot Class-Incremental Learning