This paper proposes a novel approach for unsupervised domain adaptation in deep networks, called Residual Transfer Networks (RTN). The method jointly learns adaptive classifiers and transferable features from labeled source data and unlabeled target data. Unlike previous methods that assume shared classifiers, RTN assumes that the source and target classifiers differ by a small residual function. The approach enables classifier adaptation by inserting residual layers into deep networks to explicitly learn the residual function with reference to the target classifier. Features from multiple layers are fused using tensor product and embedded into reproducing kernel Hilbert spaces to match distributions for feature adaptation. The adaptation is achieved by extending feed-forward models with new residual layers and loss functions, which can be trained efficiently via back-propagation. Empirical results show that RTN outperforms state-of-the-art methods on standard domain adaptation benchmarks. The paper discusses related work in domain adaptation, highlighting the challenges of domain shift and the effectiveness of deep neural networks in learning domain-invariant features. It also addresses the limitations of previous methods, such as the assumption that source classifiers can be directly transferred to the target domain. RTN addresses this by learning a residual function that bridges the source and target classifiers. The paper presents the RTN architecture, which includes feature adaptation and classifier adaptation. Feature adaptation is achieved by matching the feature distributions of multiple layers across domains using tensor MMD. Classifier adaptation is achieved by learning a residual function that bridges the source and target classifiers. The RTN model is trained using standard back-propagation and is shown to outperform other methods on benchmark datasets. The paper also includes experimental results on the Office-31 and Office-Caltech datasets, demonstrating the effectiveness of RTN in domain adaptation tasks. The results show that RTN achieves higher accuracy on both easy and hard transfer tasks compared to other methods. The paper concludes that RTN provides a new state-of-the-art approach for unsupervised domain adaptation in deep networks.This paper proposes a novel approach for unsupervised domain adaptation in deep networks, called Residual Transfer Networks (RTN). The method jointly learns adaptive classifiers and transferable features from labeled source data and unlabeled target data. Unlike previous methods that assume shared classifiers, RTN assumes that the source and target classifiers differ by a small residual function. The approach enables classifier adaptation by inserting residual layers into deep networks to explicitly learn the residual function with reference to the target classifier. Features from multiple layers are fused using tensor product and embedded into reproducing kernel Hilbert spaces to match distributions for feature adaptation. The adaptation is achieved by extending feed-forward models with new residual layers and loss functions, which can be trained efficiently via back-propagation. Empirical results show that RTN outperforms state-of-the-art methods on standard domain adaptation benchmarks. The paper discusses related work in domain adaptation, highlighting the challenges of domain shift and the effectiveness of deep neural networks in learning domain-invariant features. It also addresses the limitations of previous methods, such as the assumption that source classifiers can be directly transferred to the target domain. RTN addresses this by learning a residual function that bridges the source and target classifiers. The paper presents the RTN architecture, which includes feature adaptation and classifier adaptation. Feature adaptation is achieved by matching the feature distributions of multiple layers across domains using tensor MMD. Classifier adaptation is achieved by learning a residual function that bridges the source and target classifiers. The RTN model is trained using standard back-propagation and is shown to outperform other methods on benchmark datasets. The paper also includes experimental results on the Office-31 and Office-Caltech datasets, demonstrating the effectiveness of RTN in domain adaptation tasks. The results show that RTN achieves higher accuracy on both easy and hard transfer tasks compared to other methods. The paper concludes that RTN provides a new state-of-the-art approach for unsupervised domain adaptation in deep networks.