This paper presents Holistically-Nested Edge Detection (HED), a new edge detection algorithm that addresses two key challenges in edge detection: holistic image training and prediction, and multi-scale and multi-level feature learning. HED uses a deep learning model based on fully convolutional neural networks and deeply-supervised nets to perform image-to-image prediction. The method automatically learns rich hierarchical representations guided by deep supervision on side responses, which are crucial for resolving the ambiguity in edge and object boundary detection. HED achieves state-of-the-art performance on the BSD500 dataset (ODS F-score of 0.782) and the NYU Depth dataset (ODS F-score of 0.746), with improved speed (0.4s per image) compared to recent CNN-based edge detection algorithms. HED is designed to produce end-to-end edge detection by combining multi-scale and multi-level features. The architecture includes multiple side outputs that are progressively refined, with each subsequent edge map being more concise. This approach is in contrast to previous multi-scale methods that do not automatically learn or hierarchically connect scale-space edge fields. The proposed method also includes a weighted-fusion layer that automatically learns how to combine outputs from multiple scales. The HED architecture is based on a trimmed VGGNet, with side-output layers inserted after convolutional layers. Deep supervision is applied at each side-output layer to guide the learning of edge predictions. The network is trained using a combination of side-output loss and a fusion-layer loss, which helps to produce accurate edge maps. The final output is obtained by averaging the predictions from multiple scales. Experiments on the BSDS500 and NYUDv2 datasets show that HED achieves state-of-the-art performance in edge detection, with high accuracy and speed. The method is also effective when incorporating depth information through HHA features. The results demonstrate that HED outperforms previous methods in terms of both accuracy and efficiency, and that the use of deep supervision and multi-scale features is crucial for achieving these results. The method is also shown to be robust to variations in input scale and to perform well even with limited training data.This paper presents Holistically-Nested Edge Detection (HED), a new edge detection algorithm that addresses two key challenges in edge detection: holistic image training and prediction, and multi-scale and multi-level feature learning. HED uses a deep learning model based on fully convolutional neural networks and deeply-supervised nets to perform image-to-image prediction. The method automatically learns rich hierarchical representations guided by deep supervision on side responses, which are crucial for resolving the ambiguity in edge and object boundary detection. HED achieves state-of-the-art performance on the BSD500 dataset (ODS F-score of 0.782) and the NYU Depth dataset (ODS F-score of 0.746), with improved speed (0.4s per image) compared to recent CNN-based edge detection algorithms. HED is designed to produce end-to-end edge detection by combining multi-scale and multi-level features. The architecture includes multiple side outputs that are progressively refined, with each subsequent edge map being more concise. This approach is in contrast to previous multi-scale methods that do not automatically learn or hierarchically connect scale-space edge fields. The proposed method also includes a weighted-fusion layer that automatically learns how to combine outputs from multiple scales. The HED architecture is based on a trimmed VGGNet, with side-output layers inserted after convolutional layers. Deep supervision is applied at each side-output layer to guide the learning of edge predictions. The network is trained using a combination of side-output loss and a fusion-layer loss, which helps to produce accurate edge maps. The final output is obtained by averaging the predictions from multiple scales. Experiments on the BSDS500 and NYUDv2 datasets show that HED achieves state-of-the-art performance in edge detection, with high accuracy and speed. The method is also effective when incorporating depth information through HHA features. The results demonstrate that HED outperforms previous methods in terms of both accuracy and efficiency, and that the use of deep supervision and multi-scale features is crucial for achieving these results. The method is also shown to be robust to variations in input scale and to perform well even with limited training data.