This paper introduces D3IL, a benchmark for imitation learning with human demonstrations, designed to evaluate a model's ability to learn multi-modal behavior. The D3IL benchmark includes simulation environments and datasets with diverse human demonstrations, designed to involve multiple sub-tasks, object manipulation, and closed-loop feedback. The benchmark aims to address the challenge of quantifying a model's capacity to capture and replicate diverse human behaviors. The paper introduces tractable metrics to assess a model's ability to acquire and reproduce diverse behaviors, providing a practical means to evaluate the robustness and versatility of imitation learning algorithms. The authors conduct a thorough evaluation of state-of-the-art methods on the proposed task suite, highlighting their effectiveness in learning diverse behaviors. The results show that diffusion-based methods, especially those incorporating transformer backbones, excel in learning diverse behavior while maintaining strong performance across all tasks. The paper also evaluates the impact of history and prediction horizon on performance, and assesses the ability of models to learn with less data. The findings suggest that transformer-based methods generalize better with less training data and diffusion-based methods can regularize the transformer to make it less data-hungry. The D3IL benchmark provides a valuable reference for the design of future imitation learning algorithms.This paper introduces D3IL, a benchmark for imitation learning with human demonstrations, designed to evaluate a model's ability to learn multi-modal behavior. The D3IL benchmark includes simulation environments and datasets with diverse human demonstrations, designed to involve multiple sub-tasks, object manipulation, and closed-loop feedback. The benchmark aims to address the challenge of quantifying a model's capacity to capture and replicate diverse human behaviors. The paper introduces tractable metrics to assess a model's ability to acquire and reproduce diverse behaviors, providing a practical means to evaluate the robustness and versatility of imitation learning algorithms. The authors conduct a thorough evaluation of state-of-the-art methods on the proposed task suite, highlighting their effectiveness in learning diverse behaviors. The results show that diffusion-based methods, especially those incorporating transformer backbones, excel in learning diverse behavior while maintaining strong performance across all tasks. The paper also evaluates the impact of history and prediction horizon on performance, and assesses the ability of models to learn with less data. The findings suggest that transformer-based methods generalize better with less training data and diffusion-based methods can regularize the transformer to make it less data-hungry. The D3IL benchmark provides a valuable reference for the design of future imitation learning algorithms.