Evaluations of machine learning privacy defenses are misleading. Empirical privacy evaluations based on membership inference attacks often fail to accurately reflect the privacy leakage of the most vulnerable samples, use weak attacks, and avoid comparisons with practical differential privacy (DP) baselines. In five case studies of empirical privacy defenses, prior evaluations underestimate privacy leakage by an order of magnitude. Under a stronger evaluation, none of the empirical defenses studied are competitive with a properly tuned, high-utility DP-SGD baseline. The paper identifies three major pitfalls in existing empirical privacy evaluations: (1) aggregating attack success over a dataset, which fails to capture individual privacy; (2) using weak or non-adaptive attacks that do not reflect the current state-of-the-art; and (3) comparing empirical defenses to weak DP-SGD baselines with low utility. To address these issues, the paper proposes a reliable and efficient evaluation protocol that focuses on the most vulnerable samples, uses state-of-the-art adaptive attacks, and compares to strong DP baselines with similar utility. The evaluation methodology involves using canary samples that mimic the most vulnerable data to accurately reflect privacy leakage. It also adapts attacks to the specifics of each defense and compares to DP baselines that achieve similar utility. The results show that existing evaluations often fail to capture the true privacy leakage of the most vulnerable samples, and that none of the studied empirical defenses are competitive with a properly tuned DP-SGD baseline. The paper also highlights the importance of using strong DP baselines and adaptive attacks to evaluate privacy defenses accurately.Evaluations of machine learning privacy defenses are misleading. Empirical privacy evaluations based on membership inference attacks often fail to accurately reflect the privacy leakage of the most vulnerable samples, use weak attacks, and avoid comparisons with practical differential privacy (DP) baselines. In five case studies of empirical privacy defenses, prior evaluations underestimate privacy leakage by an order of magnitude. Under a stronger evaluation, none of the empirical defenses studied are competitive with a properly tuned, high-utility DP-SGD baseline. The paper identifies three major pitfalls in existing empirical privacy evaluations: (1) aggregating attack success over a dataset, which fails to capture individual privacy; (2) using weak or non-adaptive attacks that do not reflect the current state-of-the-art; and (3) comparing empirical defenses to weak DP-SGD baselines with low utility. To address these issues, the paper proposes a reliable and efficient evaluation protocol that focuses on the most vulnerable samples, uses state-of-the-art adaptive attacks, and compares to strong DP baselines with similar utility. The evaluation methodology involves using canary samples that mimic the most vulnerable data to accurately reflect privacy leakage. It also adapts attacks to the specifics of each defense and compares to DP baselines that achieve similar utility. The results show that existing evaluations often fail to capture the true privacy leakage of the most vulnerable samples, and that none of the studied empirical defenses are competitive with a properly tuned DP-SGD baseline. The paper also highlights the importance of using strong DP baselines and adaptive attacks to evaluate privacy defenses accurately.