This paper introduces SIMPLER, a suite of simulated environments for evaluating real-world robot manipulation policies. The authors demonstrate that simulated evaluations can effectively predict real-world performance, offering a scalable, reproducible, and reliable alternative to costly real-world testing. They address key challenges in sim-to-real evaluation, including control and visual disparities between real and simulated environments, by proposing approaches to mitigate these gaps without requiring full-fidelity digital twins. These approaches include system identification for control gaps and "green-screening" for visual gaps. The authors show that policies trained on real data perform consistently well in SIMPLER environments, with strong correlations between simulated and real-world performance. They also find that SIMPLER evaluations accurately reflect real-world policy behavior modes, such as sensitivity to distribution shifts. The authors open-source all SIMPLER environments and their workflow for creating new environments to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks. They also show that their results are robust across different physics simulators, including SAPIEN and Isaac Sim. The paper highlights the potential of simulation-based evaluation as a tool for scalable, reproducible, and reliable policy evaluation in robotics.This paper introduces SIMPLER, a suite of simulated environments for evaluating real-world robot manipulation policies. The authors demonstrate that simulated evaluations can effectively predict real-world performance, offering a scalable, reproducible, and reliable alternative to costly real-world testing. They address key challenges in sim-to-real evaluation, including control and visual disparities between real and simulated environments, by proposing approaches to mitigate these gaps without requiring full-fidelity digital twins. These approaches include system identification for control gaps and "green-screening" for visual gaps. The authors show that policies trained on real data perform consistently well in SIMPLER environments, with strong correlations between simulated and real-world performance. They also find that SIMPLER evaluations accurately reflect real-world policy behavior modes, such as sensitivity to distribution shifts. The authors open-source all SIMPLER environments and their workflow for creating new environments to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks. They also show that their results are robust across different physics simulators, including SAPIEN and Isaac Sim. The paper highlights the potential of simulation-based evaluation as a tool for scalable, reproducible, and reliable policy evaluation in robotics.