MAIA is a multimodal automated interpretability agent that uses neural models to automate tasks like feature interpretation and failure mode discovery in neural models. It equips a pre-trained vision-language model with tools for iterative experimentation on subcomponents of other models to explain their behavior. These tools include input synthesis and editing, maximally activating exemplar computation, and experimental result summarization. MAIA evaluates its ability to describe neuron-level features in learned image representations, showing results comparable to human experts. It also aids in reducing sensitivity to spurious features and identifying inputs likely to be misclassified. MAIA's framework allows for flexible evaluation of arbitrary systems and incorporates new experimental tools. It uses a pretrained vision-language model backbone and an API with interpretability tools. MAIA is prompted with an explanation task and designs an interpretability experiment that composes experimental modules to answer the query. MAIA's modular design enables experiments at different levels of granularity, from individual features to entire networks. MAIA's evaluation shows that its descriptions of both synthetic neurons and real neurons are more predictive of behavior than baseline methods and often on par with human labels. It automates model-level interpretation tasks where descriptions of learned representations produce actionable insights about model behavior. MAIA's iterative experimental approach can be applied to downstream model auditing and editing tasks, including spurious feature removal and bias identification. MAIA's modular design allows for the addition and removal of tools, and ablation studies show that the full MAIA system performs best when initializing experiments with dataset exemplars and running additional tests with synthetic images. MAIA's performance is also improved when using DALL-E as the image generator, suggesting that the agent's performance is limited by its tools rather than its ability to use them. MAIA can automatically surface model-level biases by generating synthetic inputs to test classifier behavior. It can identify biases in a CNN trained on a supervised ImageNet classification task. MAIA's ability to generate synthetic data is useful for identifying regions of the input distribution where a model exhibits poor performance. MAIA is a flexible system that automates model understanding tasks at different levels of granularity. It can be used to remove spurious features and identify biases in a trained classifier. MAIA's results show that it can identify and remove spurious features, improving model robustness under distribution shift. It also demonstrates the ability to identify biases in a CNN trained on a supervised ImageNet classification task. MAIA is a prototype for a tool that can help human users ensure AI systems are transparent, reliable, and equitable. It augments but does not replace human oversight of AI systems. MAIA still requires human supervision to catch mistakes such as confirmation bias and image generation/editing failures. Absence of evidence (from MAIA) is not evidence of absence: though MAIA’s toolkit enables causal interventions on inputs in order to evaluate system behavior, MAIA’s explanations do not provide formal verification of system performance.MAIA is a multimodal automated interpretability agent that uses neural models to automate tasks like feature interpretation and failure mode discovery in neural models. It equips a pre-trained vision-language model with tools for iterative experimentation on subcomponents of other models to explain their behavior. These tools include input synthesis and editing, maximally activating exemplar computation, and experimental result summarization. MAIA evaluates its ability to describe neuron-level features in learned image representations, showing results comparable to human experts. It also aids in reducing sensitivity to spurious features and identifying inputs likely to be misclassified. MAIA's framework allows for flexible evaluation of arbitrary systems and incorporates new experimental tools. It uses a pretrained vision-language model backbone and an API with interpretability tools. MAIA is prompted with an explanation task and designs an interpretability experiment that composes experimental modules to answer the query. MAIA's modular design enables experiments at different levels of granularity, from individual features to entire networks. MAIA's evaluation shows that its descriptions of both synthetic neurons and real neurons are more predictive of behavior than baseline methods and often on par with human labels. It automates model-level interpretation tasks where descriptions of learned representations produce actionable insights about model behavior. MAIA's iterative experimental approach can be applied to downstream model auditing and editing tasks, including spurious feature removal and bias identification. MAIA's modular design allows for the addition and removal of tools, and ablation studies show that the full MAIA system performs best when initializing experiments with dataset exemplars and running additional tests with synthetic images. MAIA's performance is also improved when using DALL-E as the image generator, suggesting that the agent's performance is limited by its tools rather than its ability to use them. MAIA can automatically surface model-level biases by generating synthetic inputs to test classifier behavior. It can identify biases in a CNN trained on a supervised ImageNet classification task. MAIA's ability to generate synthetic data is useful for identifying regions of the input distribution where a model exhibits poor performance. MAIA is a flexible system that automates model understanding tasks at different levels of granularity. It can be used to remove spurious features and identify biases in a trained classifier. MAIA's results show that it can identify and remove spurious features, improving model robustness under distribution shift. It also demonstrates the ability to identify biases in a CNN trained on a supervised ImageNet classification task. MAIA is a prototype for a tool that can help human users ensure AI systems are transparent, reliable, and equitable. It augments but does not replace human oversight of AI systems. MAIA still requires human supervision to catch mistakes such as confirmation bias and image generation/editing failures. Absence of evidence (from MAIA) is not evidence of absence: though MAIA’s toolkit enables causal interventions on inputs in order to evaluate system behavior, MAIA’s explanations do not provide formal verification of system performance.