AUTOATTACKER is an LLM-guided system designed to automate "hands-on-keyboard" cyber-attacks on simulated organizational networks. The system addresses challenges in attack automation, including complex task chains, high variability in action spaces, and the need for precise command execution. AUTOATTACKER incorporates a modular agent design, leveraging LLM capabilities such as planning, summarizing, and code generation. It uses a Retrieval Augmented Generation (RAG) inspired experience manager to build complex attacks from basic tasks. The system includes a summarizer, planner, navigator, and experience manager to iteratively interact with LLMs. The results show that AUTOATTACKER is highly effective when using GPT-4, achieving perfect success rates, while other LLMs like GPT-3.5 and Llama2 models perform poorly. The system is evaluated on a new benchmark with 14 attack tasks covering various stages of the attack lifecycle. AUTOATTACKER's modular design and RAG-based experience manager help reduce attack overhead and improve success rates. The system also includes a jailbreaking technique to bypass LLM usage policies. The research highlights the potential risks of LLMs in automating cyber-attacks and the need for defensive measures to counter these risks. The study contributes to the field by presenting the first comprehensive evaluation of LLMs in human-like hands-on-keyboard attacks and proposing a new system for attack automation.AUTOATTACKER is an LLM-guided system designed to automate "hands-on-keyboard" cyber-attacks on simulated organizational networks. The system addresses challenges in attack automation, including complex task chains, high variability in action spaces, and the need for precise command execution. AUTOATTACKER incorporates a modular agent design, leveraging LLM capabilities such as planning, summarizing, and code generation. It uses a Retrieval Augmented Generation (RAG) inspired experience manager to build complex attacks from basic tasks. The system includes a summarizer, planner, navigator, and experience manager to iteratively interact with LLMs. The results show that AUTOATTACKER is highly effective when using GPT-4, achieving perfect success rates, while other LLMs like GPT-3.5 and Llama2 models perform poorly. The system is evaluated on a new benchmark with 14 attack tasks covering various stages of the attack lifecycle. AUTOATTACKER's modular design and RAG-based experience manager help reduce attack overhead and improve success rates. The system also includes a jailbreaking technique to bypass LLM usage policies. The research highlights the potential risks of LLMs in automating cyber-attacks and the need for defensive measures to counter these risks. The study contributes to the field by presenting the first comprehensive evaluation of LLMs in human-like hands-on-keyboard attacks and proposing a new system for attack automation.