Keypoint Action Tokens (KAT) enable efficient in-context imitation learning in robotics by repurposing large text-pretrained Transformers. This method transforms visual observations and action sequences into sequences of keypoint-based tokens, which are then processed by a text-pretrained Transformer (e.g., GPT-4 Turbo) to generate action sequences. KAT requires only 10 demonstrations to learn general behaviors, and no additional training is needed, allowing immediate deployment. The framework leverages Vision Transformers and text-based large Transformers to create a low-level language of robotics actions, converting sequences of observations and actions into tokens that a text-pretrained Transformer can process. KAT was evaluated on a variety of real-world tasks, including aligning objects, wiping plates, sweeping, espresso preparation, and bottle placement. It demonstrated performance comparable or superior to state-of-the-art imitation learning methods like diffusion policies. The method is robust to visual distractors and generalizes well to novel objects and environments. The key components of KAT include Keypoint Tokens, which represent visual observations as 3D keypoints, and Action Tokens, which represent end-effector poses as triplets of 3D points. These tokens are used to input sequences into the Transformer, which then generates action sequences to emulate expert behavior. The method's effectiveness is attributed to the Transformer's ability to learn sequence-to-sequence patterns from few examples, even when pre-trained on text data. KAT's performance was tested with varying numbers of demonstrations and keypoint/action tokens, showing that the optimal number of keypoint tokens is between 10 and 20, and the optimal number of action tokens is around 20. The method also demonstrated robustness to different vision models and action representations, with DINO-ViTs providing the best results for keypoint extraction. The study highlights the potential of repurposing large language models for robotics tasks, particularly in low-data regimes. While KAT does not scale as well as diffusion policies, it offers a promising alternative for efficient in-context imitation learning. Future work may focus on improving adaptability and dynamic keypoint extraction to enhance performance.Keypoint Action Tokens (KAT) enable efficient in-context imitation learning in robotics by repurposing large text-pretrained Transformers. This method transforms visual observations and action sequences into sequences of keypoint-based tokens, which are then processed by a text-pretrained Transformer (e.g., GPT-4 Turbo) to generate action sequences. KAT requires only 10 demonstrations to learn general behaviors, and no additional training is needed, allowing immediate deployment. The framework leverages Vision Transformers and text-based large Transformers to create a low-level language of robotics actions, converting sequences of observations and actions into tokens that a text-pretrained Transformer can process. KAT was evaluated on a variety of real-world tasks, including aligning objects, wiping plates, sweeping, espresso preparation, and bottle placement. It demonstrated performance comparable or superior to state-of-the-art imitation learning methods like diffusion policies. The method is robust to visual distractors and generalizes well to novel objects and environments. The key components of KAT include Keypoint Tokens, which represent visual observations as 3D keypoints, and Action Tokens, which represent end-effector poses as triplets of 3D points. These tokens are used to input sequences into the Transformer, which then generates action sequences to emulate expert behavior. The method's effectiveness is attributed to the Transformer's ability to learn sequence-to-sequence patterns from few examples, even when pre-trained on text data. KAT's performance was tested with varying numbers of demonstrations and keypoint/action tokens, showing that the optimal number of keypoint tokens is between 10 and 20, and the optimal number of action tokens is around 20. The method also demonstrated robustness to different vision models and action representations, with DINO-ViTs providing the best results for keypoint extraction. The study highlights the potential of repurposing large language models for robotics tasks, particularly in low-data regimes. While KAT does not scale as well as diffusion policies, it offers a promising alternative for efficient in-context imitation learning. Future work may focus on improving adaptability and dynamic keypoint extraction to enhance performance.