Insights Gained from DeepMind’s RoboCat Paper

**DeepMind Robotics Research: Exploring Transfer and Generalization in Robotics**

**Introduction**
DeepMind Robotics recently published a paper titled “RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation (PDF).” In this post, I will summarize the most interesting results from the paper. Please note that some of the claims and inferences in this blog post are speculative based on my experience working on similar projects. For a general overview of RoboCat, you can refer to DeepMind’s blog post. If any Google DeepMind researchers find any factual inaccuracies, please contact me.

**The Quest for a Unified Robotics Model**
Academic robotics labs are currently focused on the research question of how to develop a single large Transformer model that can perform various robotic tasks, similar to how Transformers excel in language and vision tasks. The prevailing approach in recent years has been to utilize NLP architectures, such as vanilla Transformers, by transforming prediction problems into discrete input and output tokens. NLP, being at the forefront of generalization in human understanding, allows models to generalize across domains, including tasks that can be tokenized. As a result, robotics is now moving towards adopting Transformer models as well.

**Understanding Generalization and Transfer in Robotics**
RoboCat primarily focuses on studying generalization and transfer in machine learning. Generalization refers to the extent to which training on one domain benefits testing on another domain, especially when the two domains differ. Transfer learning, on the other hand, explores how much training on one domain can improve performance when fine-tuning on another domain. The paper emphasizes the importance of transfer learning in scenarios where zero-shot generalization is not yet strong. However, the boundary between transfer and generalization can be indistinct, as in-context adaptation blurs the line. The RoboCat project can be considered a sequel to the GATO paper, given the author list and infrastructure.

**Architecture and Research Contributions**
RoboCat shares similarities with the RT-1 model, as both involve learning a tokenizer for robotics images, tokenizing proprioception and future actions, and using a Transformer to predict future action tokens. However, RoboCat specifically focuses on challenging manipulation tasks, such as NIST-i gears, inverted pyramid, and tower building. The paper compares the transfer learning performance of RoboCat’s foundation models with Internet-scale foundation models. These projects are likely to be consolidated under the new Google and DeepMind re-organization.

The scientific contributions of the RoboCat paper include empirical data on unifying multiple robot embodiments in a single model, estimating cross-task transfer, evaluating the effectiveness of transferring learning recipes from simulation to reality, determining the amount of data required, conducting architecture and parameter scaling experiments, comparing different tokenization strategies for perception, and setting up reset-free automated evaluation for multi-task policies in the real world. The research involved a team of 39 authors who spent a year building infrastructure, collecting data, training models, evaluating performance, running baselines, and compiling the technical report.

**Impressive Experimental Setup**
A remarkable aspect of the RoboCat project is the extensive evaluation it underwent. The team evaluated the model on 253 tasks across simulation and real environments, involving multiple robots, including sim Sawyer, sim Panda, real Sawyer, real Panda, and real KUKA. Automating a single task for a single robot in the real world is already a significant challenge, and cross-robot transfer is rarely attempted due to the complexities involved. However, RoboCat demonstrates consistent results in all robots and action spaces, highlighting the team’s effort in detailing the training dataset, evaluation protocols, and task success rates. These tables and graphs de-risk many of the questions faced by other research projects, like the one at 1X, which aims to develop a “big model to do all the tasks.”

**The Impact of Action Space Choice**
The choice of action space greatly affects the performance of robotic systems. Task difficulty, measured in the number of samples required to learn the task, exponentially increases with episode length and the independent dimensions of the action space. According to Table 10 in the paper, RoboCat’s episode durations range from 20 to 120 seconds, which is 2-4 times longer than typical tasks in projects like BC-Z and SayCan. However, the low success rates of human teleoperators on tasks such as tower building suggest that improvement in autonomous performance can be achieved by enhancing the ease of teleoperation. A shorter demonstration time for tower building, like reducing it from 60 to 30 seconds, could potentially yield significant improvements compared to algorithmic optimizations.

**Flexible Modeling with Sequence Prediction**
RoboCat predicts cartesian velocities for the arm’s 4 or 6 degrees of freedom (DoF) and for the hand’s 1 DoF (parallel jaw gripper) or 8 DoF (3-finger). This allows a single neural network to handle action spaces of 5, 7, or 14-DoF, with varying proprioception sizes. Sequence modeling provides a simple yet universal interface for integrating observation and action spaces. While GATO and RT-1 introduced this concept, RoboCat further demonstrates that unifying multiple robot embodiments with a common interface can lead to positive transfer when training together. Instead of creating a prediction head for each embodiment, as done in HydraNet, scaling with different robot morphologies through mapping all outputs to a non-fixed length sequence is a more effective approach. Similar trends are emerging in perception, with VQA models converging towards this idea.

**The Potential of Visual Foundation Models (VFM)**
The paper addresses the question of whether visual foundation models, like GPT4 + Images (often referred to as VLMs), have the potential to achieve zero-shot robotics. The authors fine-tuned 59 VFMs pretrained on Internet-scale data for each task to investigate this. While this approach requires substantial resources, the results shed light on the possibility of leveraging visual models to tackle robotics challenges. If VLMs can effectively control motors without the need for extensive work with real robots, roboticists may shift their focus to computer vision and NLP benchmarks until models possess the necessary control capabilities.

**Conclusion**
The RoboCat paper provides valuable insights into transfer and generalization in robotics. It explores the unification of robot embodiments, the impact of different action spaces, and the potential of visual foundation models. The extensive evaluation and empirical data presented in the paper de-risk various research questions and open exciting possibilities for future advancements in the field.