- The FLEX system developed by MIT helps robots understand human intent.
- FLEX uses large language models to identify potential risks and improve robot behavior.
- The system updates robot decision-making through three phases: addition, generalization, and masking.
- Research shows that FLEX can learn human preferences in various contexts.
The Massachusetts Institute of Technology (MIT) has developed an AI-driven system designed to assist robots in understanding human intent, such as the underlying request when a person asks a robot to remove a coffee cup from a cluttered table. This system, named FLEX (Flexible Language Guided Representation Editing for Adaptive Robot Behavior), will be showcased at this month's annual Robotics Learning Conference.
Testing and Applications of the FLEX System
Researchers tested FLEX using robotic arms in simulated tasks. In one example, the robot was asked to transport a coffee cup from a cluttered table. The user said, "Don't spill it on the laptop!" FLEX identified the potential risk of coffee spilling on the laptop and connected the liquid to electronic devices using a large language model (LLM). Subsequently, the robot avoided the laptop and nearby smartphone, even though the user did not mention the smartphone in the instructions. This research was led by aerospace assistant professor Andreea Bobu, with co-authors including postdoctoral researcher Nathan Dennler from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and master's student Audrey.
Lee SM ’26 and New York University student Ilia Sucholutsky.
Large language models (LLMs), trained on vast amounts of internet text, can associate objects and situations with relevant information. For instance, in the event of coffee or water spilling, LLMs connect these to potential damage to electronic devices. FLEX utilizes these associations to help robots better predict problems that may arise from objects not explicitly mentioned by the user. The system operates in three phases: addition, generalization, and masking. In the addition phase, FLEX identifies objects and features to avoid. For example, if a user says, "Don't drop the bowling ball on my cup!"
, the cup will be added to the list of objects the robot needs to avoid. In the generalization phase, FLEX uses the GPT-4o language model to generalize the objects and features added in the previous phase. For example, the cup will be classified as a fragile item, prompting the robot to avoid all other fragile items as well. Finally, in the masking phase, the robot's standard computer vision system is used to identify objects around it. Objects that no longer exist will be removed from the robot's feature list, enabling the robot to update its decisions in real-time based on environmental changes. According to the researchers, the entire process takes about five seconds.
Learning Diverse Household Preferences
The team tested FLEX in a simulated environment with three activities, including serving coffee, moving candles, and carrying a bowling ball. The robot was able to follow instructions involving laptops, stacks of paper, and cups, and extend these to include similar objects (such as electronic devices, flammable materials, and fragile items). The research was partially supported by the MIT Generative AI Impact Alliance and the MIT Schwarzman College of Computing's Social and Ethical Responsibility in Computing program. Bobu stated, "Household preferences are extremely diverse. Therefore, robots should learn people's preferences in various contexts with just a few instructions."
FLEX can help robots learn people's preferences in different contexts with just a few instructions.
Impact of the FLEX System on Robotics Technology
The development of the FLEX system marks a significant advancement in robotics technology, enabling robots to more flexibly understand and respond to human needs. Through the application of large language models, robots can not only perform specific tasks but also anticipate potential risks, which is especially important for applications in daily life. As household environments become more diverse, FLEX can quickly learn and adapt to preferences in different contexts, promoting broader applications of robotics technology in both home and commercial settings, thereby enhancing the quality and efficiency of human-robot interactions.