Inteligência Artificial 9 min

MIT AI creates virtual worlds to train robots

Lucas Oliveira

Training a robot to work in a kitchen, move around a hotel or organize objects in a room seems simple until the first obstacle arises: the real world is full of variations. A glass changes location, a door opens in the unexpected direction and a piece of furniture completely changes the available path. Teaching each situation manually costs time, money and many hours of testing.

Researchers from the Massachusetts Institute of Technology (MIT) and the Toyota Research Institute have presented an alternative to this bottleneck. The system SceneSmith uses artificial intelligence agents to create three-dimensional, complete and physically functional virtual environments in which robots can practice tasks before being turned on in the real world.

The proposal goes beyond generating a beautiful image of a house. The environments need to contain walls, furniture and objects positioned in a coherent way, as well as physical properties that allow opening cabinets, moving utensils and testing different routes. It's an attempt to turn generative AI into a kind of set design team for robots.

What is SceneSmith

SceneSmith is a ready-made internal scene generation framework for robotics simulators. Based on a text description, the system can create spaces such as kitchens, bedrooms, restaurants, offices, garages and stores. These virtual locations serve as training camps where robotic controllers try out tasks without putting real machines, people or equipment at risk.

According to presentation published by MIT CSAIL, the project produced more than 1,300 scenes. Environments are designed to be diverse and detailed, reducing reliance on small, repetitive libraries that limit many simulators.

The work was also presented at the International Conference on Machine Learning (ICML). This is important because it puts the proposal before the scientific community and allows methods, results and limitations to be examined beyond the visual demonstration.

Three AI agents work as a team

Instead of handing over all creation to a single model, the researchers divided the process between three agents. Each one performs a specific function:

  • Designer: proposes the plan, chooses the elements and organizes the objects in the environment.
  • Critical: checks whether the scenario makes sense and points out inconsistencies, such as a bathtub placed in the middle of the room.
  • Orchestrator: monitors the conversation between other agents, decides when a step needs to be redone and ends the generation when the result reaches the expected level.

All three use a multimodal model capable of interpreting text and visual information. The process starts with the general structure of the space and progresses in layers: walls, furniture, decorative objects and, finally, items that the robot can manipulate. Then, a simulation engine adds the physical rules.

This division of functions is reminiscent of a human project team. One person creates, another reviews and a third coordinates decisions. The difference is that agents repeat the cycle automatically and are able to produce many variations of the same type of place.

Why robots need virtual worlds

Language models learn from large volumes of text. Vision systems use image and video banks. For a robot, however, observing is not enough: it needs to understand space, movement, contact and consequence.

A machine tasked with taking a fruit to a cutting board must calculate where to hold the object, which trajectory to follow and how much force to apply. You also need to avoid collisions and adapt the plan when the environment changes. Collecting this kind of experience with physical robots alone would be slow and expensive.

Simulation allows you to repeat a task thousands of times, change the shape of the room and discover faults without breaking equipment. The problem is that poor virtual settings teach narrow behaviors. If all simulated kitchens are practically the same, the robot may fail to find a different configuration.

This is where SceneSmith tries to move forward. According to the researchers, their scenes contained up to six times more objects than previous methods. More items mean more obstacles, possibilities and combinations for training.

Tests showed more than visual realism

A 3D scene can look convincing and still be useless for robotics. A door without hinges, for example, works as decoration, but is not useful for teaching a machine to open a cabinet. Therefore, the team evaluated both the appearance and physical behavior of the environments.

In one experiment, a controller trained primarily on real-world data was placed in a scene it had never seen before. He was given the task of taking an apple out of a bowl and placing it on a cutting board. The virtual robot completed the action, indicating that the environment maintained enough similarities with real situations for the skill to be transferred.

The researchers also remotely drove robots into the scenarios to open cabinets, store bottles and move between rooms. In another test, an agent evaluated action plans generated for 100 different spaces. Agreement between the AI ​​evaluator and human participants exceeded 99% when identifying faulty strategies.

In a comparison with previous approaches, more than 200 participants preferred SceneSmith scenes in more than 90% of realism ratings. These numbers don't mean that the robotic training problem is solved, but they do show that the system produces something more useful than a simple digital model.

Virtual objects are also given physical properties

SceneSmith can generate items that were not previously available in a fixed library. When ordering a service cart, for example, the system first creates a visual representation, converts the result into a 3D model and adds features such as mass, friction and inertia.

This step brings AI generation closer to engineering needs. For a robot, it is not enough to recognize that something is a cart. It needs to predict the effort required to push it, how the wheels respond and what happens when there is contact with another object.

The combination of generative models and physics engines indicates an important change. AI stops just producing content for people and starts creating training data for other machines. This movement accompanies the expansion of so-called physical AI, in which algorithms need to deal with concrete consequences, not just on-screen responses.

What are the limitations

The system is still far from instantly generating any environment. Detailed creation of a single scene can take several hours as agents need to assemble and review numerous objects. The computational cost tends to grow as environments become more complex.

Another limitation involves deformable materials. Sponges, fabrics, cables and foods change shape during manipulation and require more difficult physical models. The team intends to explore this category when there are suitable 3D libraries to support testing.

There is also the well-known distance between simulation and reality. Lighting, wear and tear, small sensor errors, and unpredictable movements can affect a physical robot in ways that the virtual world doesn't perfectly replicate. Thus, simulation reduces the amount of real testing, but does not eliminate validation outside the computer.

What this research can change in practice

Automatically generated environments can accelerate the development of robots for logistics, hospitality, factories, hospitals and homes. A company could test the same controller in hundreds of virtual warehouses before placing it near employees and merchandise.

For domestic robots, diversity is even more relevant. No two houses are exactly alike. Training in small kitchens, tight hallways and furniture positioned in different ways can help the machine react better when it finds a new home.

The research also reinforces a trend already observed in AI runs closer to the user and devices: models are no longer just generating responses and are taking on planning, evaluation and coordination tasks. In SceneSmith, different agents collaborate to deliver a result that a simulator can use directly.

The most interesting advance may not be a specific robot, but the creation of an experience factory. The more varied and physically coherent virtual worlds are, the greater the chances of preparing robots for the unpredictability of the real world.

FAQ

SceneSmith controls real robots

Not directly. It creates 3D environments for training and evaluating robotic controllers. After simulation, the systems still need to be tested with real machines.

Scenes are just 3D images

No. They include objects with physical properties and articulated elements, allowing virtual robots to move items, open cabinets and navigate space.

The technology is now commercially available

SceneSmith is a research project. THE official project website brings together examples and technical information, while the scientific article describes the method and experiments.

Sources consulted: MIT Computer Science and Artificial Intelligence Laboratory, official SceneSmith page and scientific article by the researchers.

Continue lendo