A humanoid robot walks into a room it has never seen, finds a chair and sits down. It can also sidestep through a partially open door, crawl beneath a table and push a box. For Apple’s robotics researchers, the unusual part is how it learned: without anyone remotely operating the robot to collect the training examples.
The team has introduced HALO, a humanoid autonomy system trained largely on synthetic vision, language and motion data. In a paper submitted on November 4, the researchers report a 91% average success rate across 360 single-skill trials in six real-world environments that were not mapped or scanned in advance.
The experiments used a Unitree G1 humanoid equipped with a head-mounted camera and an onboard computing enclosure. HALO combines three components. A vision-language model translates a request into subtasks, a motion generator plans whole-body movements from camera images, and a lower-level tracker converts those plans into joint commands. The system can operate locally on the robot, with its planning models running on a carried Mac mini.
Instead of recording hundreds of hours of human demonstrations, the researchers generated approximately 300 hours of paired video, language and motion data across 1,000 procedurally created environments. They placed simulated movements into those rooms, rendered camera views and used the resulting examples to train the robot’s motion generator. A separate reinforcement-learning process trained the motion tracker.
The six evaluated skills included navigating to a named object, sitting, passing sideways through a half-open door, crouching, crawling and pushing a box. Each was tested ten times in each of six physical environments. The reported 91% average applies to those individual skills, not to general household autonomy.
Combining actions proved more difficult. Three multi-step sequences achieved an average success rate of 70% in a separate set of tests. The researchers identified failures when the next target was outside the robot’s field of view, indicating that additional search and recovery behaviors are needed.
HALO also has important limits. New skills still require people to specify prompts and motion keyframes, and the tested repertoire does not demonstrate dexterous hand manipulation. The paper does not report a head-to-head performance comparison with other published systems under the same evaluation conditions.
Full video here: https://apple-aiml-research.github.io/ml-halo/
