hrmano

<p><em>This is a Plain English Papers summary of a research paper called <a href="https://www.aimodels.fyi/papers/arxiv/learning-humanoid-end-effector-control-open-vocabulary?utm_source=hackernoon&utm_medium=referral">Learning Humanoid End-Effector Control for Open-Vocabulary Visual Loco-Manipulation</a>. If you like these kinds of analysis, join <a href="https://www.aimodels.fyi/?utm_source=hackernoon&utm_medium=referral">AIModels.fyi</a> or follow us on <a href="https://x.com/aimodelsfyi">Twitter</a>.</em></p>

<h2>The generalization gap in robot manipulation</h2>
<p>Humanoid robots are surprisingly bad at picking things up. Watch a human reach for a mug in an office, then in a coffee shop, then in a home kitchen, and the task looks effortless. The hand adjusts. The eyes track. The body repositions. A humanoid robot, by contrast, can perform the same motion in one environment but fail completely in another.</p>
<p>The traditional approach to solving this seems obvious: collect lots of real robot videos performing manipulation tasks across many different environments. Let the robot learn from these demonstrations. But this strategy hits a wall so hard it's become the limiting factor in humanoid robotics. Each real-world trial is expensive. Hardware can break. The diversity required to cover different lighting, furniture heights, surface textures, and object types explodes the data collection burden. By the time you've gathered enough videos to show meaningful variation, you've invested months and thousands of dollars. And even then, show the robot a new coffee shop it hasn't seen, and performance collapses.</p>
<p>The result is that existing humanoid manipulation systems work reliably only in controlled laboratory settings. They're brittle. They don't generalize. This is the fundamental problem that HERO, a new system for loco-manipulation with humanoid robots, addresses head-on by rejecting the premise entirely.</p>
<h2>Simulation plus vision models</h2>
<p>Instead of trying to scale real-world data collection, what if the problem is framed differently? What if you use simulation for what it's genuinely good at, and large vision models for what they're good at?</p>
<p>Simulation is excellent at one thing: letting you practice control policies thousands of times without breaking hardware. A simulated robot can reach for the same object orientation 10,000 times with random variations, and each trial costs pennies. Meanwhile, large vision models trained on billions of internet images already understand object categories, spatial relationships, and scene context. They can look at an RGB-D image and reliably identify "that's a mug on a table, roughly at this position, roughly this orientation."</p>
<p>The key insight in HERO is that these aren't competing approaches, they're complementary. Vision models handle scene understanding and generalization to new environments. Simulation-trained control policies handle the precision of execution. By separating these concerns, the system should inherit the generalization properties of large vision models while maintaining the precision that comes from intensive simulation training.</p>
<p>This is a genuine departure from the prevailing approach. Most humanoid manipulation work tries to solve perception and control jointly on whatever real-world data can be collected. HERO flips this by asking: why not use proven, large-scale vision systems for what they've already learned, and simulation for what it does best?</p>
<h2>The kinematic drift problem</h2>
<p>Here's where the paper gets at something roboticists understand but rarely discuss explicitly: the gap between where a robot thinks its hand is and where it actually is.</p>
<p>Every robot has forward kinematics, mathematical models that predict "if I move joint A by 30 degrees and joint B by 45 degrees, my hand ends up at position X." In reality, this model is never perfect. Friction in joints, cable stretch, servo lag, sensor noise, and material deformation accumulate. Even small errors compound. Miss by 2 centimeters on one reach, and the next reach, which relies on the previous one, is now 4 centimeters off. After a few movements, the robot's understanding of where its hand is has drifted dramatically from reality. The robot reaches confidently toward empty air.</p>
<p>Traditional solutions to this problem are either brittle or expensive. Careful hardware calibration helps but degrades over time. Closed-loop vision tracking of the hand requires additional cameras and processing. Most existing humanoid systems accept kinematic drift as unavoidable noise and work around it with conservative safety margins.</p>
<p>HERO's insight is that this error is learnable and predictable. Instead of trying to eliminate the error entirely, train a system to predict and correct it. Instead of assuming the gap is random noise, treat it as systematic bias that can be learned from data.</p>
<h2>Residual learning for hand control</h2>
<p>The technical heart of the paper is elegant when stripped of jargon: train a neural network to predict the difference between where classical kinematics say the hand should be and where it actually ends up, then use that prediction to correct in real time.</p>
<p>This residual approach combines four components working together. Classical inverse kinematics converts "I want my hand at position X" into "move these joints by these amounts." This is well-established robotics but never perfect in reality. A learned forward kinematics model, trained in simulation, predicts where the hand actually ends up given joint positions. This captures systematic biases and errors that the classical model misses.</p>
<p>Then comes goal adjustment: if sensors see the hand is at position Y when theory predicts position X, nudge the target slightly to close that gap. Finally, replanning handles larger errors. If the error grows too large, don't keep pushing forward, replan the reaching motion mid-execution.</p>
<p>Together, these components reduce end-effector tracking error by 3.2 times compared to naive classical kinematics. This is substantial. The tracking error was the silent killer preventing reliable manipulation, and reducing it by more than 3 times directly translates to manipulation success rates that were previously impossible.</p>
<p>Critically, this approach trains entirely in simulation. The neural network learns residuals, small corrections to perfect kinematics, rather than the entire control problem. This kind of learning transfers well to real robots because the residuals capture first-order effects, not the full complexity of the real world. A learned correction for systematic arm bias should work whether the robot is in a lab or a coffee shop.</p>
<h2>Connecting vision to control</h2>
<p>Now HERO has an accurate hand control system. The next step is building a complete manipulation pipeline that takes a natural language command like "pick up the mug" and executes it reliably across different environments.</p>
<p>The full system works like this. A large-vocabulary vision model interprets the RGB-D image, understanding object categories it's never explicitly trained on and estimating 3D locations and orientations. Given the robot's current position, the object's location, and the height of the surface (which varies from 43 centimeter office desks to 92 centimeter counters), a loco-manipulation planner generates a whole-body motion that gets the robot's torso and hand into the right position. Then the residual-aware end-effector tracker executes that motion precisely. During execution, if the vision system updates its understanding of the object location, the system replans the reaching motion.</p>
<p>The power of this architecture is modularity. Each component can be developed, tested, and improved independently. The vision system doesn't need to know about kinematics. The trajectory planner doesn't need to know about learning. The hand-tracker does its job without worrying about scene understanding. If one component improves, the entire system benefits. This is sharply different from monolithic end-to-end approaches where improving perception might break control or vice versa.</p>
<p>This approach also connects naturally to related work like <a href="https://aimodels.fyi/papers/arxiv/learning-adaptive-neural-teleoperation-humanoid-robots-inverse?utm_source=hackernoon&utm_medium=referral">learning adaptive neural teleoperation for humanoid robots</a>, which explores how neural networks can refine classical control, and <a href="https://aimodels.fyi/papers/arxiv/now-you-see-that-learning-end-end?utm_source=hackernoon&utm_medium=referral">end-to-end learning for robot vision</a>, which shows the complementary challenge of visual perception in dynamic environments. HERO's modular decomposition offers a way to combine insights from both directions.</p>
<h2>Real-world validation</h2>
<p>A system that works in simulation means nothing. HERO's validation spans a spectrum designed to show both that the system works and which components matter most.</p>
<p>In simulation, the residual-aware controller reduces tracking error by 3.2 times as claimed. Then comes modular testing in real labs: Can the vision model identify objects it hasn't seen before? Can the loco-manipulation planner handle different surface heights? Finally, end-to-end real-world tests in actual environments: offices, coffee shops, homes. Real objects: mugs, apples, toys. Real variation: different table heights from 43 to 92 centimeters, different lighting, different clutter.</p>
<p>The system generalizes to environments and objects it never trained on. It reliably manipulates objects across the full range of surface heights tested. This is the proof that the modular approach works. By separating vision generalization from control learning, the system inherits the generalization properties of large vision models while maintaining the precision of simulation-trained control.</p>
<h2>A new approach to building humanoid robots</h2>
<p>HERO represents a fundamentally different way of thinking about humanoid robot training. The "collect more real data" approach has hit diminishing returns. This paper shows that simulation plus large models is a more scalable direction.</p>
<p>More broadly, HERO demonstrates that you don't need one monolithic end-to-end system solving perception and control jointly. Vision generalization and control precision are different problems better solved with different tools. And systems that learn to correct for reality rather than trying to perfectly model it are more robust and practical.</p>
<p>This approach could reshape how practical humanoid robots are built. Instead of expensive real-world data collection campaigns, future work might see improved sim-to-real transfer as simulation fidelity improves (though only for what matters for control, not perception). Vision models could become the default perception backbone, freeing roboticists to focus on control and planning. New robot systems could emerge that quickly adapt to new environments because generalization is built into the architecture.</p>
<p>The paper demonstrates this works for tabletop manipulation of everyday objects. The frontier ahead includes more complex manipulation tasks, two-handed reaching, long-horizon reasoning, objects that deform, and navigation combined with manipulation. HERO's modular approach gives a clear path forward for each. That clarity is itself the contribution: not just a working system, but a blueprint for how to think about the problem.</p>

<hr/><p><strong>Original post:</strong> <a href="https://www.aimodels.fyi/papers/arxiv/learning-humanoid-end-effector-control-open-vocabulary?utm_source=hackernoon&utm_medium=referral">Read on AIModels.fyi</a></p>

The Humanoid Robot Generalization Problem Has a New Blueprint

HackerNoon voters reveal which humanoid robot from CES 2026 is truly production-ready—and why practicality beat flashy demos.

This Humanoid Robot From CES 2026 is the Most Promising

Researchers have developed humanoid robots with the hope that they can help children with ASDs in practicing essential social behaviors.

Humanoid Robots Show Promise in Autism Therapy, but Challenges Remain

Explore how human-in-the-loop preferences refine reward functions in tasks like humanoid running and jumping. 

The Role of Human-in-the-Loop Preferences in Reward Function Learning for Humanoid Tasks

With the further development of technology, we may also see AI humanoid robots that project emotion or understand feelings. 

Are AI Humanoid’s Part of Our Future?

In the world of web 3.0, online communities are a vital part of building and sustaining projects.

Dealing With Spammy Bots - How Humanode Uses Sybil-Resistance to Bash the Bad Bots in Discord

On AI Day 2022, Tesla introduced the Optimus prototype, an improvement from the human in a costume a year prior. How close to sentience is the new bot?

Tesla's Optimus Humanoid and the Future of Sentient AI

This Tesla Humanoid Robot Optimus review will examine the features and functionality of Elon Musk’s latest invention.

MashRadar Reviews: The Tesla Prototype Humanoid Robot Optimus

While many older movies set in the future paint the 2020s as a futuristic megalopolis, the reality is that we have yet to transform into a society that sees flying cars and teleportation as normal parts of everyday life. However, new innovations and developments in healthcare, such as ambulatory surgery centers and telehealth, are slowly starting to usher us into a new age in which highly advanced technology is becoming more and more accepted and utilized. 

Should Telsa Humanoids Be Our Future Nurses?

Humanoids, or robots designed to resemble humans in form and function, have been a staple of science fiction for centuries.

The Evolution of Humanoids and What It Holds for the Future

Victor Smirnov is co-founder of Humanode, which is changing the power structure of the crypto space. He says that the true adoption of blockchain is possible.

Blockchain Lacks True Decentralization - Victor Smirnov, Co Founder Humanode

Robo-C is an anthropomorphic robot built to interact with humans.

An Honest Tale of How We Built a Humanoid Robot

<em>In very crude terms, we are a combination of our intellect (measured by Intelligence Quotient) and our emotions (measured by an Emotional Quotient). I say crude specifically because IQ and EQ’s are formulas that capture a limited aspect of our humanity. On the other hand of the spectrum, we have a least understood concept-consciousness. However, without EQ, it’s hard to imagine AI becoming more human every day. This article looks at how different businesses are busy putting a human mask on Artificial Intelligence to make robots seem more natural to actual people. While AI is undergoing a human transformation, it is also transforming the client experience and raising client expectations every day. This article looks at the evolution of AI and ends by asking a bigger question: are humanoids just another mile stone on the high way to cyborg town?</em>

Honey, The Humanoids Are Here !!

Sophia Hanson is Elon Musk’s worst nightmare.

This humanoid wants to replace you…and why it can’t

Explore the first Agentic AI governance frameworks from Singapore, UC Berkeley, and industry leaders, highlighting risks, human oversight, and safe deployment 

Agentic AI Governance Frameworks 2026: Risks, Oversight, and Emerging Standards

Humanoid robots hit an "Iron Wall" of energy. We must offload physics to an "External Cerebellum" via 5G to solve movement.

The Iron Wall of Robotics: Why Physics Will Defeat AI Hype

Agentic AI replaces passive chatbots with goal-driven agents; MCP standardizes tools, enabling safe, scalable human-AI collaboration.

MCP Explained: The Protocol That Unblocked Real AI Agent Ecosystems

Are Antivirus Software Still a Thing?

AI - Should we Be Afraid? 3 Years Later

A founder’s critique of the productivity-wellness industry—from gamified focus apps to self-optimization culture—and why our minds crave clarity, not performanc

Why I’m Tired of Productivity Wellness (And What It’s Doing to Our Minds)

The space-themed event offers its largest prize pool to date, totaling $450,000 USDC. Daily prizes include a Unitree R1 Humanoid Robot, a World Trip Prize, and 

Deribit and SignalPlus Launch 2025 Trading Competition, Featuring a $450,000 USDC Prize Pool

Humanoid robots could enter society within five years, WEF experts say — raising urgent questions about caregiving, regulation, and a world built for machines.

WEF Experts: Walking, Talking Robots Are Just Five Years Away

ROBA is breaking robotics out of closed corporate silos by ensuring full ownership of your data, your assets, and your revenue.

Roba Shatters Closed Systems: The People's Robotics Platform Launches To Dominate $218b Industry

Meet 10 influential women redefining AI—leaders, founders, and innovators shaping technology with creativity, ethics, and bold vision.

HRMANO

https://www.hrmano.com

2-10 emps

Since n.d.

hrmano's stories on HackerNoon

The Humanoid Robot Generalization Problem Has a New Blueprint

This Humanoid Robot From CES 2026 is the Most Promising

Humanoid Robots Show Promise in Autism Therapy, but Challenges Remain

The Role of Human-in-the-Loop Preferences in Reward Function Learning for Humanoid Tasks

Are AI Humanoid’s Part of Our Future?

Dealing With Spammy Bots - How Humanode Uses Sybil-Resistance to Bash the Bad Bots in Discord

Tesla's Optimus Humanoid and the Future of Sentient AI

MashRadar Reviews: The Tesla Prototype Humanoid Robot Optimus

Should Telsa Humanoids Be Our Future Nurses?

The Evolution of Humanoids and What It Holds for the Future

Blockchain Lacks True Decentralization - Victor Smirnov, Co Founder Humanode

An Honest Tale of How We Built a Humanoid Robot

Honey, The Humanoids Are Here !!

This humanoid wants to replace you…and why it can’t

Agentic AI Governance Frameworks 2026: Risks, Oversight, and Emerging Standards

The Iron Wall of Robotics: Why Physics Will Defeat AI Hype

MCP Explained: The Protocol That Unblocked Real AI Agent Ecosystems

Are Antivirus Software Still a Thing?

AI - Should we Be Afraid? 3 Years Later

Why I’m Tired of Productivity Wellness (And What It’s Doing to Our Minds)

Deribit and SignalPlus Launch 2025 Trading Competition, Featuring a $450,000 USDC Prize Pool

WEF Experts: Walking, Talking Robots Are Just Five Years Away

Roba Shatters Closed Systems: The People's Robotics Platform Launches To Dominate $218b Industry

10 Influential Women in The AI Space

Representing this company? Click here to claim and customize this page!

Reviews About hrmano

hrmano WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps