Vision-Based Robot Localization, Navigation, and Control

Professor Hesheng Wang
Shanghai Jiao Tong University, China

This report focuses on the two core functions of service robots: mobility and manipulation. First, an overview of the current status of the service robot industry and technological development, along with the challenges faced, is provided. Next, the report introduces the key achievements of the team's long-term efforts in addressing the core technical challenges of mobility and manipulation. To tackle the issue of velocity perception and localization failure caused by the complex dynamics of non-holonomic mobile robots and dynamic environmental interferences, a computation method that integrates attention-based back-end optimization and explicit occlusion handling is proposed. This method achieves robust perception and localization of mobile robots through visual fusion in complex and large scenes. To address the problem of traditional calibrated control algorithms being prone to failure in uncalibrated environments, an adaptive visual servoing framework is developed that entirely relies on visual feedback without prior environmental information. This framework solves the challenge of high-precision robotic operations without calibration. A practical and versatile vision-based method system has been established, elevating the critical technological level of service robots.

Biosketch

Hesheng Wang is a Distinguished Professor of Shanghai Jiao Tong University, China. He has published more than 200 papers in refereed journals and conferences. Dr. Wang is an Associate Editor of IEEE Transactions on Automation Science and Engineering, IEEE Robotics and Automation Letters, Robotic Intelligence and Automation and the International Journal of Humanoid Robotics, a Senior Editor of the IEEE/ASME Transactions on Mechatronics, an Editor of Conference Editorial Board (CEB) of IEEE Robotics and Automation Society. He served as an Associate Editor of the IEEE Transactions on Robotics from 2015 to 2019, a Technical Editor of the IEEE/ASME Transactions on Mechatronics from 2020 to 2023. He was the General Chair of IEEE ROBIO 2022 and IEEE RCAR 2016, and the Program Chair of the IEEE ROBIO 2014 and IEEE/ASME AIM 2019. He will be the General Chair of IEEE/RSJ IROS 2025.