Efficient Data Generation and Reasoning for Embodied Robot Navigation and Manipulation

Associate Professor Xiaodan Liang
Sun Yat-sen University, China

Large language models (LLMs) have shown great potential in reasoning and interpretability for embodied intelligent robot navigation and manipulation tasks. This presentation focuses on the development of large-scale models for embodied robot navigation and manipulation in general scenarios. We will discuss novel strategies within the Navigation Chain-of-Thought (NavCoT) designed by our research group, enabling LLMs to make autonomous navigation decisions, thereby significantly reducing domain gaps. Moreover, understanding and following natural language instructions in complex real-world environments pose significant challenges for robots designed for general purposes. Autonomous agents must possess the ability to self-correct their planning based on feedback from the surrounding environment. We introduce a novel zero-shot framework called CorNav, which utilizes large-scale language models for decision-making. Additionally, the research group has developed a 3D simulator based on UE5 rendering of real scenes and a benchmark called NavBench to evaluate the effectiveness and generalization capability of navigation agents in zero-shot multi-task settings. Finally, the report discusses the future development trends of embodied intelligent agents.

Biosketch

Xiaodan Liang, associate professor at Sun Yat-sen University, YiXian Scholar, National Ten Thousand Talent Program Young Top-notch Talent, IEEE Senior Member. Her research interests include multimodal vision-language understanding, digital human generation and animation, interpretable AI, and causal inference in machine learning models. She has been cited over 24,000 times on Google Scholar. Currently serving as Associate Editor for Image and Vision Computing and Neural Networks journals, She serves as Area Chairs for top conferences such as CVPR, ICML, ICCV, NeurIPS, ICLR, ECCV, ACM MM and served as Ombud chair of CVPR 2023. She has received numerous awards including ACM China Rising Star Nomination Award, Alibaba DAMO Academy Orange Award, CSIG Shi Qingyun Young Female Scientist Award, Wu Wenjun Artificial Intelligence Outstanding Youth Award, China Association for Science and Technology Young Talent Nurturing Program Awardee, First Prize of China Graphics Society Science and Technology, CCF Outstanding Doctoral Dissertation Award, and ACM China Outstanding Doctoral Dissertation Award.