Publications

    For the up-to-date publication list, please visit the Google Scholar page.

    * Equal contribution.  † Equal advising.

    2024

    PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
    Soroush Nasiriany*, Fei Xia*, Wenhao Yu*, Ted Xiao*, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tsang-Wei Edward Lee, Kuang-Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, Brian Ichter*
    International Conference on Machine Learning (ICML), July 2024

    DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
    DROID Collaboration
    Robotics: Science and Systems (RSS), July 2024

    InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning
    Muzhi Han, Yifeng Zhu, Song-Chun Zhu, Ying Nian Wu, Yuke Zhu
    Robotics: Science and Systems (RSS), July 2024

    DrEureka: Language Model Guided Sim-To-Real Transfer
    Jason Ma*, William Liang*, Hungju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, Dinesh Jayaraman
    Robotics: Science and Systems (RSS), July 2024

    RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
    Soroush Nasiriany, Abhiram Maddukuri*, Lance Zhang*, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, Yuke Zhu
    Robotics: Science and Systems (RSS), July 2024

    ORION: Vision-based Manipulation from Single Human Video with Open-World Object Graphs
    Yifeng Zhu, Arisrei Lim, Peter Stone, Yuke Zhu
    Technical report arXiv:2405.20321, May 2024

    Doduo: Dense Visual Correspondence from Unsupervised Semantic-Aware Flow
    Zhenyu Jiang, Hanwen Jiang, Yuke Zhu
    IEEE International Conference on Robotics and Automation (ICRA), May 2024

    Model-Based Runtime Monitoring with Interactive Imitation Learning
    Huihan Liu, Shivin Dass, Roberto Martín-Martín, Yuke Zhu
    IEEE International Conference on Robotics and Automation (ICRA), May 2024

    Open X-Embodiment: Robotic Learning Datasets and RT-X Models
    Open X-Embodiment Collaboration
    IEEE International Conference on Robotics and Automation (ICRA), May 2024
    Best Conference Paper Award

    LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discovery
    Weikang Wan, Yifeng Zhu*, Rutav Shah*, Yuke Zhu
    IEEE International Conference on Robotics and Automation (ICRA), May 2024

    AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents
    Jake Grigsby, Linxi Fan, Yuke Zhu
    International Conference on Learning Representations (ICLR), May 2024
    Spotlight Presentation

    Eureka: Human-Level Reward Design via Coding Large Language Models
    Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan†, Anima Anandkumar†
    International Conference on Learning Representations (ICLR), May 2024

    Few-View Object Reconstruction with Unknown Categories and Camera Poses
    Hanwen Jiang, Zhenyu Jiang, Kristen Grauman, Yuke Zhu
    International Conference on 3D Vision (3DV), March 2024
    Oral Presentation

    Granger Causal Interaction Skill Chains
    Caleb Chuck, Kevin Black, Aditya Arjun, Yuke Zhu, Scott Niekum
    Transactions on Machine Learning Research (TMLR), March 2024

    Voyager: An Open-Ended Embodied Agent with Large Language Models
    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan†, Anima Anandkumar†
    Transactions on Machine Learning Research (TMLR), March 2024

    PRIME: Scaffolding Manipulation Tasks with Behavior Primitives for Data-Efficient Imitation Learning
    Tian Gao, Soroush Nasiriany, Huihan Liu, Quantao Yang, Yuke Zhu
    Technical report arXiv:2403.00929, March 2024

    Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning
    Zizhao Wang*, Caroline Wang*, Xuesu Xiao, Yuke Zhu, Peter Stone
    AAAI Conference on Artificial Intelligence (AAAI), February 2024
    Oral Presentation

    2023

    Foundation Models in Robotics: Applications, Challenges, and the Future
    Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, Brian Ichter, Danny Driess, Jiajun Wu, Cewu Lu, Mac Schwager
    Technical report arXiv:2312.07843, December 2023

    Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation
    Mingyo Seo, Steve Han, Kyutae Sim, Seung Hyeon Bang, Carlos Gonzalez, Luis Sentis, Yuke Zhu
    International Conference on Humanoid Robots (Humanoids), December 2023
    Oral Presentation

    LIBERO: Benchmarking Knowledge Transfer in Lifelong Robot Learning
    Bo Liu*, Yifeng Zhu*, Chongkai Gao*, Yihao Feng, Qiang Liu, Yuke Zhu, Peter Stone
    NeurIPS 2023 Datasets and Benchmarks Track, December 2023

    Cross-Episodic Curriculum for Transformer Agents
    Lucy Xiaoyang Shi*, Yunfan Jiang*, Jake Grigsby, Linxi Fan†, Yuke Zhu†
    Conference on Neural Information Processing Systems (NeurIPS), December 2023

    Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
    Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Ming-Yu Liu, Yuke Zhu, Mohammad Shoeybi, Bryan Catanzaro, Chaowei Xiao, Anima Anandkumar
    Conference on Empirical Methods in Natural Language Processing (EMNLP), December 2023

    Learning Generalizable Manipulation Policies with Object-Centric 3D Representations
    Yifeng Zhu, Zhenyu Jiang, Peter Stone, Yuke Zhu
    Conference on Robot Learning (CoRL), November 2023

    MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
    Ajay Mandlekar, Soroush Nasiriany*, Bowen Wen*, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, Dieter Fox
    Conference on Robot Learning (CoRL), November 2023

    MUTEX: Learning Unified Policies from Multimodal Task Specifications
    Rutav Shah, Roberto Martín Martín†, Yuke Zhu†
    Conference on Robot Learning (CoRL), November 2023

    MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
    Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu†, Anima Anandkumar†
    Conference on Robot Learning (CoRL), November 2023
    Best Paper Award Finalist

    Interactive Robot Learning from Verbal Correction
    Huihan Liu, Alice Chen, Yuke Zhu, Adith Swaminathan, Andrey Kolobov, Ching-An Cheng
    Technical report arXiv:2310.17555, October 2023

    Symbolic State Space Optimization for Long Horizon Mobile Manipulation Planning
    Xiaohan Zhang, Yifeng Zhu, Yan Ding, Yuqian Jiang, Yuke Zhu, Peter Stone, Shiqi Zhang
    International Conference on Intelligent Robots and Systems (IROS), October 2023

    ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation
    Bokui Shen*, Zhenyu Jiang*, Christopher Choy, Silvio Savarese, Leonidas J. Guibas, Anima Anandkumar, Yuke Zhu
    International Journal of Robotics Research (IJRR), July 2023

    VIMA: General Robot Manipulation with Multimodal Prompts
    Yunfan Jiang, Agrim Gupta*, Zichen Zhang*, Guanzhi Wang*, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu†, Linxi Fan†
    International Conference on Machine Learning (ICML), July 2023

    Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment
    Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, Yuke Zhu
    Robotics: Science and Systems (RSS), July 2023
    Best Paper Award Finalist

    Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids
    Wei Dong, Chris Choy, Charles Loop, Or Litany, Yuke Zhu, Anima Anandkumar
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2023

    Ditto in the House: Building Articulated Models of Indoor Scenes through Interactive Perception
    Cheng-Chun Hsu, Zhenyu Jiang, Yuke Zhu
    IEEE International Conference on Robotics and Automation (ICRA), May 2023

    Learning to Walk by Steering: Perceptive Quadrupedal Locomotion in Dynamic Environments
    Mingyo Seo, Ryan Gupta, Yifeng Zhu, Alexy Skoutnev, Luis Sentis, Yuke Zhu
    IEEE International Conference on Robotics and Automation (ICRA), May 2023

    2022

    Learning and Retrieval from Prior Data for Skill-based Imitation Learning
    Soroush Nasiriany, Tian Gao, Ajay Mandlekar, Yuke Zhu
    Conference on Robot Learning (CoRL), December 2022

    VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors
    Yifeng Zhu, Abhishek Joshi, Peter Stone, Yuke Zhu
    Conference on Robot Learning (CoRL), December 2022

    MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
    Linxi Fan, Guanzhi Wang*, Yunfan Jiang*, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu†, Anima Anandkumar†
    NeurIPS 2022 Datasets and Benchmarks Track, November 2022
    Outstanding Paper Award

    Pre-Trained Language Models for Interactive Decision-Making
    Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, Yuke Zhu
    Conference on Neural Information Processing Systems (NeurIPS), November 2022
    Oral Presentation

    Causal Dynamics Learning for Task-Independent State Abstraction
    Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, Peter Stone
    International Conference on Machine Learning (ICML), July 2022
    Long Presentation

    ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation
    Bokui Shen, Zhenyu Jiang, Christopher Choy, Silvio Savarese, Leonidas J. Guibas, Anima Anandkumar, Yuke Zhu
    Robotics: Science and Systems (RSS), June 2022
    Best Student Paper Award Finalist

    COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehicles
    Jiaxun Cui*, Hang Qiu*, Dian Chen, Peter Stone, Yuke Zhu
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022

    Ditto: Building Digital Twins of Articulated Objects from Interaction
    Zhenyu Jiang, Cheng-Chun Hsu, Yuke Zhu
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022
    Oral Presentation

    Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions
    Huaizu Jiang∗ , Xiaojian Ma*, Weili Nie, Zhiding Yu, Yuke Zhu, Anima Anandkumar
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022
    Oral Presentation

    Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
    Soroush Nasiriany, Huihan Liu, Yuke Zhu
    IEEE International Conference on Robotics and Automation (ICRA), May 2022
    Outstanding Learning Paper Award

    OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation
    Josiah Wong, Viktor Makoviychuk, Anima Anandkumar, Yuke Zhu
    IEEE International Conference on Robotics and Automation (ICRA), May 2022

    Visually Grounded Task and Motion Planning for Mobile Manipulation
    Xiaohan Zhang, Yifeng Zhu, Yan Ding, Yuke Zhu, Peter Stone, Shiqi Zhang
    IEEE International Conference on Robotics and Automation (ICRA), May 2022

    RelViT: Concept-Guided Vision Transformer for Visual Relational Reasoning
    Xiaojian Ma, Weili Nie, Zhiding Yu, Huaizu Jiang, Chaowei Xiao, Yuke Zhu, Song-Chun Zhu, Anima Anandkumar
    International Conference on Learning Representations (ICLR), April 2022

    Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation
    Yifeng Zhu, Peter Stone, Yuke Zhu
    IEEE Robotics and Automation Letters (RA-L), January 2022

    2021

    Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
    Youngwoon Lee, Joseph J. Lim, Anima Anandkumar, Yuke Zhu
    Conference on Robot Learning (CoRL), November 2021

    What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
    Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, Roberto Martín-Martín
    Conference on Robot Learning (CoRL), November 2021
    Oral Presentation

    DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision
    Shiyi Lan, Zhiding Yu, Christopher Choy, Subhashree Radhakrishnan, Guilin Liu, Yuke Zhu, Larry S. Davis, Anima Anandkumar
    International Conference on Computer Vision (ICCV), October 2021

    Learning Generalizable Skills via Automated Generation of Diverse Tasks
    Kuan Fang, Yuke Zhu, Silvio Savarese, Li Fei-Fei
    Robotics: Science and Systems (RSS), July 2021

    Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations
    Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, Yuke Zhu
    Robotics: Science and Systems (RSS), July 2021

    SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
    Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, Anima Anandkumar
    International Conference on Machine Learning (ICML), July 2021

    Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition
    Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, Anima Anandkumar
    International Conference on Machine Learning (ICML), July 2021
    Long Talk

    Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning
    Anuj Mahajan, Mikayel Samvelyan, Lei Mao, Viktor Makoviychuk, Animesh Garg, Jean Kossaifi, Shimon Whiteson, Yuke Zhu, Anima Anandkumar
    International Conference on Machine Learning (ICML), July 2021

    MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
    Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A. Lee, Yuke Zhu, Ruslan Salakhutdinov, Louis-Philippe Morency
    NeurIPS 2021 Datasets and Benchmarks Track, July 2021

    Fast Uncertainty Quantification for Deep Object Pose Estimation
    Guanya Shi, Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Fabio Ramos, Anima Anandkumar, Yuke Zhu
    IEEE International Conference on Robotics and Automation (ICRA), May 2021

    Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
    Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Yuke Zhu
    IEEE International Conference on Robotics and Automation (ICRA), May 2021

    Deep Affordance Foresight: Planning Through What Can Be Done in the Future
    Danfei Xu, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Silvio Savarese, Li Fei-Fei
    IEEE International Conference on Robotics and Automation (ICRA), May 2021

    Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors
    Michelle A. Lee, Matthew Tan, Yuke Zhu, Jeannette Bohg
    IEEE International Conference on Robotics and Automation (ICRA), May 2021

    Emergent Hand Morphology and Control from Optimizing Robust Grasps of Diverse Objects
    Xinlei Pan, Animesh Garg, Anima Anandkumar, Yuke Zhu
    IEEE International Conference on Robotics and Automation (ICRA), May 2021

    Learning Multi-Arm Manipulation Through Collaborative Teleoperation
    Albert Tung, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, Silvio Savarese
    IEEE International Conference on Robotics and Automation (ICRA), May 2021
    Best Multi-Robotic Systems Paper Award Finalist

    Adaptive Procedural Task Generation for Hard-Exploration Problems
    Kuan Fang, Yuke Zhu, Silvio Savarese, Li Fei-Fei
    International Conference on Learning Representations (ICLR), May 2021

    2020

    Human-in-the-Loop Imitation Learning using Remote Teleoperation
    Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, Silvio Savarese
    Technical report arXiv:2012.06733, December 2020

    Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning
    Weili Nie, Zhiding Yu, Lei Mao, Ankit B. Patel, Yuke Zhu, Animashree Anandkumar
    Conference on Neural Information Processing Systems (NeurIPS), December 2020
    Spotlight Presentation

    Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion
    Xingye Da, Zhaoming Xie, David Hoeller, Byron Boots, Anima Anandkumar, Yuke Zhu, Buck Babich, Animesh Garg
    Conference on Robot Learning (CoRL), November 2020

    robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
    Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, Yifeng Zhu
    Technical report arXiv:2009.12293, September 2020

    RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition
    Linxi Fan*, Shyamal Buch*, Guanhzi Wang, Ryan Cao, Yuke Zhu, Juan Carlos Niebles, Li Fei-Fei
    European Conference on Computer Vision (ECCV), August 2020
    * indicates equal contribution

    OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
    Hongyu Ren, Yuke Zhu, Jure Leskovec, Anima Anandkumar, Animesh Garg
    Conference on Uncertainty in Artificial Intelligence (UAI), August 2020

    DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs
    Yunbo Wang*, Bo Liu*, Jiajun Wu, Yuke Zhu, Simon S. Du, Li Fei-Fei, Joshua B. Tenenbaum
    International Joint Conference on Artificial Intelligence (IJCAI), July 2020
    * indicates equal contribution

    KETO: Learning Keypoint Representations for Tool Manipulation
    Zengyi Qin, Kuan Fang, Yuke Zhu, Li Fei-Fei, Silvio Savarese
    IEEE International Conference on Robotics and Automation (ICRA), May 2020

    6-PACK: Category-Level 6D Pose Tracker with Anchor-Based Keypoints
    Chen Wang, Roberto Martín-Martín, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, Yuke Zhu
    IEEE International Conference on Robotics and Automation (ICRA), May 2020

    Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks
    Michelle A. Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, Jeannette Bohg
    IEEE Transactions on Robotics (T-RO), March 2020