Shiyu Huang   黄世宇

Chief Research Officer, Roban

Haidian District, Beijing, China, 100084.
Email: huangsy1314@163.com

[OpenRL]     [知乎]     [Google Scholar]     [GitHub]     [小红书]     [Linkedin]     [CV]

Visitors: 180575

Biography

I am the Chief Research Officer at Roban. Before that, I was a technical manager at XPeng Inc, worked as a principal researcher in Zhipu AI and 4Paradigm Inc., and I am the leader of the OpenRL Lab. I received my B.E. and Ph.D. degrees (co-advised by Prof. Jun Zhu and Prof. Ting Chen) from the Department of Computer Science and Technology, Tsinghua University in July, 2017 and June, 2022. My researches focus on large language models, multimodal large language models, agents, and deep reinforcement learning. I have also spent time working at RealAI Inc. , Huawei Noah's Ark Lab, Tencent AI Lab, Carnegie Mellon University and Sensetime Inc. . And I am also the founder of the OpenRL Lab(GitHub stars) and TARTRL group.

We are looking for self-motivated interns and full-timers who have a strong background in mathematics/computer science and are eager to get involved in cutting-edge, fundamental AI research. Please feel free to drop me an email if you are interested in collaborating with me.

Publications && Preprints

(* equal contribution)

GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives
Qize Yu*, Lianrui Fan*, Boyu Chen*, Jiaqi Liang*, Xini Ding*, Yue Chen, Zetian Song, Yuran Wang, Yi Zou, Kaixuan Wang, Tianxing Chen, Wenxuan Song, Bohan Zhou, Mingleyang Li, Siqiao Huang, Yuqi Ye, Caigao Jiang, Wei Wei, Ruihai Wu, Hang Zhang, Yixiao Ge, Shuchang Zhou, Shilong Liu, Xianming Liu, Ping Luo, Shiyu Huang
arXiv:2609.39601, 2026
GitHub stars
[PDF] [Project] [Code] [Model] [Demo] [BibTeX]
@misc{yu2026groundingpigroundingfoundationmodel, title={GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives}, author={Qize Yu and Lianrui Fan and Boyu Chen and Jiaqi Liang and Xini Ding and Yue Chen and Zetian Song and Yuran Wang and Yi Zou and Kaixuan Wang and Tianxing Chen and Wenxuan Song and Bohan Zhou and Mingleyang Li and Siqiao Huang and Yuqi Ye and Caigao Jiang and Wei Wei and Ruihai Wu and Hang Zhang and Yixiao Ge and Shuchang Zhou and Shilong Liu and Xianming Liu and Ping Luo and Shiyu Huang}, year={2026}, eprint={2609.39601}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2609.39601}, }
GroundAnything: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed
Qize Yu*, Lianrui Fan*, Bowen Ping*, Xini Ding*, Zetian Song*, Junbo Niu, Kaixuan Wang, Tianxing Chen, Yue Chen, Minghua He, Yuran Wang, Jie Huang, Haojun Zhang, Min Chen, Hao Li, Wenxuan Song, Ruihai Wu, Xianming Liu, Shilong Liu, Shuchang Zhou, Ping Luo, Shiyu Huang
arXiv:2609.39600, 2026
GitHub stars
[PDF] [Project] [Code] [Model] [Demo] [BibTeX]
@misc{yu2026groundanythingreconcilingparalleldecoding, title={GroundAnything: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed}, author={Qize Yu and Lianrui Fan and Bowen Ping and Xini Ding and Zetian Song and Junbo Niu and Kaixuan Wang and Tianxing Chen and Yue Chen and Minghua He and Yuran Wang and Jie Huang and Haojun Zhang and Min Chen and Hao Li and Wenxuan Song and Ruihai Wu and Xianming Liu and Shilong Liu and Shuchang Zhou and Ping Luo and Shiyu Huang}, year={2026}, eprint={2609.39600}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2609.39600}, }
Highlight
AUGUstus: Autonomous Driving VLA Model with Unified Generation and Understanding
Yuqi Ye, Caigao Jiang, Zhuonan Wang, Congyuan Pei, Chao Qu, Changhao Peng, Junhong Lin, Qize Yu, Zetian Song, Guangzhao Fu, Chongruo Wu, Shiyu Huang, Shuchang Zhou, Honggou Yang, Wei Gao
arXiv:2610.16985, 2026
GitHub stars
[PDF] [Project] [Code] [BibTeX]
@misc{ye2026augustusautonomousdrivingvlamodel, title={AUGUstus: Autonomous Driving VLA Model with Unified Generation and Understanding}, author={Yuqi Ye and Caigao Jiang and Zhuonan Wang and Congyuan Pei and Chao Qu and Changhao Peng and Junhong Lin and Qize Yu and Zetian Song and Guangzhao Fu and Chongruo Wu and Shiyu Huang and Shuchang Zhou and Honggou Yang and Wei Gao}, year={2026}, eprint={2610.16985}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2610.16985}, }
Highlight
X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
Haojun Zhang, Yi Zou, Min Chen, Qize Yu, Lianrui Fan, Xini Ding, Hao Li, Shuchang Zhou, Xianming Liu, Shiyu Huang
arXiv:2609.11412, 2026
GitHub stars
[PDF] [Code] [Hugging Face] [BibTeX]
@misc{zhang2026xautprogressiveaudioencodercompression, title={X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation}, author={Haojun Zhang and Yi Zou and Min Chen and Qize Yu and Lianrui Fan and Xini Ding and Hao Li and Shuchang Zhou and Xianming Liu and Shiyu Huang}, year={2026}, eprint={2609.11412}, archivePrefix={arXiv}, primaryClass={cs.SD}, url={https://arxiv.org/abs/2609.11412}, }
Highlight
OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
Felix Henry, Xiaochen Lin, Jiangyou Zhu, Yangfan, Bingqian Zhang, Min Chen, and Shiyu Huang
arXiv:2605.18758, 2026
GitHub stars
[PDF] [Code] [Hugging Face] [BibTeX]
@article{henry2026omnigui, title={OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments}, author={Henry, Felix and Lin, Xiaochen and Zhu, Jiangyou and Zhang, Bingqian and Chen, Min and Huang, Shiyu and others}, journal={arXiv preprint arXiv:2605.18758}, year={2026} }
Highlight
OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination
Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Chao Sun, Rongzhou Zhang, Guanyu Zhou, Lijie Wen, Xuming Hu
The Fortieth AAAI Conference on Artificial Intelligence(AAAI), 2026
[PDF] [BibTeX]
@article{chen2025omnidpo, title={OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination}, author={Chen, Junzhe and Zhang, Tianshu and Huang, Shiyu and Niu, Yuwei and Sun, Chao and Zhang, Rongzhou and Zhou, Guanyu and Wen, Lijie and Hu, Xuming}, journal={arXiv preprint arXiv:2509.00723}, year={2025} }
Highlight
Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models
Leyi Pan, Zheyu Fu, Yunpeng Zhai, Shuchang Tao, Sheng Guan, Shiyu Huang, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Felix Henry, Lijie Wen, Aiwei Liu
The ACM Multimedia(ACMMM), 2026
GitHub stars GitHub forks
[PDF] [Code] [Dataset] [BibTeX]
@misc{pan2025omnisafetybenchbenchmarksafetyevaluation, title={Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models}, author={Leyi Pan and Zheyu Fu and Yunpeng Zhai and Shuchang Tao and Sheng Guan and Shiyu Huang and Lingzhe Zhang and Zhaoyang Liu and Bolin Ding and Felix Henry and Lijie Wen and Aiwei Liu}, year={2025}, eprint={2508.07173}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2508.07173}, }
Highlight
OmniPlay: Benchmarking Omni-Modal Models on Omni-Modal Game Playing
Fuqing Bie, Shiyu Huang, Xijia Tao, Zhiqin Fang, Leyi Pan, Junzhe Chen, Min Ren, Liuyu Xiang, Zhaofeng He
arXiv:2508.04361, 2025
GitHub stars GitHub forks
[PDF] [Code] [BibTeX]
@article{bie2025omniplay, title={OmniPlay: Benchmarking Omni-Modal Models on Omni-Modal Game Playing}, author={Bie, Fuqing and Huang, Shiyu and Tao, Xijia and Fang, Zhiqin and Pan, Leyi and Chen, Junzhe and Ren, Min and Xiang, Liuyu and He, Zhaofeng}, journal={arXiv preprint arXiv:2508.04361}, year={2025} }
Highlight
SA-GCS: Semantic-Aware Gaussian Curriculum Scheduling for UAV Vision-Language Navigation
Hengxing Cai*, Jinhan Dong*, Yijie Rao*, Jingcheng Deng, Jingjun Tan, Qien Chen, Haidong Wang, Zhen Wang, Shiyu Huang, Agachai Sumalee, Renxin Zhong
arXiv:2508.00390, 2025
[PDF] [BibTeX]
@article{cai2025sa, title={SA-GCS: Semantic-Aware Gaussian Curriculum Scheduling for UAV Vision-Language Navigation}, author={Cai, Hengxing and Dong, Jinhan and Rao, Yijie and Deng, Jingcheng and Tan, Jingjun and Chen, Qien and Wang, Haidong and Wang, Zhen and Huang, Shiyu and Sumalee, Agachai and others}, journal={arXiv preprint arXiv:2508.00390}, year={2025} }
Highlight
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Wenkai Li, Wei Jia, Xin Lyu, Xuancheng Huang, Yanling Wang, Yadong Xue, Yanfeng Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuxuan Zhang, Zhanxiao Du, Zhenyu Hou, Zhao Xue, Zhengxiao Du, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, Jie Tang
arXiv:2507.01006, 2025
GitHub stars GitHub forks
[PDF] [Code] [BibTeX]
@article{hong2025glm, title={GLM-4.1 V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning}, author={Hong, Wenyi and Yu, Wenmeng and Gu, Xiaotao and Wang, Guo and Gan, Guobing and Tang, Haomiao and Cheng, Jiale and Qi, Ji and Ji, Junhui and Pan, Lihang and others}, journal={arXiv preprint arXiv:2507.01006}, year={2025} }
Highlight
Generalizable Agent Modeling for Agent Collaboration-Competition Adaptation with Multi-Retrieval and Dynamic Generation
Chenxu Wang, Yonggang Jin, Cheng Hu, Youpeng Zhao, Zipeng Dai, Jian Zhao, Shiyu Huang, Liuyu Xiang, Junge Zhang, Zhaofeng He
Neurocomputing (2025): 130912
GitHub stars GitHub forks
[PDF] [Code] [BibTeX]
@article{wang2025generalizable, title={Generalizable Agent Modeling for Agent Collaboration-Competition Adaptation with Multi-Retrieval and Dynamic Generation}, author={Wang, Chenxu and Jin, Yonggang and Hu, Cheng and Zhao, Youpeng and Dai, Zipeng and Zhao, Jian and Xiang, Liuyu and Zhang, Junge and He, Zhaofeng}, journal={Neurocomputing}, pages={130912}, year={2025}, publisher={Elsevier} }
Highlight
MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models
Wenyi Hong*, Yean Cheng*, Zhuoyi Yang, Weihan Wang, Lefan Wang, Xiaotao Gu, Shiyu Huang, Yuxiao Dong, Jie Tang
The IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 2025
GitHub stars GitHub forks
[Blog] [PDF] [Code] [Dataset] [Leaderboard] [BibTeX]
@article{hong2025motionbench, title={MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models}, author={Hong, Wenyi and Cheng, Yean and Yang, Zhuoyi and Wang, Weihan and Wang, Lefan and Gu, Xiaotao and Huang, Shiyu and Dong, Yuxiao and Tang, Jie}, journal={arXiv preprint arXiv:2501.02955}, year={2025} }
Highlight
VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation
Jiazheng Xu*, Yu Huang*, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang, Wendi Zheng, Xiao Liu, Ming Ding, Xiaohan Zhang, Xiaotao Gu, Shiyu Huang, Minlie Huang, Jie Tang, Yuxiao Dong
The Fortieth AAAI Conference on Artificial Intelligence(AAAI), 2026
GitHub stars GitHub forks
[PDF] [Code] [Huggingface] [BibTeX]
@article{xu2024visionreward, title={VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation}, author={Xu, Jiazheng and Huang, Yu and Cheng, Jiale and Yang, Yuanming and Xu, Jiajun and Wang, Yuan and Duan, Wenbo and Yang, Shen and Jin, Qunlin and Li, Shurun and others}, journal={arXiv preprint arXiv:2412.21059}, year={2024} }
Highlight
ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models
Junzhe Chen*, Tianshu Zhang*, Shiyu Huang, Yuwei Niu, Linfeng Zhang, Lijie Wen, Xuming Hu
The IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 2025
[PDF] [BibTeX]
@article{chen2024ict, title={ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models}, author={Chen, Junzhe and Zhang, Tianshu and Huang, Shiyu and Niu, Yuwei and Zhang, Linfeng and Wen, Lijie and Hu, Xuming}, journal={arXiv preprint arXiv:2411.15268}, year={2024} }
Highlight
DreamPolish: Domain Score Distillation With Progressive Geometry Generation
Yean Cheng*, Ziqi Cai*, Ming Ding, Wendi Zheng, Shiyu Huang, Yuxiao Dong, Jie Tang, Boxin Shi
arXiv:2411.01602, 2024
[PDF] [BibTeX]
@article{cheng2024dreampolish, title={DreamPolish: Domain Score Distillation With Progressive Geometry Generation}, author={Cheng, Yean and Cai, Ziqi and Ding, Ming and Zheng, Wendi and Huang, Shiyu and Dong, Yuxiao and Tang, Jie and Shi, Boxin}, journal={arXiv preprint arXiv:2411.01602}, year={2024} }
Highlight
CogVLM2: Visual Language Models for Image and Video Understanding
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Yuxiao Dong, Jie Tang
arXiv:2408.16500, 2024
GitHub stars GitHub forks
[PDF] [Code] [Huggingface] [BibTeX]
@article{hong2024cogvlm2, title={CogVLM2: Visual Language Models for Image and Video Understanding}, author={Hong, Wenyi and Wang, Weihan and Ding, Ming and Yu, Wenmeng and Lv, Qingsong and Wang, Yan and Cheng, Yean and Huang, Shiyu and Ji, Junhui and Xue, Zhao and others}, journal={arXiv preprint arXiv:2408.16500}, year={2024} }
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
Zhuoyi Yang*, Jiayan Teng*, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Xiaohan Zhang, Xiaotao Gu, Guanyu Feng, Da Yin, Wenyi Hong, Weihan Wang, Yean Cheng, Yuxuan Zhang, Ting Liu, Bin Xu, Yuxiao Dong, Jie Tang
The Thirteenth International Conference on Learning Representations (ICLR), 2025
GitHub stars GitHub forks
[PDF] [Code] [Huggingface] [BibTeX]
@article{yang2024cogvideox, title={CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer}, author={Yang, Zhuoyi and Teng, Jiayan and Zheng, Wendi and Ding, Ming and Huang, Shiyu and Xu, Jiazheng and Yang, Yuanming and Hong, Wenyi and Zhang, Xiaohan and Feng, Guanyu and others}, journal={arXiv preprint arXiv:2408.06072}, year={2024} }
Highlight
LVBench: An Extreme Long Video Understanding Benchmark
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, Jie Tang
International Conference on Computer Vision(ICCV)(Highlight), 2025
GitHub stars
[PDF] [Project] [Code] [Huggingface] [BibTeX]
@misc{wang2024lvbench, title={LVBench: An Extreme Long Video Understanding Benchmark}, author={Weihan Wang and Zehai He and Wenyi Hong and Yean Cheng and Xiaohan Zhang and Ji Qi and Shiyu Huang and Bin Xu and Yuxiao Dong and Ming Ding and Jie Tang}, year={2024}, eprint={2406.08035}, archivePrefix={arXiv}, primaryClass={cs.CV} }

Talks

Projects

Patents

Honors & Awards

Competitions

Services

Organizer for:
NeurIPS 2023 Workshop on New in ML

Reviewer for:
AAAI 2026, NeurIPS 2025, ICCV 2025, ICML 2025, CVPR 2025, ICLR 2025, AAAI 2025, NeurIPS 2024, ICML 2024, ICLR 2024, AAAI 2024, NeurIPS 2023, AISTATS 2023, AAAI 2023, ICLR 2023, NeurIPS 2022, ICML 2022, AISTATS 2022, AAAI 2022, ICLR 2022, NeurIPS 2021, ICML 2021, AAAI 2021, NeurIPS 2020

Teaching

2020 Spring, TA in Big Data and Machine Intelligence, instructed by Zhen Chen
2019 Fall, TA in Big Data and Machine Intelligence, instructed by Zhen Chen
2019 Spring, TA in Machine Learning, instructed by Prof. Jun Zhu

Experiences