Wanrong Zhu

Wanrong Zhu

CS Ph.D. Candidate

University of California, Santa Barbara


Hi, I am a final-year Ph.D. candidate in the Natural Language Processing Group at UCSB, advised by William Wang. I am honored and humbled to be named a 2023 Rising Stars in Machine Learning by University of Maryland. Before joining UCSB, I received my B.S. degree in Computer Science from Peking University.


  wanrongzhu [at] cs.ucsb.edu
   Henley Hall, UCSB


Education

  • University of California, Santa Barbara

    Ph.D. in Computer Science

    Sep. 2019 - Present

  • Peking University

    B.S. in Computer Science

    Sep. 2015 - July 2019

Interests

  • Vision-and-Language
  • Multimodal Study
  • Text Generation

Experience

Publications & Preprints

GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation

Preprint (arXiv 2311.07562)
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, Lijuan Wang
Paper Demo
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation

VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View

The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI 2024)
Raphael Schumann, Wanrong Zhu, Weixi Feng, Tsu-Jui Fu, Stefan Riezler, William Yang Wang
Paper Demo Code
VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View

VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use

The Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS D&B 2023)
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, Ludwig Schimdt
Paper Project Page Dataset Leaderboard Blog Post
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use

LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

The Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023)
Weixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Xuehai He, Sugato Basu, Xin Eric Wang, William Yang Wang
Paper Code Project Page
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

Multimodal Procedural Planning via Dual Text-Image Prompting

Preprint (arXiv 2305.01795)
Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, William Yang Wang
Paper Code
Multimodal Procedural Planning via Dual Text-Image Prompting

Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text

The Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS D&B 2023)
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, Yejin Choi
Paper Dataset GeekWire
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text

OpenFlamingo: An Open-Source Framework for Training Vision-Language Models with In-Context Learning

Preprint (arXiv 2308.01390)
Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, Ludwig Schmidt
Paper Blog Post (OF-v1) Blog Post (OF-v2) Code Demo
OpenFlamingo: An Open-Source Framework for Training Vision-Language Models with In-Context Learning

Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation

The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023, Short)
Wanrong Zhu, Xinyi Wang, Yujie Lu, Tsu-Jui Fu, Xin Eric Wang, Miguel Eckstein, William Yang Wang
Paper
Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation

Large Language Models Are Implicitly Topic Models: Explaining and Finding Good Demonstrations for In-Context Learning

The Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023)
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, William Yang Wang
Paper Code
Large Language Models Are Implicitly Topic Models: Explaining and Finding Good Demonstrations for In-Context Learning

Visualize Before You Write: Imagination-Guided Open-Ended Text Generation

The 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2023, Findings)
Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, William Yang Wang
Paper Code
Visualize Before You Write: Imagination-Guided Open-Ended Text Generation

CLIP also Understands Text: Prompting CLIP for Phrase Understanding

Preprint (arXiv 2210.05836)
An Yan, Jiacheng Li, Wanrong Zhu, Yujie Lu, William Yang Wang, Julian McAuley
Paper
CLIP also Understands Text: Prompting CLIP for Phrase Understanding

Neuro-Symbolic Causal Language Planning with Commonsense Prompting

The 11th International Conference on Learning Representations (ICLR 2023, Spotlight)
Yujie Lu, Weixi Feng, Wanrong Zhu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, William Yang Wang
Paper Project Code
Neuro-Symbolic Causal Language Planning with Commonsense Prompting

End-to-end Dense Video Captioning as Sequence Generation

The 29th International Conference on Computational Linguistics (COLING 2022)
Wanrong Zhu, Bo Pang, Ashish V. Thapliyal, William Yang Wang, Radu Soricut
Paper Video Poster
End-to-end Dense Video Captioning as Sequence Generation

Imagination-Augmented Natural Language Understanding

The 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2022, Oral)
Yujie Lu, Wanrong Zhu, Xin Eric Wang, Miguel Eckstein, William Yang Wang
Paper Code
Imagination-Augmented Natural Language Understanding

ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation

The 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2023, Findings)
Wanrong Zhu, Xin Eric Wang, An Yan, Miguel Eckstein, William Yang Wang
Paper
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation

Diagnosing Vision-and-Language Navigation: What Really Matters

The 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2022, Oral)
Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Eric Wang, Qi Wu, Miguel Eckstein, William Yang Wang
Paper Code Video Slides
Diagnosing Vision-and-Language Navigation: What Really Matters

Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation

The 16th conference of the European Chapter of the Association for Computational Linguistics (EACL 2021)
Wanrong Zhu, Xin Eric Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, William Yang Wang
Paper Code Video & Slides Poster
Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation

Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations

The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020, Short)
Wanrong Zhu, Xin Eric Wang, Pradyumna Narayana, Kazoo Sone, Sugato Basu, William Yang Wang
Paper Video & Slides
Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations

Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation

The 57th Annual Meeting of the Association for Computational Linguistics:System Demonstrations (ACL 2019 System Demonstration)
Zhiting Hu, Haoran Shi, Bowen Tan, Wentao Wang, Zichao Yang, Tiancheng Zhao, Junxian He, Lianhui Qin, Di Wang, Xuezhe Ma, Zhengzhong Liu, Xiaodan Liang, Wanrong Zhu, Devendra Singh Sachan, Eric P. Xing
Paper Code Docs
Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation

Text Infilling

Preprint (arXiv 1901.00158)
Wanrong Zhu, Zhiting Hu, Eric P. Xing
Paper Code
Text Infilling