Hello. I am a fifth-year computer science Ph.D. student at the Courant Institute of Mathematical Sciences at New York University. I am a member of the Machine Learning for Language (ML²) group (subset of the CILVR group). I am advised by Profs. He He and Kyunghyun Cho. I also frequently collaborate with Prof. Sam Bowman. Since October 2022, I have been (on-and-off) a Visiting Researcher at Fundamental AI Research (FAIR) at Meta, mentored by Dr. Jason Weston.
My research focuses on natural langauge processing and machine learning. Specifically, my interests include text generation (learning from reward, learning from feedback, machine translation, dialogue, decoding, uncertainty, etc.), long-text understanding, and reasoning. More recently, I have been trying to learn more about reasoning, scalable oversight, and ML in science.
Prior to my Ph.D., I graduated from the University of Chicago (B.S. in mathematics and B.S. in computer science). At Toyota Technological Institute at Chicago (TTIC) and the University of Chicago, I worked on text generation and structured prediction with my advisor Prof. Kevin Gimpel. In Summer 2020, I was a Research Intern at Google Research New York and worked on summarization; in Summer 2021, I was a Research Intern at Google Brain and investigated efficient pretraining.
Research
Primary research subfields: text generation (including machine translation) and structured prediction, language understanding, reasoning, and others.
[google scholar] [semantic scholar] [dblp] [abbreviations]
Overview of selected research directions
- Learning from rewards in text generation: GOLD (offline RL), amortized noisy channel NMT (off-policy RL & knowledge distillation), reward gaming (on-policy RL), AgreeSum (on-policy RL for multi-doc summarization), ENGINE for non-autoregressive NMT ("soft" knowledge distillation), implicit feedback in dialogue (extracting implicit reward in deployment data), ICE generation (extrapolating to unseen attribute values)
- Reasoning: PrOntoQA-OOD (deductive reasoning); this direction is my current main focus
- Scalable oversight benchmarks: QuALITY (long-document QA; related: SQuALITY which is the summarization form), GPQA (graduate-level Google-proof QA)
Publications and preprints (2023-)
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman
Preprint, November 2023
[paper] [abstract] [bibtex]
Leveraging Implicit Feedback from Deployment Data in Dialogue
Richard Yuanzhe Pang, Stephen Roller, Kyunghyun Cho, He He, Jason Weston
Preprint, July 2023
[paper] [abstract] [bibtex]
Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples
Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, Najoung Kim*, He He*
In Proceedings of NeurIPS 2023
[paper] [abstract] [poster at ICML 2023 Knowledge and Logical Reasoning Workshop] [bibtex]
Extrapolative Controlled Sequence Generation via Iterative Refinement
Vishakh Padmakumar, Richard Yuanzhe Pang, He He, Ankur P. Parikh
In Proceedings of ICML 2023
[paper] [abstract] [bibtex]
Reward Gaming in Conditional Text Generation
Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur P. Parikh, He He
In Proceedings of ACL 2023
[paper] [abstract] [15-min talk] [slides] [bibtex]
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey
Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex
Wang, Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang, Jason Phang, Samuel R. Bowman
In Proceedings of ACL 2023
[paper] [abstract] [website] [bibtex] | by others: [press]
Publications (2021-2022) — main focus: text generation (learning from rewards, RL), long-document understanding (question answering, summarization)
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way
Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, Samuel R. Bowman
In Proceedings of EMNLP 2022
[paper] [abstract] [data] [code] [bibtex] | by others: [zeroscrolls]
Amortized Noisy Channel Neural Machine Translation
Richard Yuanzhe Pang, He He, Kyunghyun Cho
In Proceedings of INLG 2022; best presentation award
tl;dr: amortizing the inference cost of "beam search and rerank" – learning to rerank without explicitly reranking
[paper] [abstract] [talk] [poster] [bibtex]
QuALITY: Question Answering with Long Input Texts, Yes!
Richard Yuanzhe Pang*, Alicia Parrish*, Nitish Joshi*, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, Samuel R. Bowman
In Proceedings of NAACL 2022
[paper] [abstract] [data] [code] [leaderboard] [15-min live talk] [slides] [bibtex] | by others: [tfds] [forecast] [press mention by Science] [scrolls] [zeroscrolls]
Token Dropping for Efficient BERT Pretraining
Le Hou*, Richard Yuanzhe Pang*, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, Denny Zhou
In Proceedings of ACL 2022
[paper] [abstract] [code] [talk] [bibtex] | by others: [press] [improvement]
AgreeSum: Agreement-Oriented Multi-Document Summarization
Richard Yuanzhe Pang*, Adam D. Lelkes*, Vinh Q. Tran*, Cong Yu
In Findings of ACL 2021
[paper] [abstract] [data] [short talk] [bibtex]
Comparing Test Sets with Item Response Theory
Clara Vania*, Phu Mon Htut*, William Huang*, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman
In Proceedings of ACL 2021
[paper] [abstract] [code] [bibtex]
Text Generation by Learning from Demonstrations
Richard Yuanzhe Pang, He He
In Proceedings of ICLR 2021
tl;dr: a high-precision-generation training objective to address the train/test objective mismatch and history mismatch
[paper] [abstract] [openreview] [poster] [slides] [code] [discussion] [bibtex] | by others: [ICLR blog by other authors] [GOLD in AlphaCode, Science] [GOLD as the main learning objective in AlphaCode 2, Dec 2023]
Publications (2019-2020) — main focus: text generation (textual style transfer, non-autoregressive translation, decoding), energy-based network in NLP
Improving Joint Training of Inference Networks and Structured Prediction Energy Networks
Lifu Tu, Richard Yuanzhe Pang, Kevin Gimpel
In Proceedings of EMNLP 2020 Workshop on Structured Prediction for NLP (SPNLP); spotlight paper
tl;dr: improving fast approximate+amortized inference for energy-based models in NLP structured prediction
[paper] [abstract] [my slides] [bibtex]
Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
Sean Welleck*, Ilia Kulikov*, Jaedeok Kim, Richard Yuanzhe Pang, Kyunghyun Cho
In Proceedings of EMNLP 2020
Also appearing in the non-archival DeepMath 2020
[paper] [abstract] [code] [bibtex]
ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation
Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, Kevin Gimpel
In Proceedings of ACL 2020
tl;dr: a "soft" form of knowledge distillation for non-autoregressive MT
[paper] [abstract] [code] [bibtex]
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?
Yada Pruksachatkun*, Jason Phang*, Haokun Liu*, Phu Mon Htut*, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman
In Proceedings of ACL 2020
[paper] [abstract] [bibtex]
Unsupervised Evaluation Metrics and Learning Criteria for Non-Parallel Textual Transfer
Richard Yuanzhe Pang, Kevin Gimpel
In Proceedings of EMNLP 2019 Workshop on Neural Generation and Translation (WNGT)
tl;dr: proposing more dimensions for textual transfer evaluation metrics, and losses that target them
[paper] [supplementals] [abstract] [poster] [bibtex]
The Daunting Task of Real-World Textual Style Transfer Auto-Evaluation
More info: [google scholar] [semantic scholar] [dblp] [abbreviations]
Discussion of GOLD [pdf]
As an area chair
As a reviewer / program committee member
External
At New York University
At the University of Chicago
Selected presentations
Other conference presentations with associated proceeding papers
Last updated: December 6, 2023. Contact: My NYU office is at 60 5th Ave. Get in touch at yzpang at _ dot edu (where _ is nyu)!
Richard Yuanzhe Pang
Extended abstract in EMNLP 2019 Workshop on Neural Generation and Translation (WNGT); abstract in Proceedings of the Workshop on Noisy User-generated Text (W-NUT)
tl;dr: an opinion piece arguing that the research on textual style transfer and its evaluation are going astray
[paper] [abstract] [poster] [bibtex]
Discussion
June 2022
tl;dr: GOLD does not maximize the expected reward. It maximizes the expected reward of training examples only.
More research activities
Teaching
Presentations
Please email for full CV.