Publications

Recent Works

(For a full list of publications, see below)

Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models

Learning and Language Datasets and Benchmarks

Generalized quantifiers (e.g., few, most) are used to indicate the proportions predicates are satisfied (for example, some apples are red). One way to interpret quantifier semantics is to explicitly bind these satisfactions with percentage scopes (e.g., 30%-40% of apples are red). This approach can be helpful for tasks like logic formalization and surface-form quantitative reasoning (Gordon and Schubert, 2010; Roy et al., 2015). However, it remains unclear if recent foundation models possess this ability, as they lack direct training signals. To explore this, we introduce QuRe, a crowd-sourced dataset of human-annotated generalized quantifiers in Wikipedia sentences featuring percentage-equipped predicates. We explore quantifier comprehension in language models using PRESQUE, a framework that combines natural language inference and the Rational Speech Acts framework. Experimental results on the HVD dataset and QuRe illustrate that PRESQUE, employing pragmatic reasoning, performs 20% better than a literal reasoning baseline when predicting quantifier percentage scopes, with no additional training required.

Yiyuan Li, Rakesh R Menon, Sayan Ghosh,  and Shashank Srivastava

Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2023.

[pdf]

MaNtLE: Model-agnostic Natural Language Explainer

Learning and Language

Understanding the internal reasoning behind the predictions of machine learning systems is increasingly vital, given their rising adoption and acceptance. While previous approaches, such as LIME, generate algorithmic explanations by attributing importance to input features for individual examples, recent research indicates that practitioners prefer examining language explanations that explain sub-groups of examples (Lakkaraju et al., 2022). In this paper, we introduce MaNtLE, a model-agnostic natural language explainer that analyzes multiple classifier predictions and generates faithful natural language explanations of classifier rationale for structured classification tasks. MaNtLE uses multi-task training on thousands of synthetic classification tasks to generate faithful explanations. Simulated user studies indicate that, on average, MaNtLE-generated explanations are at least 11% more faithful compared to LIME and Anchors explanations across three tasks. Human evaluations demonstrate that users can better predict model behavior using explanations from MaNtLE compared to other techniques.

Rakesh R Menon, Kerem Zaman,  and Shashank Srivastava

Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2023.

[pdf]

Identifying and Manipulating the Personality Traits of Language Models

Fairness and Social Applications Datasets and Benchmarks

Psychology research has long explored aspects of human personality such as extroversion, agreeableness and emotional stability. Categorizations like the ‘Big Five’ personality traits are commonly used to assess and diagnose personality types. In this work, we explore the question of whether the perceived personality in language models is exhibited consistently in their language generation. For example, is a language model such as GPT2 likely to respond in a consistent way if asked to go out to a party? We also investigate whether such personality traits can be controlled. We show that when provided different types of contexts (such as personality descriptions, or answers to diagnostic questions about personality traits), language models such as BERT and GPT2 can consistently identify and reflect personality markers in those contexts. This behavior illustrates an ability to be manipulated in a highly predictable way, and frames them as tools for identifying personality traits and controlling personas in applications such as dialog systems. We also contribute a crowd-sourced data-set of personality descriptions of human subjects paired with their “Big Five” personality assessment data, and a data-set of personality descriptions collated from Reddit.

Graham Caron and Shashank Srivastava

Findings of Empirical Methods in Natural Language Processing (EMNLP), 2023.

[pdf]

 

Full List of Publications

Tags: Learning and Language Datasets and Benchmarks Learning from Limited Labels Neuro-symbolic Learning Fairness and Social Applications Active Learning Language Understanding, Reasoning, and Generation Syntax and Semantics Miscellaneous

2023

  1. Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture
    Bingsheng Yao, Ishan Jindal, Lucian Popa, Yannis Katsis, Sayan Ghosh, Lihong He, Yuxuan Lu, Shashank Srivastava, Yunyao Li, James Hendler, and Dakuo Wang
    Findings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
    Active Learning
    [pdf], [arxiv]

  2. Leveraging Multiple Teachers for Test-Time Adaptation of Language-Guided Classifiers
    Kangda Wei, Sayan Ghosh, Rakesh R Menon, and Shashank Srivastava
    Findings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
    Learning and Language
    [pdf], [arxiv], [code]

  3. Identifying and Manipulating the Personality Traits of Language Models
    Graham Caron and Shashank Srivastava
    Findings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
    Fairness and Social Applications Datasets and Benchmarks
    [pdf], [arxiv]

  4. MaNtLE: Model-agnostic Natural Language Explainer
    Rakesh R Menon, Kerem Zaman, and Shashank Srivastava
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
    Learning and Language
    [pdf], [arxiv], [code]

  5. Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models
    Yiyuan Li, Rakesh R Menon, Sayan Ghosh, and Shashank Srivastava
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
    Learning and Language Datasets and Benchmarks
    [pdf], [arxiv], [code]

  6. LaSQuE: Improved Zero-Shot Classification from Explanations Through Quantifier Modeling and Curriculum Learning
    Sayan Ghosh*, Rakesh R Menon*, and Shashank Srivastava
    Findings of Association for Computational Linguistics (ACL), 2023.
    Learning and Language
    [pdf], [arxiv], [code]

  7. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
    Big-Bench Collaboration
    Transactions of Machine Learning Research (TMLR), 2023.
    Learning and Language Datasets and Benchmarks
    [openreview], [pdf], [arxiv], [code], [dataset]

2022

  1. What do Large Language Models Learn beyond Language?
    Avinash Madasu and Shashank Srivastava
    Findings of Empirical Methods in Natural Language Processing (EMNLP), 2022.
    Learning and Language
    [pdf], [arxiv], [code]

  2. Compositional Generalization for Kinship Prediction through Data Augmentation
    Kangda Wei, Sayan Ghosh, and Shashank Srivastava
    Proceedings of the 4th Workshop of Narrative Understanding (WNU), 2022.
    Learning from Limited Labels
    [pdf], [code]

  3. Predicting Difficulty and Discrimination of Natural Language Questions
    Matthew Byrd and Shashank Srivastava
    Proceedings of Association for Computational Linguistics (ACL), 2022.
    Language Understanding, Reasoning, and Generation Datasets and Benchmarks
    [pdf], [code], [dataset]

  4. ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding
    Sayan Ghosh and Shashank Srivastava
    Proceedings of Association for Computational Linguistics (ACL), 2022.
    Datasets and Benchmarks Language Understanding, Reasoning, and Generation Fairness and Social Applications
    [pdf], [arxiv], [code], [dataset]

  5. CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations
    Rakesh R Menon*, Sayan Ghosh*, and Shashank Srivastava
    Proceedings of Association for Computational Linguistics (ACL), 2022.
    Learning and Language Datasets and Benchmarks
    [pdf], [arxiv], [code], [dataset]

2021

  1. Does Social Pressure Drive Persuasion in Online Fora?
    Ayush Jain and Shashank Srivastava
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2021.
    Fairness and Social Applications
    [pdf]

  2. Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning
    Sayan Ghosh and Shashank Srivastava
    Findings of Empirical Methods in Natural Language Processing (EMNLP), 2021.
    Neuro-symbolic Learning
    [pdf], [arxiv], [code]

  3. Adversarial Scrubbing of Demographic Information for Text Classification
    Somnath Basu Roy Chowdhury, Sayan Ghosh, Yiyuan Li, Junier B Oliva, Shashank Srivastava, and Snigdha Chaturvedi
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2021.
    Fairness and Social Applications
    [pdf], [arxiv], [code]

  4. Improving and Simplifying Pattern Exploiting Training
    Derek Tam*, Rakesh R Menon*, Mohit Bansal, Shashank Srivastava, and Colin Raffel
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2021.
    Learning from Limited Labels
    [pdf], [arxiv], [code]

  5. How Helpful is Inverse Reinforcement Learning for Table-to-Text Generation?
    Sayan Ghosh*, Zheng Qi*, Snigdha Chaturvedi, and Shashank Srivastava
    Proceedings of Association for Computational Linguistics (ACL), 2021.
    Language Understanding, Reasoning, and Generation
    [pdf], [code]

2020

  1. PRover: Proof Generation for Interpretable Reasoning over Rules
    Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, and Mohit Bansal
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2020.
    Language Understanding, Reasoning, and Generation
    [pdf], [arxiv], [code]

  2. Learning Web-based procedures by Reasoning over Explanations and Demonstrations in Context
    Shashank Srivastava, Oleksandr Polozov, Nebojsa Jojic, and Christopher Meek
    Proceedings of the Association of Computational Linguistics (ACL), 2020.
    Learning and Language Learning from Limited Labels Neuro-symbolic Learning Datasets and Benchmarks
    [pdf], [dataset]

  3. An Agent for Learning New Natural Language Commands
    Amos Azaria, Shashank Srivastava, Jayant Krishnamurthy, Igor Labutov, and Tom Mitchell
    Journal of Autonomous Agents and Multi-Agent Systems (JAAMAS), 2020.
    Learning and Language Learning from Limited Labels
    [link]

2019

  1. Learning to Ask for Conversational Machine Learning
    Shashank Srivastava, Igor Labutov, and Tom Mitchell
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2019.
    Learning and Language Learning from Limited Labels
    [pdf]

2018

  1. LIA: A Natural Language Programmable Personal Assistant
    Igor Labutov, Shashank Srivastava, and Tom Mitchell
    Systems Demo, Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2018.
    Learning and Language
    [pdf]

  2. A Spatial Model for Extracting and Visualizing Latent Discourse Structure in Text
    Shashank Srivastava and Nebojsa Jojic
    Proceedings of the Association of Computational Linguistics (ACL), 2018.
    Syntax and Semantics
    [pdf]

  3. Zero-shot Learning of Classifiers from Natural Language Quantification
    Shashank Srivastava, Igor Labutov, and Tom Mitchell
    Proceedings of the Association of Computational Linguistics (ACL), 2018.
    Learning and Language Learning from Limited Labels
    [pdf]

  4. Where have I heard this story before? : Identifying Narrative Similarity in Movie Remakes
    Snigdha Chaturvedi, Shashank Srivastava, and Dan Roth
    Proceedings of the North Americal Chapter of Association of Computational Linguistics (NAACL), 2018.
    Fairness and Social Applications Datasets and Benchmarks
    [pdf]

2017

  1. Learning Classifiers from Declarative Language
    Shashank Srivastava, Igor Labutov, and Tom Mitchell
    NeurIPS Workshop on Learning from Limited Data, 2017.
    Learning and Language Learning from Limited Labels
    [pdf]

  2. Joint Concept Learning and Semantic Parsing from Natural Language Explanations
    Shashank Srivastava, Igor Labutov, and Tom Mitchell
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2017.
    Learning and Language Learning from Limited Labels Datasets and Benchmarks
    [pdf]

  3. Parsing Natural Language Conversations with Contextual Cues
    Shashank Srivastava, Amos Azaria, and Tom Mitchell
    Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2017.
    Neuro-symbolic Learning Datasets and Benchmarks
    [pdf]

2016

  1. CMUML Micro-Reader System for KBP 2016 Cold Start Slot Filling, Event Nugget Detection, and Event Argument Linking
    Bishan Yang, Ndapandula Nakashole, Bryan Kisiel, Emmanouil A. Platanios, Abulhair Saparov, Shashank Srivastava, Derry Wijaya, and Tom Mitchell
    Proceedings of the Text Analysis Conference (TAC), 2016.
    Syntax and Semantics
    [pdf]

  2. Inferring Interpersonal Relations in Narrative Summaries
    Shashank Srivastava, Snigdha Chaturvedi, and Tom Mitchell
    Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI), 2016.
    Fairness and Social Applications Datasets and Benchmarks
    [pdf]

  3. Modeling Evolving Relationships Between Characters in Literary Novel
    Snigdha Chaturvedi, Shashank Srivastava, Hal Daume III, and Chris Dyer
    Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI), 2016.
    Fairness and Social Applications
    [pdf]

2015

  1. CMU-ML System for KBP Cold Start Slot Filling
    Bryan Kisiel, Bill McDowell, Matt Gardner, Ndapandula Nakashole, Emmanouil A. Platanios, Abulhair Saparov, Shashank Srivastava, Derry Wijaya, and Tom Mitchell
    Proceedings of the Text Analysis Conference (TAC), 2015.
    Syntax and Semantics
    [pdf]

2014

  1. Vector space semantics with frequency-driven motifs
    Shashank Srivastava and Eduard Hovy
    Proceedings of the Association of Computational Linguistics (ACL), 2014.
    Syntax and Semantics
    [pdf]

  2. Spatial Compactness meets Topical Consistency: Jointly modeling link and content for community detection
    Mrinmaya Sachan, Avinava Dubey, Shashank Srivastava, Eric P Xing, and Eduard Hovy
    Proceedings of Web Search and Data Mining (WSDM), 2014.
    Miscellaneous
    [pdf]

2013

  1. A Walk-based Semantically Enriched Tree Kernel Over Distributed Word Representations
    Shashank Srivastava, Dirk Hovy, and Eduard Hovy
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2013.
    Syntax and Semantics
    [pdf]

  2. A Structured Distributional Semantic Model for Event Co-reference
    Kartik Goyal*, Sujay Kumar Jauhar*, Huiying Li*, Mrinmaya Sachan*, Shashank Srivastava*, and Eduard Hovy
    Proceedings of the Association of Computational Linguistics (ACL), 2013.
    Syntax and Semantics
    [pdf]

  3. A Structured Distributional Semantic Model : Integrating Structure with Semantics
    Kartik Goyal*, Sujay Kumar Jauhar*, Huiying Li*, Mrinmaya Sachan*, Shashank Srivastava*, and Eduard Hovy
    Workshop on Continuous Vector Space Models and their Compositionality, ACL, 2013.
    Syntax and Semantics
    [pdf]

  4. Identifying Metaphorical Word Use with Tree Kernels
    Dirk Hovy, Shashank Srivastava, Sujay Kumar Jauhar, Mrinmaya Sachan, Kartik Goyal, Huiying Li, Whitney Sanders, and Eduard Hovy
    NAACL-HLT Meta4NLP Workshop, 2013.
    Syntax and Semantics
    [pdf]

2012

  1. A Topical graph-kernel for Link Prediction in Labeled Graphs
    Snigdha Chaturvedi, Hal Daume III, Taesun Moon, and Shashank Srivastava
    ICML Workshop on Mining and Learning with Graphs (MLG), 2012.
    Miscellaneous
    [pdf]