Recent Works

(For a full list of publications, see below)

DISCERN: Decoding Systematic Errors in Natural Language for Text Classifiers

Learning and Language Fairness and Social Applications Learning from Limited Labels

Machine learning classifiers often exhibit systematic biases due to dataset artifacts or class imbalances. DISCERN is a framework that generates natural language explanations for these biases, using an interactive loop between two large language models—one that identifies error patterns and another that refines them. The explanations improve classifier debugging and can be used to augment training data through synthetic instances or active learning. DISCERN consistently improves classification accuracy and helps humans interpret systematic biases 25% more effectively than cluster-based methods.

Rakesh R Menon and Shashank Srivastava

Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2024.

[pdf]

Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion

Learning and Language Fairness and Social Applications

Model fusion aims to aggregate the knowledge of multiple models to enhance performance by combining their weights. In this work, we study the inverse problem – investigating whether model fusion can reduce unwanted knowledge. We investigate the effects of model fusion in three scenarios – the learning of shortcuts, social biases, and memorization of training data in fine-tuned language models. Through experiments covering classification and generation, we show that shared knowledge among models is enhanced during model fusion, while unshared knowledge is forgotten. Based on this, we show that model fusion can be a debiasing tool and showcase its efficacy in addressing privacy concerns with language models.

Kerem Zaman, Leshem Chosen,  and Shashank Srivastava

Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2024.

[pdf]

SOCIALGAZE: Improving the Integration of Human Social Norms in Large Language Models

Fairness and Social Applications

Large language models struggle to align with human social norms, particularly when evaluating social acceptability in everyday situations. SOCIALGAZE is a multi-step prompting framework that improves LLMs’ reasoning by encouraging them to verbalize social scenarios from multiple perspectives before making judgments. Experiments on social conflict narratives from the r/AITA subreddit show that SOCIALGAZE improves model alignment with human judgments by up to 11 F1 points. The study also uncovers several interesting biases in LLM social acceptability judgments – for example, women are much less likely than men to be called out on r/AITA.

Anvesh Rao Vijjini, Rakesh R Menon, Jiayi Fu, Shashank Srivastava,  and Snigdha Chaturvedi

Findings of Empirical Methods in Natural Language Processing (EMNLP Findings), 2024.

[pdf]

MaNtLE: Model-agnostic Natural Language Explainer

Learning and Language

Understanding the internal reasoning behind the predictions of machine learning systems is increasingly vital, given their rising adoption and acceptance. While previous approaches, such as LIME, generate algorithmic explanations by attributing importance to input features for individual examples, recent research indicates that practitioners prefer examining language explanations that explain sub-groups of examples (Lakkaraju et al., 2022). In this paper, we introduce MaNtLE, a model-agnostic natural language explainer that analyzes multiple classifier predictions and generates faithful natural language explanations of classifier rationale for structured classification tasks. MaNtLE uses multi-task training on thousands of synthetic classification tasks to generate faithful explanations. Simulated user studies indicate that, on average, MaNtLE-generated explanations are at least 11% more faithful compared to LIME and Anchors explanations across three tasks. Human evaluations demonstrate that users can better predict model behavior using explanations from MaNtLE compared to other techniques.

Rakesh R Menon, Kerem Zaman,  and Shashank Srivastava

Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2023.

[pdf]

Identifying and Manipulating the Perceived Personality Traits of Language Models

Fairness and Social Applications Datasets and Benchmarks

In this work, we explore the question of whether the perceived personality in language models is exhibited consistently in their language generation. For example, is a language model such as GPT2 likely to respond in a consistent way if asked to go out to a party? We also investigate whether such personality traits can be controlled. We show that when provided different types of contexts (such as personality descriptions, or answers to diagnostic questions about personality traits), language models such as BERT and GPT2 can consistently identify and reflect personality markers in those contexts. This behavior illustrates an ability to be manipulated in a highly predictable way, and frames them as tools for identifying personality traits and controlling personas in applications such as dialog systems. We also contribute a crowd-sourced data-set of personality descriptions of human subjects paired with their Big Five personality assessment data, and a data-set of personality descriptions collated from Reddit.

Graham Caron and Shashank Srivastava

Findings of Empirical Methods in Natural Language Processing (EMNLP), 2023.

[pdf]

 

Full List of Publications

Tags: Learning and Language Datasets and Benchmarks Learning from Limited Labels Neuro-symbolic Learning Fairness and Social Applications Active Learning Language Understanding, Reasoning, and Generation Syntax and Semantics Miscellaneous

2024

  1. SOCIALGAZE: Improving the Integration of Human Social Norms in Large Language Models
    Anvesh Rao Vijjini, Rakesh R Menon, Jiayi Fu, Shashank Srivastava, and Snigdha Chaturvedi
    Findings of Empirical Methods in Natural Language Processing (EMNLP Findings), 2024.
    Fairness and Social Applications
    [pdf], [code]

  2. Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion
    Kerem Zaman, Leshem Chosen, and Shashank Srivastava
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2024.
    Learning and Language Fairness and Social Applications
    [pdf], [code]

  3. DISCERN: Decoding Systematic Errors in Natural Language for Text Classifiers
    Rakesh R Menon and Shashank Srivastava
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2024.
    Learning and Language Fairness and Social Applications Learning from Limited Labels
    [pdf], [code]

2023

  1. Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture
    Bingsheng Yao, Ishan Jindal, Lucian Popa, Yannis Katsis, Sayan Ghosh, Lihong He, Yuxuan Lu, Shashank Srivastava, Yunyao Li, James Hendler, and Dakuo Wang
    Findings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
    Active Learning
    [pdf], [arxiv]

  2. Leveraging Multiple Teachers for Test-Time Adaptation of Language-Guided Classifiers
    Kangda Wei, Sayan Ghosh, Rakesh R Menon, and Shashank Srivastava
    Findings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
    Learning and Language
    [pdf], [arxiv], [code]

  3. Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models
    Yiyuan Li, Rakesh R Menon, Sayan Ghosh, and Shashank Srivastava
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
    Learning and Language Datasets and Benchmarks
    [pdf], [arxiv], [code]

  4. Identifying and Manipulating the Perceived Personality Traits of Language Models
    Graham Caron and Shashank Srivastava
    Findings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
    Fairness and Social Applications Datasets and Benchmarks
    [pdf], [arxiv]

  5. MaNtLE: Model-agnostic Natural Language Explainer
    Rakesh R Menon, Kerem Zaman, and Shashank Srivastava
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
    Learning and Language
    [pdf], [arxiv], [code]

  6. LaSQuE: Improved Zero-Shot Classification from Explanations Through Quantifier Modeling and Curriculum Learning
    Sayan Ghosh*, Rakesh R Menon*, and Shashank Srivastava
    Findings of Association for Computational Linguistics (ACL), 2023.
    Learning and Language
    [pdf], [arxiv], [code]

  7. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
    Big-Bench Collaboration
    Transactions of Machine Learning Research (TMLR), 2023.
    Learning and Language Datasets and Benchmarks
    [openreview], [pdf], [arxiv], [code], [dataset]

2022

  1. What do Large Language Models Learn beyond Language?
    Avinash Madasu and Shashank Srivastava
    Findings of Empirical Methods in Natural Language Processing (EMNLP), 2022.
    Learning and Language
    [pdf], [arxiv], [code]

  2. Compositional Generalization for Kinship Prediction through Data Augmentation
    Kangda Wei, Sayan Ghosh, and Shashank Srivastava
    Proceedings of the 4th Workshop of Narrative Understanding (WNU), 2022.
    Learning from Limited Labels
    [pdf], [code]

  3. Predicting Difficulty and Discrimination of Natural Language Questions
    Matthew Byrd and Shashank Srivastava
    Proceedings of Association for Computational Linguistics (ACL), 2022.
    Language Understanding, Reasoning, and Generation Datasets and Benchmarks
    [pdf], [code], [dataset]

  4. ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding
    Sayan Ghosh and Shashank Srivastava
    Proceedings of Association for Computational Linguistics (ACL), 2022.
    Datasets and Benchmarks Language Understanding, Reasoning, and Generation Fairness and Social Applications
    [pdf], [arxiv], [code], [dataset]

  5. CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations
    Rakesh R Menon*, Sayan Ghosh*, and Shashank Srivastava
    Proceedings of Association for Computational Linguistics (ACL), 2022.
    Learning and Language Datasets and Benchmarks
    [pdf], [arxiv], [code], [dataset]

2021

  1. Does Social Pressure Drive Persuasion in Online Fora?
    Ayush Jain and Shashank Srivastava
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2021.
    Fairness and Social Applications
    [pdf]

  2. Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning
    Sayan Ghosh and Shashank Srivastava
    Findings of Empirical Methods in Natural Language Processing (EMNLP), 2021.
    Neuro-symbolic Learning
    [pdf], [arxiv], [code]

  3. Adversarial Scrubbing of Demographic Information for Text Classification
    Somnath Basu Roy Chowdhury, Sayan Ghosh, Yiyuan Li, Junier B Oliva, Shashank Srivastava, and Snigdha Chaturvedi
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2021.
    Fairness and Social Applications
    [pdf], [arxiv], [code]

  4. Improving and Simplifying Pattern Exploiting Training
    Derek Tam*, Rakesh R Menon*, Mohit Bansal, Shashank Srivastava, and Colin Raffel
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2021.
    Learning from Limited Labels
    [pdf], [arxiv], [code]

  5. How Helpful is Inverse Reinforcement Learning for Table-to-Text Generation?
    Sayan Ghosh*, Zheng Qi*, Snigdha Chaturvedi, and Shashank Srivastava
    Proceedings of Association for Computational Linguistics (ACL), 2021.
    Language Understanding, Reasoning, and Generation
    [pdf], [code]

2020

  1. PRover: Proof Generation for Interpretable Reasoning over Rules
    Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, and Mohit Bansal
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2020.
    Language Understanding, Reasoning, and Generation
    [pdf], [arxiv], [code]

  2. Learning Web-based procedures by Reasoning over Explanations and Demonstrations in Context
    Shashank Srivastava, Oleksandr Polozov, Nebojsa Jojic, and Christopher Meek
    Proceedings of the Association of Computational Linguistics (ACL), 2020.
    Learning and Language Learning from Limited Labels Neuro-symbolic Learning Datasets and Benchmarks
    [pdf], [dataset]

  3. An Agent for Learning New Natural Language Commands
    Amos Azaria, Shashank Srivastava, Jayant Krishnamurthy, Igor Labutov, and Tom Mitchell
    Journal of Autonomous Agents and Multi-Agent Systems (JAAMAS), 2020.
    Learning and Language Learning from Limited Labels
    [link]

2019

  1. Learning to Ask for Conversational Machine Learning
    Shashank Srivastava, Igor Labutov, and Tom Mitchell
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2019.
    Learning and Language Learning from Limited Labels
    [pdf]

2018

  1. LIA: A Natural Language Programmable Personal Assistant
    Igor Labutov, Shashank Srivastava, and Tom Mitchell
    Systems Demo, Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2018.
    Learning and Language
    [pdf]

  2. A Spatial Model for Extracting and Visualizing Latent Discourse Structure in Text
    Shashank Srivastava and Nebojsa Jojic
    Proceedings of the Association of Computational Linguistics (ACL), 2018.
    Syntax and Semantics
    [pdf]

  3. Zero-shot Learning of Classifiers from Natural Language Quantification
    Shashank Srivastava, Igor Labutov, and Tom Mitchell
    Proceedings of the Association of Computational Linguistics (ACL), 2018.
    Learning and Language Learning from Limited Labels
    [pdf]

  4. Where have I heard this story before? : Identifying Narrative Similarity in Movie Remakes
    Snigdha Chaturvedi, Shashank Srivastava, and Dan Roth
    Proceedings of the North Americal Chapter of Association of Computational Linguistics (NAACL), 2018.
    Fairness and Social Applications Datasets and Benchmarks
    [pdf]

2017

  1. Learning Classifiers from Declarative Language
    Shashank Srivastava, Igor Labutov, and Tom Mitchell
    NeurIPS Workshop on Learning from Limited Data, 2017.
    Learning and Language Learning from Limited Labels
    [pdf]

  2. Joint Concept Learning and Semantic Parsing from Natural Language Explanations
    Shashank Srivastava, Igor Labutov, and Tom Mitchell
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2017.
    Learning and Language Learning from Limited Labels Datasets and Benchmarks
    [pdf]

  3. Parsing Natural Language Conversations with Contextual Cues
    Shashank Srivastava, Amos Azaria, and Tom Mitchell
    Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2017.
    Neuro-symbolic Learning Datasets and Benchmarks
    [pdf]

2016

  1. CMUML Micro-Reader System for KBP 2016 Cold Start Slot Filling, Event Nugget Detection, and Event Argument Linking
    Bishan Yang, Ndapandula Nakashole, Bryan Kisiel, Emmanouil A. Platanios, Abulhair Saparov, Shashank Srivastava, Derry Wijaya, and Tom Mitchell
    Proceedings of the Text Analysis Conference (TAC), 2016.
    Syntax and Semantics
    [pdf]

  2. Inferring Interpersonal Relations in Narrative Summaries
    Shashank Srivastava, Snigdha Chaturvedi, and Tom Mitchell
    Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI), 2016.
    Fairness and Social Applications Datasets and Benchmarks
    [pdf]

  3. Modeling Evolving Relationships Between Characters in Literary Novel
    Snigdha Chaturvedi, Shashank Srivastava, Hal Daume III, and Chris Dyer
    Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI), 2016.
    Fairness and Social Applications
    [pdf]

2015

  1. CMU-ML System for KBP Cold Start Slot Filling
    Bryan Kisiel, Bill McDowell, Matt Gardner, Ndapandula Nakashole, Emmanouil A. Platanios, Abulhair Saparov, Shashank Srivastava, Derry Wijaya, and Tom Mitchell
    Proceedings of the Text Analysis Conference (TAC), 2015.
    Syntax and Semantics
    [pdf]

2014

  1. Vector space semantics with frequency-driven motifs
    Shashank Srivastava and Eduard Hovy
    Proceedings of the Association of Computational Linguistics (ACL), 2014.
    Syntax and Semantics
    [pdf]

  2. Spatial Compactness meets Topical Consistency: Jointly modeling link and content for community detection
    Mrinmaya Sachan, Avinava Dubey, Shashank Srivastava, Eric P Xing, and Eduard Hovy
    Proceedings of Web Search and Data Mining (WSDM), 2014.
    Miscellaneous
    [pdf]

2013

  1. A Walk-based Semantically Enriched Tree Kernel Over Distributed Word Representations
    Shashank Srivastava, Dirk Hovy, and Eduard Hovy
    Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2013.
    Syntax and Semantics
    [pdf]

  2. A Structured Distributional Semantic Model for Event Co-reference
    Kartik Goyal*, Sujay Kumar Jauhar*, Huiying Li*, Mrinmaya Sachan*, Shashank Srivastava*, and Eduard Hovy
    Proceedings of the Association of Computational Linguistics (ACL), 2013.
    Syntax and Semantics
    [pdf]

  3. A Structured Distributional Semantic Model : Integrating Structure with Semantics
    Kartik Goyal*, Sujay Kumar Jauhar*, Huiying Li*, Mrinmaya Sachan*, Shashank Srivastava*, and Eduard Hovy
    Workshop on Continuous Vector Space Models and their Compositionality, ACL, 2013.
    Syntax and Semantics
    [pdf]

  4. Identifying Metaphorical Word Use with Tree Kernels
    Dirk Hovy, Shashank Srivastava, Sujay Kumar Jauhar, Mrinmaya Sachan, Kartik Goyal, Huiying Li, Whitney Sanders, and Eduard Hovy
    NAACL-HLT Meta4NLP Workshop, 2013.
    Syntax and Semantics
    [pdf]

2012

  1. A Topical graph-kernel for Link Prediction in Labeled Graphs
    Snigdha Chaturvedi, Hal Daume III, Taesun Moon, and Shashank Srivastava
    ICML Workshop on Mining and Learning with Graphs (MLG), 2012.
    Miscellaneous
    [pdf]