Prompt-Counterfactual Explanations for Generative AI System Behavior

ArXiv.org · 2026-01-06

As generative AI systems become integrated into real-world applications, organizations increasingly need to be able to understand and interpret their behavior. In particular, decision-makers need to understand what causes generative AI systems to exhibit specific output characteristics. Within this general topic, this paper examines a key question: what is it about the input -- the prompt -- that causes an LLM-based generative AI system to produce output that exhibits specific characteristics, such as toxicity, negative sentiment, or political bias. To examine this question, we adapt a common technique from the Explainable AI literature: counterfactual explanations. We explain why traditional counterfactual explanations cannot be applied directly to generative AI systems, due to several differences in how generative AI systems function. We then propose a flexible framework that adapts counterfactual explanations to non-deterministic, generative AI systems in scenarios where downstream classifiers can reveal key characteristics of their outputs. Based on this framework, we introduce an algorithm for generating prompt-counterfactual explanations (PCEs). Finally, we demonstrate the production of counterfactual explanations for generative AI systems with three case studies, examining different output characteristics (viz., political leaning, toxicity, and sentiment). The case studies further show that PCEs can streamline prompt engineering to suppress undesirable output characteristics and can enhance red-teaming efforts to uncover additional prompts that elicit undesirable outputs. Ultimately, this work lays a foundation for prompt-focused interpretability in generative AI: a capability that will become indispensable as these models are entrusted with higher-stakes tasks and subject to emerging regulatory requirements for transparency and accountability.

Prompt-Counterfactual Explanations for Generative AI System Behavior

Open MIND · 2026-01-06

As generative AI systems become integrated into real-world applications, organizations increasingly need to be able to understand and interpret their behavior. In particular, decision-makers need to understand what causes generative AI systems to exhibit specific output characteristics. Within this general topic, this paper examines a key question: what is it about the input -- the prompt -- that causes an LLM-based generative AI system to produce output that exhibits specific characteristics, such as toxicity, negative sentiment, or political bias. To examine this question, we adapt a common technique from the Explainable AI literature: counterfactual explanations. We explain why traditional counterfactual explanations cannot be applied directly to generative AI systems, due to several differences in how generative AI systems function. We then propose a flexible framework that adapts counterfactual explanations to non-deterministic, generative AI systems in scenarios where downstream classifiers can reveal key characteristics of their outputs. Based on this framework, we introduce an algorithm for generating prompt-counterfactual explanations (PCEs). Finally, we demonstrate the production of counterfactual explanations for generative AI systems with three case studies, examining different output characteristics (viz., political leaning, toxicity, and sentiment). The case studies further show that PCEs can streamline prompt engineering to suppress undesirable output characteristics and can enhance red-teaming efforts to uncover additional prompts that elicit undesirable outputs. Ultimately, this work lays a foundation for prompt-focused interpretability in generative AI: a capability that will become indispensable as these models are entrusted with higher-stakes tasks and subject to emerging regulatory requirements for transparency and accountability.

Beware of "Explanations" of AI

ArXiv.org · 2025-04-09 · 1 citations

Understanding the decisions made and actions taken by increasingly complex AI system remains a key challenge. This has led to an expanding field of research in explainable artificial intelligence (XAI), highlighting the potential of explanations to enhance trust, support adoption, and meet regulatory standards. However, the question of what constitutes a "good" explanation is dependent on the goals, stakeholders, and context. At a high level, psychological insights such as the concept of mental model alignment can offer guidance, but success in practice is challenging due to social and technical factors. As a result of this ill-defined nature of the problem, explanations can be of poor quality (e.g. unfaithful, irrelevant, or incoherent), potentially leading to substantial risks. Instead of fostering trust and safety, poorly designed explanations can actually cause harm, including wrong decisions, privacy violations, manipulation, and even reduced AI adoption. Therefore, we caution stakeholders to beware of explanations of AI: while they can be vital, they are not automatically a remedy for transparency or responsible AI adoption, and their misuse or limitations can exacerbate harm. Attention to these caveats can help guide future research to improve the quality and impact of AI explanations.

Observational vs. Experimental Data When Making Automated Decisions Using Machine Learning

INFORMS Journal on Data Science · 2025-06-03

Decisions supported by machine learning often aim to improve outcomes through interventions, such as influencing purchasing behavior with ads or increasing customer retention with special offers. However, using observational data to estimate these effects can introduce confounding bias. Although experimental data can mitigate confounding, it is not always feasible to obtain and can be costly when it is. This paper presents theoretical results focusing on the impact of confounding on decision making, emphasizing that optimizing decisions often involves determining whether a causal effect exceeds a threshold rather than minimizing bias in the estimate. Consequently, models built with readily available but confounded data can sometimes yield decisions as good as or better than those based on costly, unconfounded data. This can occur when larger effects are more likely to be overestimated or when the benefits of larger, cheaper data sets outweigh the drawbacks of confounding. We validate the theoretical findings using benchmark data from the 2016 Atlantic Causal Inference Conference causal modeling competition, encompassing 77 scenarios and 7,700 data sets. We then introduce theoretical conditions, weaker than ignorability, that characterize when confounding preserves effect rankings. These conditions allow for empirical heuristic tests to assess whether observational data aligns with this structure. Finally, we apply our findings in a large-scale case study using advertising data, demonstrating how these insights can guide decision making in practice. Funding: This research, including Yanfang Hou’s contributions, was supported by the Research Grants Council [Grant 26500822]. The authors thank Ira Rennert and the New York University/Stern Fubon Center for support. Data Ethics & Reproducibility Note: The code capsule is available on Code Ocean at https://doi.org/10.24433/CO.6587526.v1 and in the e-Companion to this article (available at https://doi.org/10.1287/ijds.2023.0012 ).

Naive Algorithmic Collusion: When Do Bandit Learners Cooperate and When Do They Compete?

SSRN Electronic Journal · 2025-01-01

What If the Prompt Were Different? Counterfactual Explanations for the Characteristics of Generative Outputs

2025-06-12

As generative AI systems become increasingly integrated into realworld applications, the need to analyze and interpret their outputs grows in importance.This paper addresses the challenge of assessing whether generative outputs exhibit specific characteristics-such as toxicity, a certain sentiment, or bias.We borrow a concept from the traditional Explainable AI literature-counterfactual explanations-but argue that it needs to be significantly rethought.We propose a flexible framework that extends counterfactual explanations to non-deterministic generative AI systems, specifically in scenarios where downstream classifiers can reveal characteristics of their outputs.

The Impact of Cloaking Digital Footprints on User Privacy and Personalization

Big Data · 2025-01-10 · 3 citations

Our online lives generate a wealth of behavioral records—digital footprints—which are stored and leveraged by technology platforms. These data can be used to create value for users by personalizing services. At the same time, however, it also poses a threat to people’s privacy by offering a highly intimate window into their private traits (e.g., their personality, political ideology, sexual orientation). We explore the concept of cloaking: allowing users to hide parts of their digital footprints from predictive algorithms, to prevent unwanted inferences. This article addresses two open questions: (i) can cloaking be effective in the longer term, as users continue to generate new digital footprints? And (ii) what is the potential impact of cloaking on the accuracy of desirable inferences? We introduce a novel strategy focused on cloaking “metafeatures” and compare its efficacy against just cloaking the raw footprints. The main findings are (i) while cloaking effectiveness does indeed diminish over time, using metafeatures slows the degradation; (ii) there is a tradeoff between privacy and personalization: cloaking undesired inferences also can inhibit desirable inferences. Furthermore, the metafeature strategy—which yields more stable cloaking—also incurs a larger reduction in desirable inferences.

Causal Post-Processing of Predictive Models

arXiv (Cornell University) · 2024-06-13

Organizations increasingly rely on predictive models to decide who should be targeted for interventions, such as marketing campaigns, customer retention offers, or medical treatments. Yet these models are usually built to predict outcomes (e.g., likelihood of purchase or churn), not the actual impact of an intervention. As a result, the scores (predicted values) they produce are often imperfect guides for allocating resources. Causal effects can be estimated with randomized experiments, but experiments are costly, limited in scale, and tied to specific actions. We propose causal post-processing (CPP), a family of techniques that uses limited experimental data to refine the outputs of predictive models, so they better align with causal decision making. The CPP family spans approaches that trade off flexibility against data efficiency, unifying existing methods and motivating new ones. Through simulations and an empirical study in digital advertising, we show that CPP can improve intervention decisions, particularly when predictive models capture a useful but imperfect causal signal. Our results show how organizations can combine predictive modeling with experimental evidence to make more effective and scalable intervention decisions.

The Illusion of Collusion

arXiv (Cornell University) · 2024-11-25 · 1 citations

Algorithmic agents are used in a variety of competitive decision-making settings, including pricing contexts that range from online retail to residential home rental. We study the emergence of algorithmic collusion when competing agents employ multi-armed bandit algorithms and competition is modeled as a repeated Prisoner's Dilemma game. Notably, agents in our setting perform online learning with no prior model of game structure and have no direct knowledge of competitor states or actions, thus they cannot learn strategies that depend on these factors. These context-free bandits nonetheless frequently learn seemingly collusive behavior, a phenomenon we term naive collusion. Our results reveal that whether naive collusion emerges depends starkly on the choice of behavior policy employed by bandit learners. The mechanism underpinning the emergence of collusive outcomes is synchronicity in agent action plays, where synchronicity captures how often agents play the same action. We show that in the long-run, naive algorithmic collusion never emerges when both agents use a broad class of persistently random algorithms, including the epsilon-greedy algorithm without epsilon decay, sometimes emerges when both agents use greedy-in-the-limit algorithms which feature randomness during exploration but are asymptotically deterministic, and always emerges when both agents use deterministic bandit learning algorithms like those in the well-known upper confidence bound (UCB) family. We highlight market and algorithmic conditions under which one can and cannot predict a priori whether collusion will occur. Our findings have several policy implications: preventing pricing algorithms from conditioning their actions on competitor prices may not preclude algorithmic collusion, symmetry in algorithms may increase collusion potential, and the emergence of algorithmic collusion is path dependent.

Who’s Watching TV?

Information Systems Research · 2023-04-05 · 2 citations

This work addresses the problem of “user disambiguation”—estimating the likelihood of each member of a small group using a shared account or device. The specific focus is on television set-top box (STB) viewership data in multiperson households, in which it is impossible to tell with certainty which household members watch what. We formulate user disambiguation as a predictive problem and develop a solution for estimating the likelihood that each individual in a multiperson household watches each TV segment. This method learns priors for viewership in single-person households and then adapts them to the specifics of each multiperson household’s viewership history. We formalize two ad hoc heuristics that are currently used in industry (and research) for estimating audience composition of STB data and conduct a comparative analysis using three data sources: simulated data, real large-scale viewership data, and fully labeled panel data. The results show that our method has superior performance. This approach has practical value for both advertisers and researchers who seek better understanding of TV viewership. It also has applications beyond TV advertising, such as detecting the sharing of streaming passwords among multiple households or any other situation in which multiple users share devices or accounts.