Craft Better Prompts: Using AI to Improve Your Predictions

Key Points:

• Collaboration with Experts: Phil Tetlock, in collaboration with experts from the Federal Reserve, MIT, and the London School of Economics, explores improving human forecasting through the use of large language models (LLMs).
• Study Findings: The study, titled “LLM Assistants Improve Human Forecasting Accuracy,” shows that collaborating with GPT-4 on forecasting tasks enhances decision-making by addressing cognitive biases such as overconfidence.
• Overconfidence in Decision-Making: Overconfidence, a common cognitive bias, leads to misjudgment of risks and overestimation of predictive accuracy. The study presents an innovative approach to mitigate overconfidence using LLMs.
• Experimental Design: Participants could consult with two types of AI advisors: a “superforecasting” model providing optimized guidance and a biased model exhibiting overconfidence. The control group used a less sophisticated model without direct forecasting assistance.
• Performance Improvement: Participants using advanced models—whether superforecaster or biased—showed a 23% improvement in forecasting accuracy compared to the control group, highlighting the benefits of structured AI assistance.
• AI-Assisted Decision-Making: AI can enhance human skills by encouraging users to scrutinize forecasts critically, consider a broader range of information, and challenge assumptions, leading to better decision-making outcomes.

Our focus is deeply rooted in leveraging AI to enhance decision-making, so you can imagine my enthusiasm when I discovered that Phil Tetlock, the author of Superforecasters and a pivotal figure behind many strategies in our book, Make Better Decisions, has collaborated with experts from the Federal Reserve, MIT, and the London School of Economics. Their goal? To investigate if human forecasting can be improved through the use of a large language model.

The study, titled LLM Assistants Improve Human Forecasting Accuracy, explores how AI augmentation interacts with human cognitive biases. The findings reveal that collaborating with GPT-4 on forecasting tasks can markedly improve decision-making processes, particularly by addressing issues of overconfidence and enhancing forecasting precision.

Overconfidence in decision-making is characterized by an inflated belief in one's own judgment, capabilities, or control over outcomes. It's been called the mother of all biases and most of us are overconfident most of the time. This cognitive bias often leads to a misjudgment of risks and an overestimation of one’s predictive accuracy, resulting in decisions that fall short of optimal, especially in complex scenarios where uncertainty is high and information is incomplete.

The paper finds an innovative approach to mitigating the adverse effects of overconfidence through the strategic use of LLMs, including an experiment involving a "biased" LLM designed to neglect base rates and exhibit overconfidence.

The design of the study is especially intriguing for its structured approach. It allowed participants the option to consult with two distinct types of AI advisors: one being a high-quality "superforecasting" prompt designed to provide optimized forecasting guidance, and the other, a biased advisor characterized by overconfidence and a disregard for base rates. Conversely, the control group was only equipped with a less sophisticated model that lacked any direct forecasting assistance.

Participants leveraging the advanced models—regardless of whether they opted for the superforecaster or the biased forecaster—demonstrated a 23% improvement in performance over those in the control group.

The improvement can be attributed to the enhanced cognitive framework provided by the advanced models, which aids in mitigating inherent biases and improving analytical capabilities. Even when using the biased advisor, participants benefited from engaging with a structured approach to decision-making that prompted critical thinking and a more careful consideration of available information.

The "superforecasting" prompt directly addresses and reduces cognitive biases like overconfidence and the neglect of base rates, leading to more accurate forecasts. Meanwhile, even the interaction with a biased model exposes forecasters to a diversity of perspectives, encouraging a reevaluation of their assumptions and methodologies. This dynamic illustrates the power of AI-assisted decision-making, where the mere act of systematic analysis and the challenge of inherent biases can significantly enhance human forecasting accuracy.

What at first seems paradoxical is actually a perfect example of how AI, used well, can enhance human skills. Contrary to conventional strategies aimed at reducing cognitive biases, by encouraging users to scrutinize forecasts more critically, people consider a broader range of information and perspectives thereby enhancing forecast accuracy.

Likewise, the interaction between human intuition and skepticism, coupled with the AI's computational efficiency, even when biased, can lead to a more balanced and insightful decision-making process. By confronting users with overconfident forecasts, it challenges them to reassess assumptions, critically evaluate evidence, and delve into counterarguments, which are vital for refining forecasting accuracy. Experiencing overt bias in AI outputs may sharpen the ability to recognize and adjust for similar biases in human thought processes.

How can you use this yourself? Here are some refined strategies for iterative engagement with an LLM for better forecasting and decision making:

Initiate Dialogue with Open-Ended Questions

Objective: Start with broad, open-ended questions to gather initial insights and set the stage for deeper exploration.

Sample Prompt: "What are the key factors influencing the future of remote work, and how might they evolve over the next five years?"

Follow Up with Specific Inquiries

Objective: Based on the LLM's initial response, ask targeted follow-up questions to delve deeper into specific areas of interest or uncertainty.

Sample Prompt: "Considering the factors you mentioned, which one is most likely to be underestimated in its impact on remote work adoption, and why?"

Challenge the LLM's Analysis

Objective: Critically evaluate and challenge the LLM's forecasts or assumptions, encouraging the model to provide justifications or consider alternative viewpoints.

Sample Prompt: "You predicted a high rate of adoption for remote work due to technological advancements. Can you explore scenarios where this might not hold true due to economic or social factors?"

Request for Counterarguments

Objective: Ask the LLM to present counterarguments or reasons why its initial forecast might be wrong, aiming for a balanced analysis.

Sample Prompt: "Based on your forecast, what are the strongest counterarguments or data points that could invalidate your conclusions?"

Engage in Scenario Analysis

Objective: Use the LLM to explore various scenarios, including best-case, worst-case, and most likely outcomes, to understand the range of possible futures.

Sample Prompt: "Can you provide a best-case, worst-case, and most likely scenario for the impact of artificial intelligence on job markets over the next decade?"

The approach highlights the synergistic potential of combining human and AI strengths. The research reveals that improving forecasting accuracy and decision quality may not solely rely on eradicating biases. Instead, it suggests a more strategic interplay, proposing that a conscious engagement with biases, under certain conditions, can catalyze superior outcomes.

Again we see a common dividing line, for now at least: humans bring nuanced judgment and contextual insight, while LLMs offer unparalleled data processing and pattern recognition capabilities. This synergy, even in the presence of intentional AI bias, can result in more effective decision-making by drawing on the best of both worlds.