Amplifying Foresight Compilation (AC Ep81)

“We wanted to see what the effect of AI might be on forecasting accuracy… to our surprise, we find that even when the model gives biased or noisy advice, human forecasters still improve—something we didn’t expect.”

– Philipp Schoenegger

“I kind of call these Gen AI systems a mirror. Pose it a question, play with scenarios, and see what comes out. It’s like an accelerant for thinking—pushing the boundaries of what’s possible.”

– Nikolas Badminton

“Future thinking is an everyday practice. It’s about becoming more aware of what’s happening around us, sensing signals, and collectively imagining what’s next.”

– Sylvia Gallusser

“The question of the future isn’t ‘How creative are you?’ but ‘How are you creative?’ Because what we can imagine, we can create—and we have a responsibility to build a better future.”

– Jack Uldrich

About Philipp Schoenegger, Nikolas Badminton, Sylvia Gallusser, & Jack Uldrich

Philipp Schoenegger is a researcher at London School of Economics working at the intersection of judgement, decision-making, and applied artificial intelligence. He is also a professional forecaster, working as a forecasting consultant for the Swift Centre as well as a ‘Pro Forecaster’ for Metaculus, providing probabilistic forecasts and detailed rationales for a variety of major organizations.

Nikolas Badminton is the Chief Futurist of the Futurist Think Tank. He is a world-renowned futurist speaker, award-winning author, and executive advisor, with clients including Disney, Google, J.P. Morgan, Microsoft, NASA, and many other leading companies. He is author of Facing Our Futures and host of the Exponential Minds podcast.

Sylvia Gallusser is Founder and CEO of Silicon Humanism, a futures thinking and strategic foresight consultancy. Previous roles include a variety of strategic roles at Accenture, Head of Technology at Business France North America, General Manager at French Tech Hub, and Co-founder at big bang factory. She is also a frequent keynote speaker and author of speculative fiction.

Jack Uldrich is a leading futurist, author, and speaker who helps organizations gain the critical foresight they need to create a successful future. His work is based on the principles of unlearning as a strategy to survive and thrive in an era of unparalleled change. He is the author of 9 books including Business As Unusual.

Websites:

Nikolas Badminton

Nikolas Badminton

Sylvia Gallusser

Jack Uldrich

University Profile:

Philipp Schoenegger

LinkedIn Profile:

Philipp Schoenegger

Nikolas Badminton

Sylvia Gallusser

Jack Uldrich

What you will learn

• How AI-augmented predictions enhance human forecasting
• The surprising impact of biased AI advice on accuracy
• Why generative AI acts as a mirror for future thinking
• The role of signal scanning in spotting emerging trends
• How creativity and imagination shape the future
• The evolving nature of community in an AI-driven world
• Why unlearning is key to adapting in a changing era

Episode Resources

People

• Philip Tetlock
• Jonas Salk

Books & Publications

• Superforecasting
• Facing Our Futures

Technical Terms & Concepts

• AI-augmented predictions
• Large language models (LLMs)
• The Ten Commandments of Forecasting
• The Ten Commandments of Superforecasting
• Forecasting accuracy
• Signal scanning
• Scenario planning
• Foresight strategy
• Generative AI
• Base rate
• Bias in AI
• Cognitive augmentation

Transcript

Ross Dawson: Now, it’s wonderful to see the work which you’re doing. Speaking of which, recently, you were the lead author of a paper, AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy.

So first of all, perhaps just describe the paper at a high level, and then we can dig into some of the specifics.

Philipp Schoenegger: Yeah. So the basic idea of this paper is: how can we improve human forecasting?

Human judgmental forecasting is basically the idea that you can query a bunch of very interested and sometimes laypeople about future events and then aggregate their predictions to arrive at surprisingly accurate estimations of future outcomes.

This goes back to work on Superforecasting by Philip Tetlock, and there are a lot of different approaches on how one might go about improving human prediction capabilities.

There might be some training—it was called The Ten Commandments of Forecasting—on how you can be a better forecaster. Or there might be some conversations where different forecasters talk to each other and exchange their views.

And we want to look at how we can—how we could—think about improving human forecasting with AI.

I think one of the main strengths of the current generation of large language models is the interactive nature of the back and forth, having a highly competent model that people can interact with and query whenever they want really.

They might ask the model, “Please help me on this question. What’s the answer?” They might also just say, “Here’s what I think. Please critique it.

And so this opens up for human forecasters a whole host of different interactions, and we wanted to see what the effect of this might be on forecasting accuracy.

Ross: So that’s fascinating. I suppose one of the starting points is thinking about these forecasters. So I suppose, just so people can be clear, human forecasting in complex domains is superior to AI forecasting because they don’t have those capabilities.

So now you’re saying humans are better than AI alone, but now the results of the paper suggest that humans augmented by AI are superior to either humans alone or AI alone.

Philipp: At the current ammount of papers that I have published, yes, but depending on when this airs, there might be another paper coming out that adds another twist to this.

But yes, in early work, we find that just a simple GPT-4 forecaster underperforms a human crowd, and on top of that, it underperforms just seeing 50% of every question.

But in this paper, we find that if we give people the opportunity to interact with a large language model, which in this case was GPT-4 Turbo, and we prompted it specifically to provide super forecasting.

So our main treatment had a prompt that explained The Ten Commandments of Superforecasting and instructed the model to provide estimates that take care of the base rate.

So you look at how often things like this have typically happened, quantify uncertainty, and identify branch points in reasoning.

But then we also looked at what happens if the large language model doesn’t give good advice. What if it gives what we call biased advice? It might be more noisy advice.

So what if the model is told to not think about the base rate—not think about how often things like this happen—to be overconfident, to basically give very high or very low estimates, and be very confident?

And to our surprise, we find that actually, these two approaches similarly effectively improve forecasting accuracy, which is not what we expected.

Ross: So I think that this is a really interesting point because, essentially, this is about human cognition.

It is human cognition taking very complex domains and coming up with a forecast of a probability of an event or a specific outcome in a defined timeframe.

So in this case, the interaction with the AI is a way of enhancing human cognition—th