The AI Forecasting Agenda
AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

AI forecasting systems are reported to match or outperform top human forecasters on conventional questions, but that performance has not been established for conditional or causal questions. A new Forecasting Research Institute benchmark is expected in the coming months to assess conditional forecasting; the available source does not specify a timeline for testing causal forecasts.

An Astral Codex Ten report says specialized AI forecasting systems now match or exceed top human performance on traditional forecasting questions, while warning that results have not been established for conditional and causal forecasts. The distinction matters because policy decisions often depend not just on whether an event will happen, but on what would happen if a particular choice or condition changed.

Age 18–24?Offer from Amazon

Prime made for students and young adults

  • Fast, free delivery for dorm and study essentials
  • Prime Video and Amazon Music included
  • Member-only deals
Try Prime for Young Adults Free trial for eligible 18–24 year olds
As an affiliate, we earn on qualifying purchases.

The report points to results on traditional forecasting tasks, including ForecastBench and the Metaculus Cup. It describes current high-performing systems as specialized AIs paired with forecasting tools, or “harnesses,” and names forecasting startups FutureSearch, Mantic and Preseen. It also says general-purpose models such as ChatGPT and Claude were about six months behind specialized systems at the time of the report; its suggestion that they might catch up by year-end is a forecast, not a confirmed outcome.

The report’s main focus is what happens when the question changes. A conventional question asks whether a candidate will win an election. A conditional question asks, for example, how the chance of a market crash would change if a particular candidate won. The report says companies offer conditional forecasting as a feature, but it remains uncertain whether their systems perform well on it. It cites a Forecasting Research Institute test from about a year earlier that found the AI systems tested then did significantly worse on conditional questions.

The author says experts consulted for the report expected conditional performance to improve gradually as systems become more capable, rather than suddenly reaching parity. Those experts did not offer a firm estimate of whether current AI systems outperform an average well-informed person on such questions, and considered them below top superforecasters. The institute is working on a new benchmarking experiment that the report says should provide a clearer measure in a few months.

At a glance
reportWhen: Published in an Astral Codex Ten report…
The developmentAn Astral Codex Ten report outlines the remaining research tests for AI forecasting after systems reached top-human performance on traditional forecasting questions.

Testing Forecasts for Policy Decisions

Strong results on questions with clear outcomes could make forecasts easier to obtain and apply. The report contrasts AI systems with hiring teams of superforecasters, which it says can cost thousands of dollars and take weeks, and with prediction markets, whose questions may be limited and whose resolution rules can be contested. These are the author’s comparisons; the report does not provide a systematic cost or performance study of all those options.

The larger potential use is evaluating choices. Policymakers and researchers may want to know whether changing a policy affects an outcome, not merely whether the outcome is likely. For example, a forecast about crime after a change in police funding does not by itself show that the funding change caused the result. If AI systems cannot reliably handle conditional or causal questions, their strong performance on ordinary event forecasts would not settle those policy questions. Forecast accuracy and causal evidence are different claims, and the report treats the latter as an open research problem.

Roborock Qrevo S Pro Robot Vacuum and Mop with Multifunctional Dock

Roborock Qrevo S Pro Robot Vacuum and Mop with Multifunctional Dock

Ninja Foodi Smart Dual Heat Air Fry Oven, 13-in-1, Silver, SP351

Ninja Foodi Smart Dual Heat Air Fry Oven, 13-in-1, Silver, SP351

As an affiliate, we earn on qualifying purchases.

From Event Odds to Causation

Traditional forecasts usually concern an observable event with a defined resolution, such as an election result. Conditional forecasts ask about an outcome under a specified circumstance. The report argues that conditional questions are closer to policy decisions, but notes that prediction markets have practical difficulties with them: if the stated condition does not occur, contracts may need to be refunded.

Even a conditional forecast does not automatically establish cause and effect. If one party wins an election and the economy later performs poorly, the earlier state of the economy or other events may have influenced both the election result and the later outcome. The report says a causal question requires considering such relevant factors, potentially by combining forecasts across different conditions. It presents this as a possible approach, not as a validated method or a demonstrated AI capability.

“AI forecasters now equal or beat top human performance at traditional forecasting questions.”

— Astral Codex Ten report

How Far Accuracy Carries

The source does not give the benchmark scores, sample sizes, or detailed comparison methods behind the reported results on traditional questions. It also does not identify the exact AI model versions or the full set of human forecasters used as comparisons. The claim that general-purpose models were about six months behind is an estimate in the report, and its year-end prediction is not verified here.

For conditional forecasts, the report says experts were unwilling to estimate current performance relative to average smart humans, while judging systems still below top superforecasters. The planned institute benchmark may clarify performance, but the source provides no detailed protocol or publication date. For causal forecasts, it describes a conceptual way to use many conditional estimates, but reports no finding that this approach works reliably. The scope of the report’s claims should therefore remain limited: traditional forecasting results do not establish causal accuracy.

A Benchmark for Conditional Forecasts

The next reported milestone is a Forecasting Research Institute experiment intended to measure AI performance on conditional questions. The report expects results in a few months but does not provide a specific date. Its findings could show whether recent progress on conventional questions has carried over, and how current systems compare with human forecasters on this different task.

The source does not announce a timetable for a comparable test of causal forecasting. Until such evidence is available, readers should treat the proposed use of AI forecasts for policy evaluation as a research possibility, not a demonstrated capability. The immediate open question is whether systems can reliably reason about specified conditions; a separate challenge is whether they can isolate what caused an outcome.

Key Questions

What is the news in the AI forecasting report?

The report says specialized AI forecasters match or outperform top humans on traditional forecasting questions, while identifying conditional and causal forecasting as unresolved tests.

What is a conditional forecast?

It estimates an outcome under a stated condition—for example, how the chance of a market crash might differ if a particular candidate wins an election.

Does a conditional forecast prove that a policy caused an outcome?

No. A conditional relationship can reflect other factors that affect both the condition and the outcome. The report says causal forecasting requires accounting for relevant variables and has not been shown to work reliably with AI.

When will there be more evidence?

The report says the Forecasting Research Institute is working on a conditional-forecasting benchmark expected in a few months. It does not give a specific release date or a timeline for a causal-forecasting test.

Source: astralcodexten.com

You May Also Like

Will It Rain In Chicago On Aug 27, 2026?

Market data indicates a possibility of rain in Chicago on August 27, 2026, but no definitive weather forecast is available yet. Read for details.

Cambridge, UNILAG Identify 10 Locations For Urban Heat Research In Lagos – THISDAYLIVE

Cambridge University and UNILAG have identified 10 locations across Lagos for urban heat research, aiming to address climate challenges in the city.

Will Minneapolis, MN Face Tornado Risk On September 21?

Weather forecasts suggest potential tornado risk for Minneapolis on September 21, but no official warnings have been issued yet. Details remain uncertain.

Summer solstice brings Portland nearly 15 hours of daylight

Portland’s summer solstice brings approximately 15 hours of daylight, marking the longest day of the year. The event is confirmed for June 21, 2026.