Special Volume: Updating and Reviewing our 15+ Prior Forecasts
Metaculus Mondays Forecast Retrospective

Special Volume: Updating and Reviewing our 15+ Prior Forecasts

Clay Graubard
Andrew Eaddy
Clay Graubard, Andrew Eaddy

Table of Contents

Welcome to Volume 7 of Metaculus Mondays!

Unlike past volumes, we will not make any new forecasts this week. Instead we will review our previous forecasts, see how they compare to the Metaculus community’s median today, and determine whether or not to update our forecast.

Updating forecasts is one of the ten commandments of Superforecasting: The best forecasters are those who generally update their predictions more often (approximately 2x) than general forecasters. This volume is therefore designed to force us to update our predictions and hopefully end up with stronger forecasts. Let’s begin!

Summary

Open Questions

NK Tests ICBM again before 2022

  • GG: 33%
  • Community: 55%
  • Close Date: 10/14/21
  • Volume: 4
New / Revisited EvidenceEffect
Recent AirForce General StatementWeak-Moderate: 52%-48% Weight Towards ICBM Test
US Intelligence AssessmentWeak-Moderate: 52-48, ICBM Test
Ignoring Back Channel CommunicationsWeak: 51-49, ICBM Test

Over the last week North Korea has made news headlines as they seem to be approaching a potential ICBM test in the near future. On March 17th Choe Son Hui, the first foreign minister, stated that “We don’t think there is a need to respond to the U.S. delaying-time trick again...” This statement was in reference to offers from the United States to enter into talks regarding North Korea’s nuclear future. Hui went on to share that until the United States makes concrete steps to roll back their oppressive sanction regime on the country, talks would continue to be ignored.

Around the same time, U.S. Air Force General Glen VanHerck shared that the ICBM unveiled in North Korea last October may be tested in the near future, given Kim Jong-Un’s comments about the April 2018 moratorium placed on North Korea regarding long-range missiles and nuclear testing. Finally, CNN reported on March 17th that “North Korea could be preparing to carry out their first weapons test since President Joe Biden came into office.” While key decision-makers in the United States foreign affairs apparatus were not able to comment on these reports for security reasons, there are a number of factors that would lead people to believe these warnings are credible.

Some people in the White House believe that not only has North Korea seen success in the development of the nuclear program, but that former president Trump’s lax stance on Kim Jong-Un’s actions at the end of his presidency may embolden North Korea to test an ICBM in the near future. Some have also shared that there is historical precedent that North Korea may try to test President Biden early into his presidential tenure, making a test more likely.

All of these lead us to believe that the likelihood of an ICBM test by North Korea before 2022 is more likely than originally forecasted, increasing our forecast from 33% likelihood to a 38% likelihood (37.6%).

Credible claim by 2024 of Lab-Leak Hypothesis for COVID-19

  • GG: 19%
  • Community: 19%
  • Close Date: 12/31/23
  • Volume: 6
New / Revisited EvidenceEffect
WHO-led Team Believes Wildlife FarmsWeak: 51%-49% Weight Towards Zoonotic
Change in Confidence of Chimera to 80%Moderate: Lab Leak
80,000 Hours Interview with Andrew C. WeberWeak: 51-49, Lab Leak

Since we first made this forecast last week, we’ve done further research on the subject and added a few new pieces of evidence affecting our overall forecast.

The first piece of evidence comes from a WHO-led mission, which believes “wildlife farms in southern China are the most likely source of the COVID-19 pandemic” according to NPR. Although there is some evidence provided behind this hypothesis (mainly that China shut down the exotic farms in southern China that could have been a bed for SARS-CoV-2 to emerge), the messenger (who reportedly has a conflict of interest) and lack of concrete evidence makes this evidence have a weak overall effect in favor of a zoonotic origins.

Moreover, we’ve further done more research and read further analyses which increases our confidence in the chimeric analysis conducted by Rootclaim from 70% to 80%--decreasing the effect multiple on Zoonotic origins from ⅓ to ⅕.

Finally, a comment in the Metaculus question pointed us to an interview with Andrew C. Weber, a former Assistant Secretary of Defense for Nuclear, Chemical & Biological Defense Programs and Deputy Coordinator for Ebola Response. In a podcast interview, Weber--an Obama appointee--stated a 50-50 confidence that SARS-CoV-2 was a result of a lab leak incident. We found this to be weak evidence in favor, showing further institutional-type buy-in.

The overall result is increasing our overall confidence that SARS-CoV-2 was a result of a leak-leak incident to 48.45%.

Accounting for political and other factors, we still apply a 50% discount to this probability, therefore increasing our likelihood of a lab-leak origin for COVID-19 from 19% to 24%.

Credible claim by 2022 of Lab-Origins for COVID-19

  • GG Forecast: 25%
  • Community: 14%
  • Close Date: 5/1/21
  • Volume: 6
New / Revisited EvidenceEffect
WHO-led Team Believes Wildlife FarmsN/A
Change in Confidence of Chimera to 80%N/A
80,000 Hours Interview with Andrew C. WeberN/A
Unlike the other COVID-19 origins forecast, none of the above evidence is related to resolution criteria or the community forecast and therefore has no impact on our forecast.

Will there be a 'WW3' before 2050?

  • GG: 9%
  • Community: 14%
  • Close Date: 12/31/44
  • Volume: 1
New / Revisited EvidenceEffect
Greater than expected continued hostility
between US and China
Weak-Moderate: 52%-48% Weight Towards WW3
Slightly Greater than expected continued
hostility between US and Russia
Weak: 51-49, WW3
Scope and Appearance of US RetrenchmentWeak: 51-49, WW3

In the last episode of the Global Guessing Weekly Podcast, we discussed the combative remarks expressed by Secretary of State Anthony Blinked and his Chinese counterpart at the Alaska Summit last week. Between the brewing U.S.-China tensions, and Biden butting heads with Russia early in his presidential term, it is a viable hypothesis that, should these relations continue to deteriorate, major conflict is possible.

There are a number of global trends and political science theories that push back on the aforementioned narrative, however, some of which are persuasive enough to withstand the potential influence of those stories. With the introduction of Deterrence Theory and Democratic Peace Theory, imagining another World War is extremely difficult. Even Clay’s favorite author has written about the growing peace in the world in his book The Better Angels of Our Nature.

While these theories will keep our forecast low, lower than the community median of 14%, the geopolitical trajectory set by Donald Trump, and being managed by President Biden, cannot be ignored. We’ve also talked about waning U.S. influence on the world stage and the growth of the ‘World Island” with Ross of AR Global Security on episode 4 of the Global Guessing Weekly Podcast. Taking all of this together we believe that conflict of the magnitude provided in the question criteria, a World War III, is more likely than our original forecast by 1%.

We therefore slightly increase our forecasted probability of WW3 by 2050 to 10%.

Peace in Yemen in 2021

  • GG: 11%
  • Community: 45%
  • Close Date: 7/15/21
  • Volume: 6
New / Revisited EvidenceEffect
Houthi drone strike on Saudi oil refinery
+ Saudis lead retaliatory airstrike on Yemen's capital
Weak: 51%-49% Weight Towards Continue Fighting
Houthi major territorial advances Moderate: 55% Continued Fighting,
55% Peace / Houthi Victory, 45% Ceasefire

In recent weeks, events have occurred surrounding the Yemeni-Saudi conflict that have had the potential to affect our likelihood that Yemen sees a 30-day period of peace in 2021. On the Yemen-side of the conflict, Yemeni Houthi rebels claimed responsibility for a drone strike on an oil refinery in the Saudi capital of Riyadh. The attack consisted of six drones and hit facilities belonging to Saudi Aramco, one of the largest companies (by revenue) in the world. The Houthis have also made territorial advances in Yemen, seizing boht city and mountain territory previously held by the Yemeni government.

On the Saudi side of the conflict, an attack was carried out in the Yemen capital of Sana’a in retaliation for the oil refinery strike, hitting key Houthi military bases as well as a Yemeni port, hurting the country’s economy. The recent escalation in violence between Yemen and Saudi Arabia is thought to be a potential hindrance to any peace deal in the near future. Specifically, this escalation might result in a tit-for-tat series of attacks from both countries, a strategy derived from mathematical psychologist Anatol Rapoport in one of political scientist Robert Axelrod’s game theory tournaments in the 1970s and 1980s. If you have an interest in political science, political theory, or game theory, definitely look into those tournaments!

Given our sensitivity towards moving percentages below 10% or over 90%, we want to be more moderate when examining the recent news. We find the air- and drone-strike news to be more likely noise rather than signal. Conflict between Yemen and Saudi Arabia has persisted at a high level since 2015, making this news, while potentially relevant to our forecast, not novel in any way. On the other hand, we do not want to completely discount the effect of the Houthis recent territorial gain.

As a result, we slightly decrease our likelihood of peace in Yemen in 2021 from an initial forecast of 11% to a 10% probability.

US rejoins Iran Deal before 2022

  • GG: 15%
  • Community: 44%
  • Close Date: 8/31/21
  • Volume: 2
New / Revisited EvidenceEffect
Recently Analysis by Valdai ClubWeak-Moderate: 52-48, No Return

According to Valdai Club, a Moscow-based think tank with a focus on foreign affairs, President Biden will be more likely to lift Trump-era sanctions on Iran in the short-term than to strike a new deal akin to that of 2015. This move would be a huge boon for Iran’s economy, which has been choked due to Trump’s sanctions on Iran’s rare earth production and other economic aspects.

The Club’s report goes on to suggest that Biden priority regarding Iran should be getting back in the good graces of the United States’ allies i.e. Western Europe as opposed to actually solving the issue of Iran in the short-term. This thinking supports the idea that Biden may work to remove the most oppressive sanctions on Iran imposed in the last four years, but take longer to reach a long-term solution with the country.

Iran has also been accused of manufacturing the missiles used in the Houthi attack on Saudi Aramco facilities last week, as well as threatening to kill a United States Army general by attacking a military base in Washington D.C.

Given the rising tensions between the United States and Iran, the other foreign affairs priorities for President Biden, and the still-existing time constraints on the resolution criteria proposed on Metaculus, we are lowering our likelihood that the U.S. rejoins the JCPOA from an already-low 15% to 14%.

SA, Israel to establish diplomatic relations?

  • GG: 21%
  • Community: 40%
  • Close Date: 5/31/21
  • Volume: 3
New / Revisited EvidenceEffect
Informal talks on 4-State Defense Alliance Against Iran Moderate-Weak: 53-47, Establishment
SA and Qatar Deny Israel Normalization Until Palestine ResolutionWeak-Moderate: 52-48, No Establishment
Netanyahu Promises Direct Flights to IsraelWeak: 51-49 Establishment
Overstated Economic Importance of Deal in Short-term for Saudi ArabiaWeak-Moderate: 52-48 No Establishment

Relations between Saudi Arabia and Israel have been wildly active since the Abraham Accords were put in place in 2020. Activity between the two countries have picked up in recent weeks however, as questions around Saudi Arabia’s normalization of relations with Israel have increased.

Last weekend Israeli Prime Minister Benjamin Netanyahu promised voters that he would establish direct flight routes between Israel and Saudia Arabia should he be reelected in this week’s legislative elections in Israel. As we predicted in our original article, the results of that election will likely be influential in the outcome of this normalization discussion. While direct flights are not a sufficient indication of normalization, they are certainly a signal of open talks between the countries which may belie normalization in the near future.

This stance is not new for the Israel Prime Minister, however. In early March, Netanyahu demonstrated his desire to get closer with Saudi Arabia when he opened communication with Saudi Arabia, the UAE, and Bahrain to expand military and intelligence communication to mitigate a potentially threatening Iran. The UAE and Bahrain have already normalized relations with Israel, putting pressure on Saudi Arabia to join the group.

Talks first opened in 2015 when they shared a feeling that the JCPOA deal was insufficient to keep Iran under control. This explains their warm reception to former president Trump’s ‘iron fist’ with Iran. But now that President Biden seems close to removing the effects of that ‘iron fist’ and potentially entering into an even more concessionary agreement with Iran, talks have heated back up. To this end, Israeli Intelligence Minister Eli Cohen shared last week that Saudi Arabia was one of four countries (including Oman, Niger, and Qatar) that are closest to normalizing relations with Israel.

All of these indicators would suggest that we should increase our prior forecast of 21% closer to the current community median of 40%. However there are two bearish factors drawing down that number. First, an article from Bloomberg reported that Saudi Arabia’s minister of state for foreign affairs, Adel Al-Jubeir said normalization of relations with Israel is predicated on their solving the Palestine issue in a way that is amenable to Saudi Arabia.

Unlike some of the other countries that have normalized relations with Israel, Saudi Arabia is incredibly wealthy and doesn’t need beneficial trade deals or commerce to prosper, so the economic incentives to normalize relations with Israel are not as strong of a pull as they might be for other countries.

Second, given Saudi Arabia’s importance to the Arab World, it may be necessary for other countries in the Levant, Gulf, and North Africa to normalize relations first so that it can appear Saudi Arabia has a mandate from the Arab World to move forward with this trend. And finally, MBS is still second-seat to his father when working with the United States making his influence on a potential deal moot.

Taken together, these factors create a wash, keeping the likelihood of Saudi-Israeli normalization by 2022 even at a 21% likelihood.
  • One factor that we are watching closely is the result of the upcoming Israeli election. Tune in next week to see how it impacts our overall forecast.

Nuclear weapon detonation as an act of war by 2050?

  • GG: 10%
  • Community: 20%
  • Close Date: 12/31/39
  • Volume: 1
New / Revisited EvidenceEffect
Pakistan-India potential rapprochementWeak-Moderate: 52-48, Against Detonation

When we initially calculated our forecast for the likelihood that there would be more than two nuclear weapons detonated as an act of war by 2050, we first considered which countries we believe would be most likely to engage in this sort of activity, and the most heated conflict areas where nuclear weapons exist. This led us to examine the case of India and Pakistan, where both countries have animosity towards one another and also possess nuclear weapons. In our analysis we predicted that if a war of nuclear-proportion were to begin in the next 40 years, that conflict area would likely be involved. However, recent news released by Al Jazeera regarding the relationship between the two nations seems to be pouring water on that hypothesis.

According to the source, Pakistan’s army chief said that ‘it is time to bury the past’ with India, a comment following a recent ceasefire between India and Pakistan. With the Kashmir dispute having strong potential to lead to escalated war in the future, these statements and actions signal that may very well not happen. Given the lengthy time horizon for this forecast as well as our (we know we keep re-stating it!) new consideration regarding forecasting percentages below 10% of above 90% this news is likely not significant enough to greatly affect our ultimate prediction, especially as bilateral ties between the countries could easily shift back to one of violence in the coming 40 years.

Therefore we update our forecast slightly lower to a 9% likelihood another nuclear weapon is used as an act of war by 2025.

Vaccine Doses Administered in EU by July

  • GG: [25% = 190M, Median = 249M, 75% = 324M]
  • Community: [25% = 148M, Median = 183M, 75% = 236M]
  • Close Date: 5/29/21
  • Volume: 4
New / Revisited EvidenceEffect
Not on Track to Reach Herd ImmunityLow: Bearish on Doses
AstraZeneca Confidence PlummetsModerate: Bearish
EU Now Threatening Stopping ExportsUnclear

There have been three major developments in the EU vaccination campaign since we made our first forecast.

First, the EU currently remains off-track on reaching herd immunity in 2021 despite pledges that 70% of the adult population will be vaccinated by the end of summer.

Second, the recent decision of many EU member countries to pause dosages in the AstraZeneca vaccine, after previously disparaging the virus publicly, has caused a significant drop in confidence of EU citizens in the shot. For instance, only 20% of French adults trust the vaccine. Lack of vaccine trust will hamper vaccine rollout and therefore decrease the number of doses the bloc could administer by the end of July.

Finally, the EU President is now threatening to stop the export of viruses made within the bloc after AstraZeneca announced a cut in their projected number of delivered vaccines to the EU. We are unclear of how this evidence will impact the short-term numbers of doses administered. On one hand, actually banning the export of vaccines made within the EU (who currently exports the second most vaccines in the world behind China, at over 34 million so far) could potentially increase the number of doses administered.

On the other hand, the threat indicates that vaccinations are falling far behind and drastic actions are needed. It is unclear whether the effect of a ban could overcome the bearish indicator of needing to ban exports by the end of July. Moreover, while the EU currently exports a large number of vaccines, it also has millions of unused dosages making it unclear the effect of a greater supply in the short-run.

As a result of these factors, we lower the forecasted administered COVID-19 vaccine doses in the EU by July to:
  • 25%: 142.5M (75% decrease )
  • Median: 195M (80%)
  • 75%: 259.2M (80%)

Vaccine Doses Administered in EU by October

  • GG: [25% = 340M, Median = 439M, 75% = 538M]
  • Community: [25% = 318M, Median = 446M, 75% = 568M]
  • Close Date: 9/29/21
  • Volume: 4
New / Revisited EvidenceEffect
AstraZeneca Confidence PlummetsLow-Moderate: Bearish
EU Now Threatening Stopping ExportsLow: Bullish
Not on Track to Reach Herd ImmunityUnclear

Unlike the July forecast, we believe the effects of the AstraZeneca confidence drop will subside overtime. Moreover, we believe the greater time horizon reduces the effect of the EU’s current position on their future success. Finally, we instead give a slight positive impact of the EU’s export ban threat for their long-term vaccination effort.

As a result, we slightly lower the forecasted administered doses of COVID-19 vaccine in the EU by October to:
  • 25%: 291M (85% reduction)
  • Median: 421M (95%)
  • 75%: 511M (95%)

We also changed the distribution of our forecast to capture a wider-range of outcomes on the lower-end of doses (200-450) than on the higher-end of doses (600+). While it is possible for the number of doses to greatly exceed 600 should boosters be necessary for a potential future variant, it is our current intuition that this is far less likely than the current trajectory. However, this is something we are likely to reconsider and come back to alter.

  • Do you disagree or have a different perspective? Let us know in the comments or on social media.

Will any Member State leave the Eurozone by 2025?

  • GG: 4%
  • Community: 15%
  • Close Date: 1/1/22
  • Volume: 5
New / Revisited EvidenceEffect
No change

Similar to the other predictions in this article which we have decided not to change, the issue of Eurozone membership has not been broached in any significant way in the news recently. There has been conversation around Europe’s ailing economy as a result of Covid-19-related factors, with some members re-entering lockdown protocols due to spikes in certain variants. But given the Eurozone’s economic and political cohesiveness which was bolstered by its (previously-discussed) shared debt structuring, these economic woes are not indicative of a Eurozone departure by 2025.

Episodes of increased inflation will be a phoenome to watch for, as it has the potential (albeit small at this point) to create a situation where a country may be inclined to leave the Eurozone, but for now things seem stable enough for us to keep our prediction as is.

Number of major wars in the 2020s

  • GG: [Win points: 2, 3, 5, 6; Break-even: 4]
  • Community: 3.99
  • Close Date: 12/31/23
  • Volume: 2
New / Revisited EvidenceEffect
No change

Unlike some of the other prediction updates and retrospectives in this article, there has been no recent news that would lead us to change our original forecast on this question, especially given the far-off time horizon of this question's resolution. The community median on Metaculus also remains unchanged since our initial prediction, giving us more confidence that more long-term observation will be required to accurately determine which events will truly impact the outcome of this question. Given all of this, our forecast for this question remains as displayed above: Points at 2, 3, 4, 5, 6; Break-even at 4.

  • GG: 85%
  • Community: 70%
  • Close Date: 2/17/26
  • Volume: 3
New / Revisited EvidenceEffect
Starlink, NASA collaboration No change

Given this question’s extremely long time horizon, we concluded that most of the news regarding SpaceX and Starlink that has been released recently is not significant enough to affect our current prediction percentage. With that being said, SpaceX has announced more close collaboration with NASA, namely Space Act Agreement which will preside over coordination and space safety between SpaceX’s satellites and the ISS as well as other space crafts.

While not a new phenomenon -- SpaceX and NASA have collaborated in the past -- it might be an indication that SpaceX is trying to smooth over potential hurdles leading to an IPO at some point in the future. Again, however, given the time horizon of the question, the effect of this agreement on our forecast is too difficult to discern at this moment.

Closed Questions

% S:N501 in U.S. for week of 29 March?

  • GG: [25% = 71, Median = 78, 75% = 85]
  • Metaculus: [25% = 60, Median = 73, 75% = 83]
  • Community: [25% = 57, Median = 71, 75% = 83]
  • Resolution Date: 4/12/21
  • Volume: 1

As things stand right now, this prediction is going to be a fairly large miss. In the week of March 8th, S:N501 COVID-19 variants made up 14% of all sequenced viruses in GISAID. According to Dr. Fauci, the B.1.1.7 variant from the UK (the dominant S:N501 variant in the United States) made up “only” 30% of all new infections last week.

Why will this forecast be such a miss? We can identify two major mistakes.

First, too much of our growth assumptions were based in smaller, geographically-different countries in comparison to the United States. For instance, one of the studies we cited stated their growth forecast was based on the growth path seen in the United Kingdom.

  • What could we have done differently? We should have compared COVID-19 growth in the UK from February-March 2020 to the United States’ COVID-19 growth from the same period. We likely would have a greater lag in terms of geographic spread which would have cautioned us towards any rapid domination of variants in comparison to Europe.

Second, we did not strongly consider the possibility that other variants existed--and more importantly that those variants could be from the United States which would make amassing a large foothold of S:N501 by the start of April difficult (of course the variants could have been in the same family, but that would not be guaranteed).

  • What could we have done differently? We should have modeled the emergence of Variants of Concern to the number of cases-per-capita a country had. Such an analysis would have indicated a highly likelihood of US variants existing.

Vaccine Doses Administered in EU by April

  • GG: [25% = 65M, Median = 75M, 75% = 87M]
  • Metaculus: [25% = 58M, Median = 65M, 75% = 72M]
  • Community: [25% = 55M, Median = 64M, 75% = 76M]
  • Resolution Date: 4/2/21?
  • Volume: 4

We already discussed what likely went wrong with this forecast in last week’s volume of Metaculus Mondays:

  1. Overstating the EU’s ability to speed up vaccinations at a similar (but delayed) pace as the United States.
  2. The ability for public health officials in many European Union member states to make the decision to temporarily halt AstraZeneca vaccinations (and subsequently tanking public trust in that vaccine, as we mentioned earlier).

U.S. Troops in Afghanistan 2021

  • GG: 90%
  • Metaculus: 87%
  • Community: 82%
  • Resolution Date: 5/1/21
  • Volume: 5

As we approach the resolution date for Metaculus’ question regarding U.S. troop withdrawal from Afghanistan, news has been released that is pertinent to our prediction. Last week it was announced that, with the May 1st deadline for troop withdrawing rapidly approaching, President Biden is considering a 6-month extension for the troop withdrawal plan first arranged by former President Trump in 2019, saying that a May 1 withdrawal will be “tough.”

This extension has been chalked up to a few recent developments in the region. First, Sen. Bob Menendez, chairman of the Senate Foreign Affairs Committee, shared that "The Taliban is clearly not abiding by all of its Update commitments under the Feb. 29 agreement, and it's raising serious questions about the future of Afghan security and governance..." This is likely referring to the continued violence that has spread over Afghanistan in the past month, as well as the Talbian having yet to cut ties with Al-Qaeda.

Given these developments and quotes from key decision-makers in the United States’ government, as well as the already-existing time constraints for this question, we believe that these developments significantly strengthen our confidence in our prediction. However, after our most recent podcast interview (releasing this coming Saturday) we have gained a new sensitivity for prediction percentages above 90% or below 10% (when you enter those ranges, the level of confidence/evidence needed should scale proportionately) so we are keeping our 90% forecast but increasing our conviction in that percentage.