Welcome to Volume 6 of Metaculus Mondays!
There have not been any question resolutions since the last volume was released, but we can probably kiss our closest EU vaccination prediction goodbye (75M doses by April 1). If you haven’t followed the story, many European Union member states decided to pause AstraZeneca vaccinations this week after some health officials realized 37 of 17,000,000 vaccinated people–or 0.0002% of those vaccinated–had a reported blood clot.
Was this a smart move? Probably not. As cited in a 2018 article in Thrombosis Research (Blood Clot Research), 543,454 people die from blood clots in the EU every year (or 45,288 people per month). 37 blood clots per 17 million applied to the entire EU population would mean 971 blood clots (not deaths) over the past 2.5 months. (Edit: Turns out the 17M vaccinated includes people in the UK, so this is more like 1093 blood clots.)
Mini prediction retrospective: We underestimated the EU’s ability to...well, do the most EU thing: Pause vaccinations during a pandemic without compelling, significant evidence.
Another one of our forecasts is set to resolve shortly, but we expect to get this one right: Whether U.S. troops will remain in Afghanistan after the May 1st deadline. We gave a 90% likelihood troops stay. Since that forecast, seemingly every outlet across the political spectrum has told Biden the reasons why troops should stay–at least for now, they say–and it came out we had 3,500 troops left in Afghanistan and not 2,500. If we had the chance to update our forecast before it closed today, we would’ve upped it to a 92.5% likelihood, but we forgot. Clay swears the countdown timer deceived him. 🤷
Moving ahead to this week’s volume, we want to provide some updates on the Metaculus community and forecast how the conflict in Yemen will unfold throughout the rest of 2021. But first, we want to move past our usual uncontroversial topics—US rejoining the JCPOA, future of Saudi-Israeli relations—and tackle a question that has implications for global health regulations, intergovernmental institutions, and relations between states.
*Deep breath* What was the source of SARS-CoV-2?
Where Did Covid-19 Originate?
No one knows the origin of SARS-CoV-2. And we are not guaranteed to ever know for sure. Although we first identified Ebola in 1976, we still do not know its exact origins and it took over a decade to identify the origins of COVID-19’s younger brother, SARS. And for months China blocked (and continues to interfere with) investigations into the origins of COVID-19.
There may be no definitive thesis yet, but there are four major hypotheses worth mentioning:
- The virus evolved in nature and was transmitted to humans from animals
- The virus evolved in nature and was transmitted to humans when researchers attempted to collect virus samples from animals
- The virus was collected from either animals or humans, genetically modified during lab research, and was released by accident
- The virus was genetically engineered as a bio-weapon and was deliberately released
Although preliminary findings from the World Health Organization (WHO) indicate uncertainty of the virus’ origins, it is the WHO’s opinion that the “most likely” possibility was that SARS-CoV-2 originated in animals. This would indicate a zoonotic origin (hypothesis 1 and 2). And while uncertain of the source, the WHO is mostly certain what the source was not: A lab leak–or hypothesis 3.
The Metaculus community appears to agree, at least based on their forecasts to these two questions:
- Community Median: 13% Likelihood
- Community Median: 19% Likelihood
According to the December 30, 2020 NPR story, “Even If It’s ‘Bonkers,’ Poll Finds Many Believe QAnon And Other Conspiracy Theories”, 40% surveyed American adults believed “coronavirus was made in a lab in China even though there is no evidence for this” (emphasis added).
So we’re not about to make a prediction indicating otherwise. Right?
Although you can never make enough, here are a few disclaimers before we get to our forecast.
First of all, we do not have any significant confidence that SARS-CoV-2 was created as a biochemical weapon or released from the Wuhan Virology Lab or any other laboratory intentionally. Although we are using Rootclaim’s analysis (more on this shortly) as the template for our prediction, we remove the probability of biochemical origins established in their analysis and normalize the remaining probabilities to determine our prediction.
Second, considering the possibility that the virus was released by accident is not supposed to be political. As Nicholson Baker writes in “The Lab-Leak Hypothesis” published in January’s New Yorker:
A lab accident — a dropped flask, a needle prick, a mouse bite, an illegibly labeled bottle — is apolitical. Proposing that something unfortunate happened during a scientific experiment in Wuhan — where COVID-19 was first diagnosed and where there are three high-security virology labs, one of which held in its freezers the most comprehensive inventory of sampled bat viruses in the world — isn’t a conspiracy theory. It’s just a theory. It merits attention, I believe, alongside other reasoned attempts to explain the source of our current catastrophe.
Third and finally, just to be on the safe side, we don’t believe in Q.
^ The above statement is not code for supporting Q.
(Or is it, Q-ers?! 🤔🤔🤔🤔🤔 😲)
How does one begin forecasting the origins of Covid-19? Unlike nearly all our previous forecasts, this one pertains to a past event. Instead of predicting the future, we are investigating the past and assigning likelihood probabilities to each potential hypothesis.
The hypotheses we are seriously examining are whether, SARS-CoV-2:
- Evolved in nature and was transmitted to humans from animals -> Pure Zoonotic
- Evolved in nature and was transmitted to humans when researchers attempted to collect virus samples from animals -> Zoonotic Collection
- Was collected from either animals or humans, genetically modified during lab research, and released by accident -> Lab Leak
To analyze these hypotheses, we are following the framework and general evidence established by Rootclaim. Rootclaim is a site “focused on exposing the truth on many issues in public discourse using probabilistic inference.” In other words, Rootclaim picks (mostly) controversial events with unclear hypotheses behind their cause and tries to determine the truth using Bayesian statistics, crowd forecasting, and clear reasoning and evidence. Rootclaim has a limited, yet generally positive track record despite making what would otherwise be thought of as controversial, and incorrect, claims such as:
- 87% chance opposition forces in Syria (Liwa al-Islam) carried out the August 21, 2013 Ghouta chemical attack.
But controversy and a feeling of incorrectness does not mean prohibition. Not only that, but Rootclaim provides a full accounting of their reasoning and calculation, meaning we can use their framework to reach our own conclusions.
By removing the biochemical weapon probability and merging hypotheses 1 and 2 (zoonotic and zoonotic collection), we start with the following base-rate probabilities as determined by Rootclaim’s analysis:
- 99.08% Origin is Zoonotic (77.19% Pure Zoonotic; 21.89% Zoonotic Collection)
- 00.92% Origin is Lab Leak
The next step is to update this base-rate likelihood given the effect collected evidence has on the different hypotheses. Given that we are following much of Rootclaim’s analysis, we will only comment when we diverge from their analysis.
The first piece of evidence offered by Rootclaim is the contagion and mortality of Covid-19 which is unimportant to our forecast since we are not seriously considering the likelihood of a bio-weapon origin. At the same time, we want to expand the first piece of evidence to further include the overall genetic structure of the virus to account for other analysis we find worth including. Although many of the factors identified by Trevor Bedford in late February of 2020 that made a lab-leak incident unlikely have since changed (for instance, research in May by the Chinese CDC found SARS-CoV-2 did not originate in a Wuhan seafood market), Bedford identified the outgrowth of the virus being more indicative of a random-draw event.
As a result, we multiply the Zoonotic outcomes 1.1x to account for this fact and normalize the probabilities, resulting in the new likelihoods:
- 99.16% Zoonotic (77.25%; 21.91%)
- 0.84% Lab Leak
The outbreak location and virus sources near Wuhan were the second and third pieces of evidence identified by Rootclaim. The former reduced the likelihood of Zoonotic origin, while the latter reduced the likelihood of both:
- 96.29% Zoonotic (21.71%; 74.58%)
- 3.71% Lab Leak
The next two pieces of evidence are the most critical to the lab leak theory and center around the chimeric nature of SARS-CoV-2 and a furin cleavage found in the virus.
To make a very complicated amount of reading somewhat simple, SARS-CoV-2 has a chimeric structure, with the virus being 96% similar to RaTG13, however in a particularly important part of the virus it is less than 80% similar while being 99% similar to a coronavirus found in pangolins (both of these viruses were at the Wuhan Institute of Virology). An indication of a chimeric structure does not mean Covid-19 was engineered since chimeric viruses do emerge naturally, however, given the circumstances it is less likely to have done so.
At the same time, we are less confident in Rootclaim’s effect analysis, and feel there is too much confidence. One piece of evidence for our lack of confidence (Rootclaim’s confidence is 90%) is that recently researchers in China found a small-cluster of viruses similar to SARS-CoV-2, with “one of the viruses isolated from a Rhinolophus pusillus bat shared 94.5% of its genome with the pandemic virus.” As a result, we lowered our confidence to 70%, dropping the Rootclaim’s multiplier from 1/10 to ⅓.
The other peculiar aspect of SARS-CoV-2 is its furin cleavage, one of the factors that makes the virus so transmissible and more deadly in animals (but also more complicated; as the same Nature article shows, the removing the furin cleavage makes the virus less transmissible in E6 laboratory cells.) Given some uncertainty injected by recent studies into the furin cleavage in SARS-CoV-2, we lower the confidence in this effect analysis from Rootclaim’s 90% to 80%, thereby lowering the reduction from 1/10 to ⅕.
The effect of these two pieces of evidences is we now have the following probabilities:
- 60.89% Zoonotic (13.73%; 49.42%)
- 39.11% Lab Leak
For the remaining evidence, we kept the evidence and effect analysis done by Rootclaim, leading to the final likelihoods of:
- 64.32% Zoonotic Origin (6.96% Pure Zoonotic; 57.36% Zoonotic Collection)
- 35.68% Lab Leak Origin
However, we wanted to further include three pieces of non-quantitative evidence into our forecast using the 51-49 rule (meaning if we find the evidence in favor of one hypothesis we give it 51% weight, while we give the alternate 49% weight before normalizing).
We then round both numbers down, and gave the accumulated 1% likelihood to all other outcomes including the possibility of a bio-weapon origin, creating the final likelihoods for each hypothesis of:
- Zoonotic Origin (Pure and Collection): 61%
- Lab Leak Origin: 38%
- Other Origin: 1%
Now that we’ve calculated the odds that SARS-CoV-2 originated in a lab—with the nearly certain candidate (>99%) being the Wuhan Institute of Virology—we now calculate the probabilities for the two Metaculus questions above.
Metaculus Forecast: Q1
In truth, there was no need for any of this analysis for the first question...because my god is it busted.
The first question was as follows: “Will it turn out that Covid-19 originated inside a research lab in Hubei?” Makes sense, right?
Well, before we comment on the resolution criteria, let’s pause for a second. Try to think how you would determine whether or not it turned out “Covid-19 originated inside a research lab in Hubei?” Write down your resolution criteria in the comment section if you have confidence yours will be better than the actual one.
Okay, the directly-quoted resolution criteria for this question is:
- If at any time after the date of May 1, 2020 the community prediction is > 97% or < 3%, the question closes. Then, with 90% probability (as called by a quantum RNG), resolves positively or negatively, respectively. (With 10% probability the question is referred to the below committee.)
- Otherwise, on or about the close/resolve date of May 1, 2021, the question will be decided by unanimous vote of a council of three people as to whether the proposition listed above is true, at 50+% credence (i.e. more true than its negation), resolving ambiguous in the case of disagreement. The council of three will be chosen by quantum mechanical random numbers from a list of 12 people that will be composed by the author around the time of 2020-05-01, and held secretly until the time of question resolution.
Probably should’ve posted that comment.
Honestly, what a horrible question, with horrible^2 resolution criteria.
First of all, we did not originally notice the resolution date of this question was May 1, 2021. Why is that absurd? Because the date expects we will know the origins of a pandemic a year and a half after its onset.
But the resolution date is but a scratch. The resolution criteria is where the W I L D stuff is at. How are we to determine if Covid-19 originated inside a research lab in Hubei? According to this question, in two ways you probably never expected:
- Either a whole lot of people come to the same, strong conclusion that Covid-19 did originate (>97% median) or did not originate (<3% median) in a lab [or voted in a way that matched voting with a strong conviction one way or another]; OR
- The Politburo ☭ decides. (Technically the alternative is that 3 of the 12 people the question-poster pre-selected have to unanimously agree or disagree on lab origins, but–ya know–same thing.)
While we strongly feel in this time horizon the true probability is around 1%, there is a secret committee deciding the question. Your fate is never certain in those instances.
Therefore, we predict a 25% chance that by May 1 it will turn out SARS-CoV-2 originated inside a research lab in Hubei.
I mean we only get slightly fewer points this way anyways.
What would make us reconsider?
- If the resolution criteria changed. Assuming the change was not even more absurd, we would lower our forecast to either 1% or 2%
- If the community began approaching 3% or 97%, then we 🚀🚀🚀
Metaculus Forecast: Q2
Thankfully, the second Metaculus question is much much much better: “Credible claim by 2024 that COVID-19 likely originated in a lab?”
Not only is the 2024 resolution date much more reasonable, 4 years after the onset, but the criteria also makes sense. This question resolves positively if, before the end of 2024, at least two of the following public health agencies claim “it is more likely than not” that Covid-19 originated in a Chinese virology laboratory:
- European CDC
- Chinese CDC
- Centre for Health Protection
- Robert Koch Institute
- The National Institute of Infectious Diseases
- Public Health England
- The National Centre for Infectious Diseases
- CDC Korea
- The Public Health Agency of Canada
More specifically, the question requires that at least two of these agencies “broadly state that it is more likely than not that COVID-19 originated from a Chinese virology or biology laboratory, after having been released accidentally or deliberately. Synonyms for probability assessments must be considered by an admin to be broadly consistent with at least a 50% chance. Examples of such synonyms include "probably", "likely", "with high probability" and "almost certainly".”
With this much clearer resolution criteria, we feel comfortable making a real forecast. Using our previously analyzed likelihood of 38%, we then cut this number in half for two reasons:
- Time (less significant)
- Politics (more significant)
Therefore, we forecast there is a 19% likelihood that by 2024 at least two of the above public health agencies will claim SARS-CoV-2 likely originated in a Chinese virology lab.
What would make us reconsider?
- Any change in our understanding of the facts underpinning the different steps in analysis, with the main factors being into the mosaic structure of the virus and the importance and structure of the furin cleavage.
- A change in global geopolitical relations with China, with a trend towards positive meaning the odds go down and a trend against meaning the odds go up (as political friction to positive resolution reduces)
Before getting to our final prediction, here are a few Metaculus community updates!
Community Guidelines: For the first time, Metaculus now has a codified set of community guidelines. While this seems fairly humdrum on the surface, we found this to be an important step for the growing platform. As any online social community hits certain thresholds with respect to scale and notoriety, the introduction of things like community guidelines become necessary. So we see this as an indication that the Metaculus community is healthy and growing which is exciting for the forecasting community writ-large.
Moderator Elections: What good are community guidelines if there isn’t someone to arbitrate on them? Metaculus is in the midst of holding an election process for community moderators. This, like the creation of the community guidelines, is a positive indication of Metaculus’ health, and will ostensibly only help the platform improve moving forward, which we at Global Guessing are all for! If you are interested in applying to become a moderator, or just want to see if you’re eligible, check out the criteria and description here.
Will There Be Peace in Yemen in 2021?
For our second and final forecast in this week’s Metaculus Monday, we are predicting whether there will be a 30-day period in 2021 when either a cease-fire or peace agreement is in place in over 90% of the territory in Yemen. The community median is 52%, while the mean is 50%. So far there have only been 15 predictions: One of our earliest yet!
But before we get into the forecast, it’s important to have an understanding of the geopolitical events which have led to the current moment in Yemen.
Background: Yemen Civil War
In 2011 the Arab Spring led to regime changes in several Middle Eastern countries, Yemen included. Then President Ali Abdullah Saleh was ousted on claims of corruption and rising unemployment, and his Vice President Abdrabbuh Mansur Hadi came to power. This transition of power was not successful, however, and has played a key role in Yemen’s current crises.
Under Hadi, unemployment continued to rise unmitigated. Hadi also led Yemen’s move in 2014 to be divided into six discrete federal regions. While some viewed this as a welcome change to the status quo of violence and poverty in Yemen, potentially offering a more responsive government option, groups like the Houthis were outraged. They viewed this division as an attack on their influence in the country.
The Houthis, a Shia rebel group who existed in the pre-Hadi era, ultimately partnered with former President Saleh to gain greater political influence and troops. The Houthis went on to capture Yemen’s capital Sana’a, and marched on to threaten Hadi’s rule in the coastal city of Aden. It was this move toward Aden which saw Hadi flee to Saudi Arabia (where he currently resides) and in turn which rang alarm bells for Saudi Arabia that something must be done.
And so in 2015 violence in Yemen escalated, with Saudi Arabia leading a massive coalition of countries to fight off the Houthi rebels, whose Shia identity made them a likely ally of Iran, one of the few predominantly-Shia countries in the Middle East. Carpet bombing ensued, with thousands of Yemeni civilians being killed, and even more leaving the country catalyzing what is arguably the worst ongoing humanitarian crisis in the world.
To make matters worse, violent non-state actors have leveraged the chaos in Yemen to gain footholds on the borders, making conflict in the region more complicated. Additionally, some members of the Saudi-led coalition in Yemen have begun to work towards their own goals in the country as the conflict has gone on, weakening the efficacy of the group. So now we are at a point where Yemen has become the violent playground for several countries to assert their interests all while civilians continue to starve and be killed.
To start, we tried to identify a base rate for ceasefires agreements since the civil war began in 2014. In the war’s six-year span, there have been two instances of ceasefires. One in 2015 and another in 2020, leading to a 33.3% base rate. However, neither of these ceasefires lasted more than a week. As a result, we cut the base-rate for a 30-day ceasefire by 10x, resulting in a 3.33% base-rate for a 30-day ceasefire.
Next, to get outside of the binary of ‘continued fighting’ and ‘ceasefire,’ we attached a 5% possibility of a peace deal or Houthi victory to take place. We then subtracted the peace and ceasefire possibilities from 100% to get the continued fighting base-rate. The resulting base rates were:
- 91.67% Likelihood of ‘Continued Fighting’
- 3.33% Likelihood of ‘Ceasefire’
- 5.00% Likelihood of ‘Peace Deal or Houthi Victory’
We then took the same approach as the Rootclaim analysis, using our own evidence and key factors:
- Biden ending support for the Saudi-led coalition
- The 2020 prisoner exchange
- Covid-19 pandemic / vaccines
- Recent Envoy developments / Other US Actions
The recent news that Biden is ending US support for the Saudi-led coalition in Yemen is important for two reasons. On one hand, it has led to clear demonstrations of power by the Houthis ostensibly raising the likelihood of a ‘Houthi Victory’ or peace deal struck due to their strong position. It also, however, indicates potential ‘Continued Fighting’ with the violence becoming protracted on the back of a Houthi resurgence thanks to less United States influence. We have an 80% confidence that US disengagement from the conflict made the likelihoods for continued fighting and peace deal / Houthi victory far likelier, giving them a 5x multiple and creating the new probabilities of:
- 94.18% Likelihood of ‘Continued Fighting’
- 0.68% Likelihood of ‘Ceasefire’
- 5.14% Likelihood of ‘Peace Deal or Houthi Victory’
Our second factor is the recent prisoner exchange and UN involvement in the crisis. Last fall, in October, the largest prisoner swap in the six-year conflict took place. Both rebel fighters and pro-Yemeni government affiliates were released to their respective sides, with another exchange already in the works at the time of the swap, showing massive amounts of collaboration between the two warring sides.
The success of the swap increases the chances that a ‘Peace Deal’ will occur, or also potentially a ‘Ceasefire’. To calculate the impact of the prisoner swap on our outcomes, we gave it a double 51-49 weighing, in favor of both a peace deal / Houthi victory and a Ceasefire, leading to the following values:
- 93.72% Keep Fighting
- 0.74% Ceasefire
- 5.54% Peace Deal / Houthi Victory
The next factor is the one most in favor of a ceasefire taking place this year: Covid-19. Covid-19 served as the backdrop for last year’s ceasefire. We also believe Covid-19 could potentially serve as a peace-generating factor as opposed to a peace-limiting factor. Getting past the pandemic is a desired goal for many groups that span ethnic and political bounds. As Yemen just received vaccines in March, this issue is exceedingly relevant for the Yemeni people of various creeds making cooperation to achieve herd immunity much more plausible. Given this analysis, we gave a 0.8x multiple on continued fighting, a 5x multiple on a ceasefire, and a 1.1x multiple on a peace deal / Houthi victory. The resulting percentages were:
- 88.46% Likelihood of ‘Continued Fighting’
- 4.35% Likelihood of ‘Ceasefire’
- 7.19% Likelihood of ‘Peace Deal or Houthi Victory’
Finally we found that both the recent envoy news–that the US’ proposed ceasefire was rejected by the Houthi’s–and U.S. Secretary of State Blinken’s assertion that there can be no military solution to the conflict in Yemen signals a high likelihood of continued violence in the region. The Houthis’ rejection following US-withdrawal from the conflict could indicate a confidence in their strong position further increasing the odds of continued fighting. Given our uncertainty of the final effect these recent developments will have, we give a 51% weight to continued fighting, 49% to ceasefire, and 50% to peace deal / Houthi victory. Given this, we came to the following final values:
- 88.74% Likelihood of ‘Continued Fighting’
- 4.19% Likelihood of ‘Ceasefire’
- 7.07% Likelihood of ‘Peace Deal or Houthi Victory’
Taken together, the four factors’ influence on the three potential outcomes of this conflict by 2021 given the question criteria yields a 11.26% likelihood that there is peace in Yemen this year.
What would make us reconsider?
There are also a number of factors that have the potential to affect this outcome which we will be watching over the next 9 months, and more pertinently over the next four months before this question closes on Metaculus.
We have broken those factors into two categories. The first, the endogenous factors, are those which are already baked into our initial prediction. These will be changes of values already present in our model, and so if any of them do sway the question’s resolution, we should be able to understand that outcome with our analysis. The second group are the exogenous factors–those factors which exist outside of our model’s considerations but which still have a significant potential impact on the outcome of the question. While the relevant importance of each factor varies, each is significant enough to affect the outcome and should be considered when forecasting this question. They are as follows:
Endogenous Factors (Part of original prediction)
- COVID Vaccine News in Yemen
- Progress of the UN mission
- Whether Houth’s continue to dominate on the battleground
- Saudi Arabia’s position on the blockade
- Outcome of Iranian election (note: Iran has influence, not control over Houthis in Yemen)
- JCPOA Talks (quasi-endogenous)
- Saudi Arabia’s reconsideration of spending in Yemen (They are spending billions while Iran is spending only 10s of millions)
As this question does not close until mid-summer, we will be watching how both our central considerations, and our endogenous and exogenous factors behave over the coming months. Should there be any big news in any one factor or consideration we plan to update this forecast accordingly. Additionally there are only 15 predictions on this Metaculus question at the time that this article is being published, so we hope that this article encourages more people to make predictions on that question.
Global Guessing Loves Geopolitics!
Here at Global Guessing we love geopolitics. That’s made pretty clear from our last forecast of this Metaculus Mondays volume which takes into account a massive about of knowledge regarding bilateral relations and historical ties between countries. It’s also very timely and important to know about as it is affecting hundred of thousands of people each day.
If you want to hear more of our thoughts on the current geopolitical landscape check out our most recent Global Guessing Weekly Podcast episode. We spoke with Ross C. of AR Global Security all about China, Russia, the evolving form of warfare, and more interesting topics.
And subscribe to his YouTube channel for more interesting content and further collaborations!