Coronavirus: The limits of forecasting

A lot of people think a model and a forecast are the same thing. They aren’t. They are very different and knowing when you can forecast, and when you can only model is a critical investing skill. Coronavirus forecasting is a testament to the difference – many elements can be modelled, very few can be forecast. 

If we take coronavirus deaths/hospitalisations as an example, the range of outcomes from very minor changes in assumptions is so extreme that forecasts are almost meaningless. However, virus models themselves are not useless as they do let us forecast other outcomes.   

Coronavirus deaths/hospitalisations forecasting 

I’m going to use the online model at http://gabgoh.github.io/COVID/index.html to illustrate the issue. I’ll use this with a population of a little over 300m to show the vast difference in outcomes for a country the size of the United States from simple changes in assumptions. 

There are lots of assumptions, but I’m going to focus on the three main ones: 

  • R0 – how many people (on average) will one person spread the virus to in an unconstrained society. Case 1: using the model default of 2.2 people. Case 2: A more aggressive 2.8 people.
  • Intervention delay – how long after the first case are quarantines/lockdowns implemented. Case 1: using the model default of 100 days. Case 2: Assume 5 more days before intervention.
  • Intervention effectiveness – how are social interactions reduced by because of the quarantines/lockdowns. Case 1: using the model default of a 66.6% reduction. Case 2: Assume quarantines are slightly less effective, reducing transmission by 60%.

Here is the output for case 1: 86,721 peak hospitalisations and 20,773 deaths.

Now, stop for a second. How many more hospitalisations/deaths do you think case two will create? 10% more? 100% more? 1,000% more?

Scroll down to see how right you are.

 

 

 


If you guessed a 32,500% increase in peak hospitalisations and 23,000% for deaths, then you take the prize. 

Keep in mind the above assumptions are not known to any precision. Last month estimates for R0 were around 2.5, the US Centre for disease control just released a paper suggesting it is 5.7. So the difference between assuming 2.2 and 2.8 isn’t much. A five-day intervention delay is not unheard of. The difference between a 66.6% effective quarantine and a 60% effective is probably unmeasurable, but again a relatively trivial difference in assumptions.

Net result:

We can model deaths and hospitalisations. We can’t forecast them in any meaningful way.

Aggregating or comparing COVID-19 case counts is futile 

I find the focus on global case counts or comparisons also irrelevant. There are four main gaps between reported cases and actual cases:

  1. Insufficient testing: Countries that do not have the systems or capacity to test properly
  2. Yet to be reported cases: People who have the virus but are either: yet to show symptoms, yet to be tested, or had a test that showed a false negative   
  3. Asymptomatic cases: Estimates are that around half of all people who get the virus will never show symptoms
  4. Deliberately under-reported: Some countries look to be deliberately under-reporting cases 

The net result is the number of global cases is likely to be significantly different from reported statistics. Some estimates are that global infections are as much as 500% to 1,000% higher than reported. Which basically makes any aggregated reported numbers useless. 

We started our analysis in January by excluding Wuhan data, then all Chinese data. Then Iranian data looked suspect. Then Italy changed definitions to under-report cases. After that, we gave up on aggregates and focussed on individual countries. At least that way, changes would be based on a relatively consistent methodology.

Ideally, testing would be randomised, significant in size and from a trustworthy source. Iceland, Norway, Australia, Germany and South Korea (in green below) rate the best. Switzerland, Italy, Japan, US and Netherlands (in red below) rank poorly. Dashed lines are countries with widespread facemask use.

Reported coronavirus cases percent population

However, the above chart is misleading. For example, Iceland rates at the top, but not because it has the most number of severe cases. Instead, Iceland ranks highly as it has been doing far more testing – including random sampling. 

Hospitalisations are the better statistic 

We are focussed on the number of critical or severe cases. While the definition of these will vary between countries, they are likely to represent hospitalisations which we believe are a better indication of the current severity. Note: for these statistics updated daily, see here

Serious/Critical coronavirus cases per 100,000 population

Note also these are a measure of the current active numbers. Total case numbers can only increase, but these numbers can decrease as sick people either die or are cured and are removed from the count. So, it represents a better indication of the stress on the hospital system.

Experimenting with trade-offs 

We note the economic/humanitarian trade-off:

  1. Humanitarian. The bigger the shutdowns, the greater the preventative measures, the fewer people will die.
  2. Economic. The bigger the shutdowns, the greater the preventative measures, the more significant the economic impact will be.

We are of the view that governments around the world are experimenting, trying varying levels of shutdowns to get hospitalisations down to a level that does not overwhelm the healthcare system.

As Tomas Pueyo eloquently puts it, the solution is in two forms, the Hammer and the Dance:

We are making the assumption that governments will eventually come to the same conclusion.

The Hammer is needed to get cases down to an acceptable level. Then the Dance begins where governments will need to adding capacity and mitigation strategies while gradually opening up.

South Korea has probably been the best example.

What level of hospitalisation can a country withstand? 

So what level of hospitalisation is acceptable during the dance?

We are assuming that an acceptable level of hospitalisations is around 25% of existing beds or intensive care units. This accounts for regular patients and case clustering. So, the target needs to be considerably lower than the capacity.  

We estimate:

  • it takes about two weeks for patients to die.
  • it takes four weeks for patients hospitalised to be released.
  • about a quarter of hospitalisations need to be admitted to intensive care.

So new patients will need to be below 4% of capacity each day. As Intensive Care Unit beds are created and treatments become more effective this will improve.

There are two ways to look at whether a country needs the hammer or whether it can advance to the dance.

Hammer/Dance 1: Is the hospital system is overwhelmed?

We have used the capacity available prior to the coronavirus outbreak to assess this measure. While many countries are adding temporary capacity to deal with patients, these are emergency measures. If we are looking at an extended period before vaccines are available then coronavirus patients should take up only a relatively low proportion of a country’s intensive care beds: 

Prior ICU Capacity vs Estimated Critical Coronavirus Cases

Hammer/Dance 2: How fast are cases and deaths growing?

The second is the speed of growth of cases and deaths. If cases are quickly growing then the hospital system will be overwhelmed, even if it is currently coping.

We use a target of 20 days to double cases/deaths. This is roughly the time for an Intensive Care Unit bed to become free either through a cure or death. For now, those levels are a reasonable target:

Number of Days to Double: Covid-19 Cases vs Deaths

Is Winter an issue?

At the moment our base case is coronavirus is worse in winter but delayed or poor responses in tropical or summer countries can offset most of the benefit.  

We split cases into: 

  • Northern Hemisphere Countries that are now in spring and are getting warmer 
  • Tropical countries: Countries near the equator where temperatures are relatively high all year
  • Southern Hemisphere Countries now in autumn and are getting colder

In aggregate numbers the differences are still stark:

Coronavirus cases Summer vs Winter

The COVID-19 statistics in these charts are based on where the case was caught. For example, as at 01-Mar two-thirds of the cases in Thailand were caught in China or Iran but diagnosed in Thailand. So, in that example, one-third of cases are allocated to Equatorial and two-thirds to Winter. 

There are suggestions that UV-B radiation and vitamin D played a role in reducing deaths in the Spanish Flu pandemic. Other studies suggest humidity greatly reduces the aerosol transmission of viruses, but some suggest humidity increases the surface transmission. It is unknown how these affect COVID-19.

The below charts are the pessimistic take on the summer/winter divide. Cases are spreading just as quickly and tropical countries are sustaining significant outbreaks. Also, a number of large population tropical countries have poorer testing/reporting: 

Covid-19 cases Summer vs Winter log scale

What can we forecast?

Viruses are complex systems, but virus models do have what we call emergent properties. The shape of the initial two curves above is an example – the shape is similar even if the scale and the timeline aren’t. And these emergent properties are the factors that can be used.

The emergent properties from the models and data that we think important:

  • Lockdowns work: If they don’t slow the virus enough then stricter ones will. However, the consequence of not being harsh enough or delaying makes a massive difference to the outcome. i.e. calibrating the hammer
  • Lockdowns = curve flattening:  Lockdowns take about 10 days or so to take effect.  
  • Summer beats Winter: But complacency overwhelms any benefit
  • Facemasks work: Countries with broad use doing better than those without. 
  • Opening up needs to be measured: too fast creates more cases. i.e. calibrating the dance
  • It is all about the hospital system: Not overwhelming the hospital system is the key measure

————————————————-

Damien Klassen is Head of Investments at the Macrobusiness Fund, which is powered by Nucleus Wealth.

Follow @DamienKlassen on Twitter or Linked In

The information on this blog contains general information and does not take into account your personal objectives, financial situation or needs. Past performance is not an indication of future performance. Damien Klassen is an authorised representative of Nucleus Wealth Management, a Corporate Authorised Representative of Nucleus Advice Pty Ltd – AFSL 515796.

 

Follow me
Latest posts by Damien Klassen (see all)

Comments

  1. Those Cruise boats are like little micro nations.
    They would probably give the best “true” “trustworthy” numbers.

    Yes the Dance is the way to go.
    I suspect they are holding off lifting restrictions as not enough PPE etc to meet demand.
    And I would like the option of having proper masks avaiable in our household if someone has to go out shopping for example.
    I imagine due to the world wide demand/shortage of PPE then we are going to be delayed in going to the Dance.

    • cruise ships are not so good for stats because they have different demographics (much older than general population)

      the other important question is why do we still don’t have enough PPE (masks etc). We live in a world where apple and samsung can produce 1 billion smartphones in few months (some of the most complicated devices out there) yet three months after the outbreak we still cannot produce billions of masks?

  2. Firstly and most importantly: “the hammer and the dance” is not “flattening the curve”, there are few major differences:
    the dance:
    – doesn’t keep number of active cases up to the capacity of the health care system
    – requires much stricter lockdown
    – makes less people sick/dead
    – doesn’t create immunity within the population – has no end game and needs to continue until vaccination does create immunity (or forever or until it gets converted back to flattening the curve)

    various countries use various approaches, some Asian countries like Singapore and South Korea managed to implement successful “the hammer and the dance” but some other like Germany, Switzerland and soon France managed to get flattening right, they keep high number of active cases high but stable while heath care functions and population gain immunity.
    These two approaches lead to two different results:
    Korea, Singapore, … will have to keep current level of lockdowns until vaccine gets found (that could be as little as 9 mnths or as long as forever)
    Germany and Switzerland will if they keep going get into a herd immunity within 3 or 6 or 12 months depending on % of asymptomatic and mild cases. Some randomized tests in Germany indicate there could already be as many as 14% of population with immunity. Flattening the curve countries will see more deaths but lower cost due to weaker measures

    also there are countries that failed to implement measures or purposely let virus wild and they may see heard immunity much sooner at the expense of more deaths.
    assuming that the virus is more or less equally deadly around the world (adjusted for health care issues like lack of it or overrun), it can be said that some countries like Italy and Spain may be well ahead into immunity but with higher cost – not so much higher number of dead because epidemic will end there sooner (both Italy and Spain had healthcare overruns because of local hotspots not because of high total number of hospitalized people – hospitals in some other big cities iwere never even close to capacity: for example while Lombardy had 10k deaths, other big cities Piedmont (Turin) not so far away had much less – 1k deaths while Campania (Napoli) and Rome had 250 deaths and no hospital overrun). If number of cases in Italy was more equally spread there would have been not so many problems with health care system. Same is in Spain and USA. USA healthcare system can easily handle these numbers in all of the states but NYC area

    bottom line is that only time will tell us what approach was better. many factors can affect this, like success in vaccination, potential discovery of a cure, some virus changes (could be both ways although more likely to make it weaker), …
    based on current state of information, Germany is doing the best job, running a successful end game strategy with not so high costs

    • Fingers crossed that Oz picks the best possible path out of our current position.
      Thing is everyone may have a different interpretation of ” best”.

    • desmodromicMEMBER

      Dr X, is there another example of a highly infectious pathogen with similar case fatality rate to COVID for which a human population achieved ‘herd immunity’ without a vaccine?

      • How about the viruses most similar to a novel coronavirus? Common cold viruses?
        Kids without immunity don’t die from common cold for the same reason they don’t from the new coronavirus, elderly don’t die because they have immunity?

        • desmodromicMEMBER

          Using your example, the pathogen is either common but not deadly (cold) or deadly and not common (SARS and MERS). COVID sits somewhere between and is more destructive as a result . ‘Herd immunity’ is usually associated with vaccination not infection and a return to globalised society is unlikely without a vaccine.

          • Reality is that common cold maybe identical to new coronavirus (deadly for young without immunity and benign for young without protection) but since everyone already got it when young and developed immunity we don’t see people dying from it.
            I’m talking about immunity (in the normal sense having immune system able to recognise and fight more effectively) not herd immunity used to suppress spread of the virus

  3. This guy reckons we are all playing for time………..we badly need an accurate antibody test as a first step, otherwise we are just guessing

    https://medium.com/@wpegden/a-call-to-honesty-in-pandemic-modeling-5c156686a64b

    Food production starting to get hit now everywhere……..seems like fruit and vegies and grains will be available usually but meat protein is going to take a hit, this is no surprise to anyone who has been on a slaughterhouse floor……..go long tinned meats.

    • Beans! The poor man’s meat.

      We’re turning Mexicanese …

      Better get some chooks…

  4. Great stuff Damien. I like to think this the sort of analysis the govt is getting – not the superficial rubbish the MSM feeds us.

    The key has always been to figure out when the hospital system is overwhelmed. If we hadn’t flattened the curve when we did we would have been at that point a fortnight ago. We’ve done well. But as you say there’s still a long dance ahead

    And thank you for reporting numbers in per capita terms…you never see this

    • We did not flatten the curve to be able to deal with the sick (we never used 5% of ICU capacity), we killed the curve and now we don’t know what to do? To restart the epidemic by letting few thousands get infected daily to keep ICUs at capacity while building immunity ? To keep hard lockout until vaccine?

  5. pyjamasbeforechristMEMBER

    Treatment trial underway worth watching

    https://clinicaltrials.gov/ct2/show/NCT04332991

    And here is why

    Explaining how Zinc blocks RdRP so it can’t creating more virus RNA if (it is a big if) you can get the Zinc into your cells Cytoplasm
    https://youtu.be/Eeh054-Hx1U

    Explaining how the existing drug Chloroquine might be able to get Zinc into your cells Cytoplasm
    https://youtu.be/U7F1cnWup9M

    Explaining how Hydroxychloroquine has actually been used against Covid-19 with positive results.
    https://youtu.be/vE4_LsftNKM

  6. david collyerMEMBER

    I am bemused by arguments around whether isolation rules should be tighter or looser or strategies for ‘herd immunity’ be pursued.

    This is an entirely second order matter with more opinions on offer than grains of sand on the beach. We are better served examining what government Is actually doing – which levers and why – and the social and economic implications of these actions. It is most amusing to watch this deeply conservative government abandon so many dearly held principles with indecent speed.

    I wonder how government will restart the economy after we bury our dead. What will our economic landscape look like? How to repay the extra $200 billion and counting in government debt, plus the ballooning mortgage debt from repayment moratoria.

    And what will land trade for later? Louis Christopher is the only one to have a decent stab at future prices, predicting -30%.

    With rents unknown, population unknown, animal spirits unknown, inflation/deflation unknown, commodity prices unknown – is there any aspect of human activity that CAN be reliably forecast beyond surges in both pregnancy and divorce?

    • Greed, or ‘Getting Ahead” is the go to animal spirits button. They just have to reduce the stronger Fear button to acceptable levels. Fear of losing their own seats is the biggest fear of all, or we’d be Swamped already.

  7. Thank you Damien, very informative.

    Do you have any estimated ranges for which you think the asymptomatic cases are currently spread in the population for Australia?

    The reason i ask is that the US seems to have a very high rate of Covid for their tested population already, is there any modelling youve seen around a scenario where herd immunity is achieved after a smoothed curve of death over time given actual infection seems to be way higher than deaths and ICU visits as you point out. So the ‘dance’ stage stops or reduces when covid goes through enough of the population to be restricted to periodic outbreaks

  8. fantastic article Damien. Thank you and goes a long way to explain my gut issues with our current “flattening” (refer comment on grattan article in MB today). Thank you!!