Simulation Trials vs. Scenarios

by Sam L. Savage

Justin Schell is part of a team bringing the discipline of probability management to Highmark Health, an $18 billion, Pittsburgh-based integrated healthcare firm. So far he is getting a good reception from a number of quarters in his organization. However, as he has introduced simulation trials to management, he has encountered confusion with the concept of scenarios. Justin invited me and Dr. Sarah Lukens, a Data Scientist at GE Digital, to discuss this with him and record our conversation in a video, which he has just posted here.

From my perspective, if simulating a business plan before investing in it is like shaking a ladder before climbing on it, then scenarios correspond to where you will unexpectedly find your ladder when the climbing begins--will it be beside your house, over broken beer bottles, next to a shark tank, etc.?

Although scenario analysis was made famous by Royal Dutch Shell’s fortuitous preparedness for the collapse of the former Soviet Union, Reidar Bratvold, a Professor of Investment & Decision Analysis at the University of Stavanger in Norway, points out a potential big problem with the approach. By focusing on a few, causal stories, it diverts attention from “a broader, more systematic representation of the decision situation.” In this way, there is the potential that it “grossly overestimates the probability of the scenarios that come to mind and underestimates long-term probabilities of events occurring one way or another.” Bratvold also compares scenario analysis to risk matrices, which many people consider worse than useless by providing a palliative that lulls one into the sense that they have done risk management.

Both scenario analysis and risk matrices are often used in an attempt to do probabilistic analysis without using probability. That is like trying to learn how to swim without getting wet. Understanding probability, or what I call the arithmetic of uncertainty, has to come first. Then when you conjure up a new scenario for consideration, you can address whether it is more or less likely than an asteroid strike. I think of simulation and scenario analysis as dual to each other. You will fall into the traps described by Bratvold if you don’t understand the probabilities involved, and you may end up simulating the wrong things if you haven’t explored the parallel universes in which you might find yourself.

Through simulation analysis, new scenarios may suggest themselves and vice versa.

© Copyright 2020, Sam L. Savage

Scenario Analysis on Steroids

New 4.0 Enterprise SIPmath Modeler Tools

Herman Kahn

Scenario Analysis

Scenario analysis was invented by futurist Herman Kahn in the 1950s at the Rand Corporation. He was a friend of my dad’s and I remember him vividly from early childhood as energetic, humorous, and rotund. Perhaps not traits one would expect in the author of On Thermonuclear War, his controversial 1960 treatise of war in the nuclear age, which introduced the Doomsday Machine. In fact, Kahn is one of the characters upon whom Dr. Strangelove (of the movie) is based.

Scenario analysis involves forecasting major political and economic shifts through a few self-consistent narratives around potential futures. One does not specify probabilities of these scenarios up front but uses them to guide analysis into the unknown.

Scenario analysis has been widely used at Royal Dutch Shell and has been credited with preparing the company for the collapse of the Soviet Union. Here is a short docudrama of how it occurred.

“What would happen if the Soviet Union Collapsed?” … “Get out of town.” … “But what if it did?” … “That’s ridiculous!” … “But what if it did?” . . . . . . . . . . . . . “Maybe we’d better think this through.”

I am all for the out-of-the-box thinking that scenario analysis encourages. But if each of your handful of scenarios is rooted in the Flaw of Averages, where are you? Among other new features, the latest SIPmath Tools make it easy to combine interactive Monte Carlo with scenario analysis for the best of both worlds.

An Application – Climate Smart Agriculture

We have been assisting a team of environmental scientists at World Agroforestry (ICRAF) in exploring sets of climate smart agriculture projects in Africa, which will be the subject of a future webinar. Because all of these projects coexist in the same uncertain environment, there are strong portfolio effects, which can be modeled well with coherent SIP libraries. But beyond the sorts of uncertainties that are amenable to Monte Carlo simulation, there are potential world scenarios involving political upheaval, carbon pricing, etc., for which it is difficult to estimate probabilities. Therefore, we added the capability to the SIPmath Modeler Tools to run the same simulation through multiple experiments. You can then quickly scroll through either different portfolios of projects in one world scenario, or the same portfolio in multiple scenarios as shown in the graphics below.

Changing Portfolios

Changing Scenarios

Free Webinars

Brian Putt, our Chair of Energy Applications, and I will be offering a series of ongoing free webinars on the new SIPmath Tools. For a limited time, attendees will have the option to purchase the Enterprise Tools at a 30% discount, $150 off the regular price of $500. The first three webinars are listed below.

Introduction to the 4.0 Tools

This webinar will start with the basics of using either the Free or Enterprise versions of the SIPmath Modeler Tools for Excel. We will then briefly describe the exciting new features of the 4th generation tools below.

• Advanced HDR generator from Hubbard Decision Research

• Scatterplots of input and output cells

• Multi-scenario simulation and multi-scenario libraries

• Save and retrieve PMTable sheets for advanced analysis

Scenario Analysis on Steroids

This webinar will show how to create scenarios based on several variables such as discount rates, price levels, political upheaval, etc., which may easily be run through a single simulation model. The new Repeated Save command, coupled to Danny O’Neil’s “Enigma” formulas, automatically creates SIP Libraries containing multiple scenarios. These may be accessed by other models that can in turn filter the results by scenario. Topics include:

• Using the HDR Generator to coordinate models

• The “Enigma” formulas for performing experiments with arbitrary numbers of variables

• The Repeated Save command

• Filtering the results on Input and Output

More Power to the PMTable

The PMTable sheet is the heart of SIPmath in Excel, as it is the location of the Data Table that allows the simulation to run in native Excel. This webinar will show how to use the new “Save PMTable” command, which lets you save multiple versions of your analysis. For example, you may perform a multiple output simulation, save the resulting PMTable, then run a single output multiple experiment, save that PMTable, then return to the original. This also enables complex analysis techniques to include: 

• Storing a base or reference case for comparison

• Sensitivity analysis

• Tornado diagrams

• Critical path identification

© Copyright 2020 Sam L. Savage

COVID-19: The Solution is Obvious

Regardless of Your Political Position

by Sam L. Savage

The entire world is facing tradeoffs between public health and economics. But at a high level it is obvious what to do. We must choose between rational tradeoffs that balance these two objectives (the O’s in the graph above) or decisions that could be improved in both dimensions (X’s). There are many more X’s than O’s, and the O’s are hard to find. But we have the analytical technology to find them and should start now.

When we do find the O’s, some will favor healthcare outcomes, some economic outcomes, and some will be in between. So how should we choose among them? The way our country has traditionally made decisions that impact various stakeholders differentially, with democracy. Regardless of your politics, you want an O, not an X, and it’s nice that we can all agree on something.

So, what is the technology that can help us find the O’s? It is called stochastic optimization, and it has been used in the financial and insurance industries for decades. But how can you optimize when everything is so uncertain?  The word stochastic means explicitly modeling the uncertainty, as opposed to rolling it into a single average number as in the Flaw of Averages.

Modern Portfolio Theory

In the early 1950’s, future Nobel Prize winner Harry Markowitz was writing his doctoral dissertation on investing at the University of Chicago’s Department of Economics. The academic literature at the time prescribed maximizing average return. But Harry realized that this would have you investing all your money in the single hottest stock in the market. This flies in the face of not putting all your eggs in one basket. So, he explicitly added a new dimension to the investment problem: risk, measured as the uncertainty in return as shown below. Every investment is a point on this graph, and Harry calculated what he called the “Efficient Frontier,” an optimal risk/return tradeoff curve, arcing up from the origin.

This was the first stochastic optimization of which I am aware.

No rational investor would choose an investment to the right of the curve because going straight left to the curve would yield an investment with lower risk at the same return. Going straight up to the curve would yield an investment with more return at the same risk. So, an investment to the right of the curve is just plain nuts. Because the curve was found through optimization in the first place, nothing can exist to its left. And people first detected that Bernie Madoff was a fraud because he was promising the impossible on this graph.

A rational investor might pick any point on the curve depending on their risk attitude as shown. Harry’s 1952 paper on Portfolio Selection led to Modern Portfolio Theory (MPT), which revolutionized Wall Street, led to other stochastic optimization methods,  and ultimately garnered him a Nobel Prize in Economics in 1990.

SIP Libraries

In 2006, I helped Royal Dutch Shell apply MPT to finding efficient frontiers of risky exploration projects. A small prototype model (shown below) has risk on the horizontal axis and expected return on the vertical as in Harry’s original approach. It is available for download here. 

The Shell portfolio model was assembled from smaller models of the individual exploration projects using the concept of the SIP (Stochastic Information Packet). This is a data structure that represents uncertainties as auditable arrays of Monte Carlo trials and metadata. The project SIPs were interactively aggregated into portfolios in Excel, allowing managers just two steps below the CEO to add or remove projects in real time and see the resulting risk/return tradeoffs. The idea of SIP libraries, which had its foundations in the fields of financial engineering and insurance, has now been democratized by 501(c)(3) nonprofit ProbabilityManagement.org, of which Harry Markowitz and I were founding board members in 2013.

COVID-19

Meanwhile, back in the pandemic, there is so much uncertainty about the progression of the contagion, the effects of the disease itself, and human behavior in the face of it all, that the Flaw of Averages abounds.

Again, in theory, stochastic optimization can be applied to this problem, as we applied it at Shell. But due to the size and complexity, a single model would collapse under its own weight before producing useful results. In fact, I am not sure a single team of modelers could do it.

So, our nonprofit has begun experimenting with an approach that would allow teams in diverse disciplines to collaborate on this problem by decomposing it into manageable chunks. Models of contagion, government policy, and economics created separately in such common environments as Excel, R, and Python could be snapped together like Lego blocks using common SIP libraries. We have been working with colleagues at Kaiser Permanente and other healthcare organizations on this project and are actively seeking other potential partners.  

To leave you with something tangible, the blue curves in the graph display what we call the sample paths of a contagion model, with one for each of hundreds or thousands of simulated uncertain outcomes. Taken together, they can help us find the O’s. The red curve is the simulated “average” pandemic, which leads to an X.

© Copyright 2020 Sam L. Savage

Balancing Broomsticks

Coming Out of Coronavirus Captivity

by Sam L. Savage

We have met the enemy and he is us.

The symptoms of COVID-19 range from not even knowing you have it to a death sentence, depending on the patient. Writing on this disparity in the Atlantic, Dr. James Hamblin says that COVID-19 is “proving to be a disease of uncertainty.” He quotes Dr. Robert Murphy, an infectious disease specialist at Northwestern, who was in the trenches of the early HIV epidemic.

“As Murphy puts it, when doctors see this sort of variation in disease severity, ‘that’s not the virus; that’s the host.’” Apparently the virus can make some people’s immune systems have a panic attack called a cytokine storm.  This can be brought on by a number of diseases and other conditions, but because the body is fighting itself, it’s tricky for medicine to know whose side to take, and it is often fatal.

Graphic from The Conversation

Of course, medical science has made huge strides since 1918 (the year of the last big pandemic), or since 2018 for that matter, which we hope will help with COVID-19. In a recent article in the Economist on Learning to fight the next pandemic, Bill Gates points to recent “giant leaps in vaccinology.” In particular, he cites mRNA vaccines that teach your body to create its own immunity, rather than injecting antigens for your immune system to tussle with. He identifies two other relevant areas of medical advances: diagnostic testing, which is our ultimate gauge of pandemic control, and antiviral drugs, which will reduce the impact of contracting the disease.

Update

These uncertain developments may take months or years, but there are also uncertainties being resolved day by day, that shed light on the pandemic. On March 26, 2020, which seems like a lifetime ago, I posed three questions about COVID-19 and wondered if they would be resolved in upcoming weeks. My questions then, and what we have learned about them in the past month, appear below.

1. What percentage of the total population has been infected?
   
   As of my March blog, I had only seen statistics from sick people (a biased sample, which underestimates the total). What do we want today’s total infected to be? 100%, of course. If we all had the virus, we could just go back to our work and play. Furthermore, it would imply a lethality rate comparable to infected hangnails. But 100% of us have not been infected. The good news, however, is that recent studies show that the percentage is perhaps 50 times larger than indicated by the previous studies on symptomatic patients. The New England Journal of Medicine reported on April 13 that of 215 women who delivered babies in two New York hospitals between March 22 and April 4, 15% tested positive, and over 80% of those were asymptomatic. The bad news is that 15% is way below herd immunity levels. But these results, if they are applicable to the general population, show a vast reduction in the effective lethality of the disease.
   

2. How badly will our healthcare system be overwhelmed?
   
   Back in March, the worst had not hit, and there were visions of one giant wave of infection crashing across the country, swamping every ICU and ventilator at once. The lethality increases by perhaps an order of magnitude if you are really sick and there is no room for you in an ICU. Now we see that both the timing and intensity vary by geography, allowing the mutual sharing of resources with the ebb and flow of the contagion across regions. The ICU problem is now being mitigated with time and money. And I would like to put in a plug for combat pay for our healthcare workers on the front lines, our most valuable resource, who are being disproportionally impacted.
   

3. When will we develop a test for antibodies?
   
   Was I really asking this only a month ago? Antibody tests are all over the news today. However, you need to read the fine print. For example, one such test warns that “Positive results may be due to past or present infection with non-SARS-CoV-2 coronavirus strains, such as coronavirus HKU1, NL63, OC43, or 229E.” The New York Times has a good podcast on the current state of both diagnostic and antibody testing, but stay tuned, as things in this area are evolving fast.
   
   Furthermore, according to the World Health Organization, having antibodies does not necessarily provide immunity. However, with such widely varying outcomes from the disease, having antibodies at least proves that if you catch it again it won’t be your first rodeo, and that you did pretty well in your last one. 

Getting Back to Work – An Unstable Equilibrium

There is now a healthy debate about the risks to the economy of staying shut in too long vs. the risks to our health of opening up too soon. We won’t just make a single decision as a nation and charge ahead regardless of outcome, but instead we will monitor the situation location by location and adjust as needed. But controlling a pandemic is not a stable equilibrium, like driving a car that tends to go straight when you let go of the wheel. It is unstable, like balancing a broomstick on your hand. You must continually monitor its motion, and if it falls over it will rekindle the pandemic. Furthermore there are three additional complications. First, there are many brooms, and if you let one fall over it may spread infection to the others. Second, the positions of the brooms are monitored by clinical testing and contact tracing, which we are still not set up to carry out on a large scale. Third, instead of observing the positions of the brooms, we are seeing a delayed video of the positions, because it takes a couple of weeks for new infections to show up. In short, we don’t know how it will turn out.

My Next Questions

Below are three more questions I hope we get answers to by next month.

1. How will the economy opening experiments go?
   
   Different experiments are being tried in different countries, states, and geographic regions and there will be both health related and economic lessons from each. In particular, Brazil’s relaxed approach and those of Sweden, Denmark, and New Zealand, as described by the BBC, are worth watching. Who will be successful at balancing their broomsticks? And when some inevitably fall over, how big a second wave will they make?

2. Do antibodies make us immune?
   
   Hopefully in a month we will have a better understanding of this issue. Depending on what we learn, we may be able to offer immunity passes for people to head out into the world again.

3. New therapies
   
   Two things have moved in the right direction since March 26. First, we continue to build our ICU capacity, and second, due to the higher background prevalence of COVID-19, the lethality is less than we first thought. Now imagine that through survivor plasma, some new antiviral drug, or a way to treat cytokine storm, we further reduce the lethality while continuing to grow our capacity. Might we reach a tipping point at which could open the economy much faster?  We are not there yet, but what will things look like in a month?  

In March I wrote that demonstrating some control over the pandemic would be rewarded by the financial markets. Since then we clearly have been able to flatten the curve by hunkering down, and in the last month the Dow has risen about 10%. That would be great in normal times if that’s any comfort.

But just as uncertain as COVID-19 are the economic impacts of shutting in. The next month will reveal much in this area as well, hopefully enabling us to make better econopandemic decisions than than we can now.

© Copyright 2020 Sam L. Savage

Riding My Book

In which I provide a loose translation of Jensen’s 1906 “Inequality” paper from the original French.

by Sam L. Savage 

At a time when the coronavirus pandemic has put many people out of work, a number of people I know find themselves busier than ever. I am blessed to be in the latter category and my heart goes out to those in the former. Thank you Thomas Paine, Milton Friedman, and Andrew Yang for the continuing dialog on Universal Basic Income. I hope those checks start arriving soon.

Why are the rest of us so busy? Here are some anecdotal explanations heard from friends. “I no longer commute.” “I used to leave work at the office, but now I wake up and start working and the next thing I know it’s 10PM.” “Our Zoom meetings aren’t as effective as face to face discussion, so everything takes longer.” “The university has just switched to totally online teaching without warning and there are tremendous setup issues.”

In my case the transition was easy. I have worked from home since 1997 and am used to filling up 16 hours a day with procrastination and a little work. I have stayed up to speed on teleconferencing technology and had already been teaching some of my Stanford classes via Zoom. The worst thing so far has been the closing of the beloved YMCA, 300 steps from my house, which I used to visit twice a day.

So why am I so busy? Recall that I am the primary publicist for Jensen’s Inequality, (which I have rebranded as the strong form of the Flaw of Averages). There is a link below to Johan Ludwig William Valdemar Jensen’s original 1906 paper from Acta Mathematica. If you can’t read mathematics and French at the same time, I have provided a loose translation below.

Loose translation of Jensen’s 1906 paper:

Sur les fonctions convexes et les inégalités entre les valeurs moyennes

Seventy-six years from now when they invent electronic spreadsheets, most people will be uncertain about the numbers they are plugging in, so they will just enter the average values. And a majority of those dumbasses won’t have a clue that the numbers coming out are generally not the average outputs.

And by the way, there are no exemptions during pandemics when the extra uncertainty only accentuates this problem. So remember, boys and girls, my inequality works 24/7, 365 days a year, rewarding options traders and others who thrive on uncertainty, and punishing those who insist on basing decisions on single average numbers.

Jensen must be turning over in his grave to see a billion people ignoring his advice today. But where was I?

Oh yes. As Jensen predicted in 1906, one reason I’m so busy is that the pandemic has created a target-rich environment for his famous inequality. But on top of that, this is the perfect time to finally push out the second edition of my book on the Flaw of Averages. So to create extra time in my day, and maintain my exercise routine, I constructed the CVWP (cardio-vascular word processor) out of spare parts, pictured above. You might think it would be hard to type and spin at the same time. First of all, as it turns out, a lot of what I am doing at this stage is proofreading, which has led to an interesting discovery. I am a slow reader, but if type something like this, leaving out the word “I,” by using Microsoft Word’s Read Aloud function, I can proofread quickly with a part of the brain that was not guilty of the original typo, all the while spinning my heart out. When I do find a problem, I slow up a bit and have plenty of keyboard bandwidth to fix it with redlines on. For the real diehards, the Dictation feature allows you to speak into Word as well, but I am not yet fully proficient at that.

For those interested in updates to the book, you may visit FlawOfAverages.com to explore some new material. In particular there is a link to over 20 annotated SIPmath models in Excel covering a wide range of applications.

Copyright © 2020 Sam L. Savage

Introduction to the Value of Information And the XLTree™ Software

by Dr. Sam L. Savage

A classic example of the value of information involves the decision of whether or not to purchase a $25 umbrella in the face of a known 10% chance of rain. If it does rain and you do not buy the umbrella, you will do $100 damage to your suit. This is displayed in the decision tree below, in which, in keeping with tradition, a square represents a decision and a circle represents an uncertainty.

If you purchase the umbrella, you incur a $25 cost regardless of weather (upper branch). If you take your chances, there is a 10% chance of a $100 penalty, for an average of -$10. The decision tree software is no fool and goes for an average of losing $10 over a sure loss of $25 (green path).

The tree above was created with the XLTree Excel add-in (a beta version of which will be available with documentation to all participants in my upcoming webinar before we post it on our Tools page). This software was developed for my textbook, Decision Making with Insight, which was published in 2003. I donated XLTree to ProbabilityManagement.org, and updated it to use the Excel ribbon interface as shown below.

But back to the value of information. You have already decided not to buy the umbrella when a fairy walks down a moon beam and says, “I can tell you whether or not it will rain tomorrow.” So you say, “Cool, give me the scoop.” And the fairy says, “We haven’t worked for free since the 20th century, that’ll be two bucks.” Is it worth it? Here’s how to think about the value of information.

Without the information, you need to DECIDE what to do and then FIND OUT whether or not it will rain.

With the information, you will FIND OUT whether or not it will rain and then DECIDE what to do.

This is called flipping the tree, which you can do with the software. And when you flip the tree above you get the figure below.

The fairy will tell you that it will rain with 10% probability and that it won’t rain with 90% probability because those are the true probabilities and fairies don’t lie. Now, 10% of the time you will buy the umbrella for $25 and save your suit, while 90% of the time you will do nothing (see the green lines representing your decision under each case). On average you are out $2.50. So, what’s the information worth? You went from an average of average cost of $10 down to $2.50, an information value of $7.50. The Fairy’s offer of two bucks is a steal.

The case above describes what is known as the value of perfect information. Had you received the tip from a gnome (who are known to lie occasionally), we would have applied the value of imperfect information and it would have involved a more complicated tree.

© Copyright Sam Savage 2020

The Value of Information About COVID-19: What Will the Information We Learn In the Next Two Weeks Be Worth?

by Sam L. Savage

The Coming Surge in the Value of Information

Video source: Khan Academy

Because we are not having a probabilistic discussion of COVID-19 (which it is our mission to correct), most models are based on single number assumptions for infection rate and other critical factors. This, of course, leads to the Flaw of Averages, about which I have written in previous blogs. But in addition, point estimate thinking masks the economic benefit in information value we receive every day that we learn more about this pandemic.

It’s as if we are living in a thriller TV series, and I have a feeling that the next few weeks will provide a lot of reveals. It is complicated to project the future in any event, and I suggest the excellent video above on predicting the impact. But remember that this model, like all the others, will be greatly impacted by things we learn soon.

Most of our current data on this subject is biased because it is taken from patients who are exhibiting symptoms. John Ioannidis argues that we are making decisions without reliable data. An un-mentioned benefit of flattening the curve is that it buys time to reduce some of the uncertainty about this pandemic and alter our decisions.

Some Uncertainties That May Be Reduced

What information could we learn in the next couple weeks, and what could it be worth? Here are a few observations.

1. What percent of the population has already been infected? A recent Wall Street Journal article reports that when they started testing professional basketball players, several, including Kevin Durant, were positive but had no symptoms. They write,

“This small, accidental experiment echoes what more scientific studies are finding: People with no symptoms are carrying the sometimes-deadly virus without knowing it—and might be inadvertently helping it spread.”

Remember that for information to have value it must have the potential to change a decision. As we learn more about the rate of infection and severity of symptoms or lack thereof of the population at large, it could change our decisions about who stays shut in. It may be good news in the sense of indicating herd immunity, which would put a damper on the overall numbers who could become acutely ill.

2. Will we truly overwhelm our healthcare facilities? The view that the entire country is having the same pandemic is a spatial version of the Flaw of Averages. Because the degree of criticality will vary from place to place over time, we may have the option to move ventilators, army field hospitals, and perhaps even healthcare workers from place to place as needed. This could have a big effect on overall fatalities, and I don’t see how this could be accomplished without Federal coordination. Any such victory would not only save lives, but also signal that we are regaining control, which the markets would love.

I presume that if we conquer the critical care shortfall, we will be encouraged to stand at the bottoms of escalators with our tongues on the handrails to get this ordeal behind us as quickly as possible. 

On the other hand, if fatalities spike in one or more of our cities, as they did in Italy and Spain, the tragedy is likely to change behavior in other cities and increase our tolerance for hunkering down.

3.  Will we develop a test for immunity? Current tests indicate whether you are shedding the virus or not. But researchers are hard at work on tests to determine if you have antibodies that indicate you are immune. This could be a game changer. Imagine that we knew a large fraction of the population was infected with the virus, but as described in 1 above, are symptom free and don’t know they had it. A test that proved immunity would identify the people who could go back to work, to restaurants, movies, and airports, and reboot the economy. Surely that would be worth $1 trillion about now. I can imagine being issued a license to mingle, once you have tested for the antibodies.

Graph by Connor McLemore

Michael Levitt, a Stanford Nobel Laureate, recently told the LA Times that he “sees signs that the United States may get through the worst of the COVID-19 pandemic well before many health experts have predicted.” And ProbabilityManagement.org’s Chair of National Security Applications, Connor McLemore, points out in a recent post that perhaps we are seeing an acceleration of positive tests because we are testing so much now, but that the virus itself may actually be slowing. He created a scatterplot below from University of Oxford data, comparing by country confirmed cases of COVID-19 to number of tests performed, as explained in further detail in his post.

Let’s Not Squander the Information

Now back to flattening the curve. If things turn out better than the direst warnings of the healthcare professionals, will it have been a bad decision to shelter in place? Do you own a house? Did you buy fire insurance last year? Did your house burn down? No? I guess you won’t waste your money on that again!

No. Buying insurance is a good decision for most of us, and regardless of what happens, flattening the curve was a good decision for now. But as questions like the ones above are answered, we must be prepared to change our decisions, or we will squander the value of the information that we will be gaining in boatloads over the next few weeks.

© 2020 Sam L. Savage

The Flaw of Averages in Flattening the Curve

by Sam Savage

Pandemics are fraught with probabilistic issues that I believe our organization can shed light on. Several projects in this area are being pursued by the probability management community right now and are likely to find their way into this blog soon.

The current installment is an interactive model to get your limbic system engaged with the issue of flattening the contagion curve. An article in VOX by Eliza Barclay and Dylan Scott brought this issue to my attention. COVID 19, or novel coronavirus, appears to be extremely contagious because our immune systems have never seen it before, hence the name novel. Luckily it only makes a small fraction of the population critically ill, and most of them will recover if they are on ventilators. The problem is that if the contagion grows too fast, that small fraction will be big enough to swamp our critical care capacity. This happened in Italy, where there are more critically ill patients than the hospitals can handle and doctors have been forced to make agonizing decisions over which lives to save. So, one solution is to flatten out the curve, to get more of it below the capacity line. As I write this, many of us are sheltered in place (with our vast supplies of toilet paper) in an attempt to slow the rate of infection, which could change the curve dramatically.

However, it’s not that simple. As pointed out by economist Joshua Gans in The Startup, there are drawbacks to flattening the curve. First, if fewer people are catching the virus, fewer people are becoming immune to it, which means the social distancing measures must continue for a longer period of time (or stop and then start up again). Second, there is an enormous economic cost associated with being hunkered down with your TP instead of being at work, and the only way to avoid this cost is to achieve a certain level of immunity in the general population. Third, if the critical care capacity is very low, then more people will die, because even flattening the curve doesn’t move many people below the critical line. Of course, if you don’t flatten the curve, then Gans’ recommendation is to “dramatically increase health care capacity” by constructing makeshift hospitals. Experiment with the capacity on the downloadable model and you will see the sensitivity to the initial level.

Then of course there is the Flaw of Averages (Jensen’s Inequality to mathematicians). This implies that if your calculation of lives saved (by either flattening the curve or increasing capacity) is based on the average contagion rate, then on average you will overestimate the benefits of either intervention. Also see my earlier blog post on another effect of the Flaw of Averages on contagion.

So, the decision on how much to spend on flattening vs. increasing capacity is complicated. If your country has no critical care capacity whatsoever, flattening makes no sense. In other cases, it might save hundreds of thousands of lives.

The bottom line is that such decisions should be made by people way above both my pay grade and knowledge of epidemiology. But ProbabilityManagement.org has created an open standard that allows experts and high-level decision makers the ability to communicate and perform risk calculations with the uncertainties surrounding the pandemic in a way that is unambiguous and auditable. This would allow respected researchers to create open SIP libraries of contagion rates, etc. from their own models. These could be accessed using Excel, R, Python or virtually any other modeling environment, allowing organizations to aggregate consolidated risk statements across the globe under different assumptions.

As for now, my gut tells me that we should do everything we can to both flatten the curve and increase capacity for a few weeks until we better understand what we are up against. At that point, depending on what we learn, it may become more obvious how to optimally allocate our resources. Until then, if you become bored counting your rolls of Charmin, you may enjoy playing with the accompanying interactive model, which is designed only to be directionally correct, and should not be used for decision making.

Limbic Analytics from ProbabilityManagement.org RSS

© Copyright 2020 Sam L Savage

A Case Study in the Value of Information (V.o.I)

Cost of Postponing V.o.I. Conference is Outweighed by the V.o.I. of Doing So

Join us for a Conversational Webinar with Ron Howard, the Inventor of V.o.I.

by Sam L. Savage

Stanford’s Ronald A. Howard, Inventor of the Value of Information

As of January 30th 2020, there was huge uncertainty in the future evolution of coronavirus infections worldwide (see my recent blog post on the topic). This provided the management team at ProbabilityManagement.org with a perfect case study in the Value of Information as applied to our Value of Information conference scheduled for April 21 - 22. Because uncertainty is the nonprofit’s stock-in-trade, our readers may be interested in how we have attempted to turn the lemons of this uncertainty into lemonade.

This story begins with my father, Jimmie Savage, who had laid the foundations for what would later be called Decision Analysis (DA). However, I had never taken a class in DA, nor had I heard of Stanford’s Professor Ronald A. Howard, one of its founding fathers, and inventor of the Value of Information. A casual conversation with Ron shortly after I arrived at Stanford in the early 1990s set off resonances with scores of dinner table conversations from my childhood and I decided to sit in on his class. Just as you can see the minute hand move on a very large clock face, I could feel myself getting smarter with every minute of lecture. Like everyone else I know who took Ron’s course, I cannot imagine where I would be today without it.

Influence Diagram

To me the most beautiful and unexpected concept from the course was the Value of Information (see my blog post). It is defined as the expected economic benefit you would receive through a particular reduction in uncertainty.

Ron wrote a short guest essay for my book on The Flaw of Averages which I will quote directly in discussing our decision- making process. I will also make use of another of his inventions, the Influence Diagram. Quoting from Ron:

“When you really think about it, the three elements of any decision are alternatives, information, and preferences. I like to think of a three-legged stool because, if any one of these is missing, you really don’t have a decision. If you don’t have any alternatives, you don’t have a decision.”

For us, the alternatives for our April V.O.I. conference were:

1. Do nothing and hope for the best.

2. Do a hybrid conference which people could attend either live or online, and switch to completely online if things got really bad.

3. Postpone the conference.
   or

4. Do something else (read on)

   
   “If you don’t see a connection between what you do and what’s going to happen —  that’s the information —  you don’t have a decision.”

The information we were lacking was:

1. How was the contagion going to progress?
   and

2. How would other organizations and individuals behave in the face of this uncertainty? That is, would our speakers and attendees be unable to make the conference?

“And if you don’t care what happens, you can do anything and it doesn’t make any difference. So you need all three.”

Care? Are you kidding? Since 2014, our Annual Conference, the brainchild of board member Michele Hyndman, has been the heart and soul of ProbabilityManagement.org, and all our team has poured tremendous effort into the upcoming April conference. Furthermore, the numerous speakers and attendees were all eagerly looking forward to the event. For the umpty-umpth time, risk is in the eye of the beholder, and we had a lot of stakeholders to consider.

The Influence Diagram

In 1981 Ron and co-author James Matheson invented the influence diagram, also known as the Bayesian network, to visualize and aid with decisions. Although our group did not explicitly start with such a diagram, I had one in my head from the start, which I have graphed above.

1.       The Alternatives Box (the potential decisions)

We needed to act quickly enough so our speakers and attendees could plan their travel, but obviously could not wait to see how the disease would play out. All we had was the information on how others, rational or not, were reactive to the same uncertainties. So, the influence arrow going into the Alternatives node comes solely from the Behavior Uncertainty node.

2.       The Uncertainty Nodes

The actual progression of the contagion would clearly impact both people’s behavior and the conference itself. The behavior node would impact both our decision and the success of the conference.

3.       The Stakeholder Satisfaction Node

This would be impacted by what we decided to offer, whether or not an organization would send a speaker or attendee, and of course the state of the contagion in April.

How We Decided

First, we faced up to the uncertainty early and had begun researching our various alternatives. It pays to be proactive rather than reactive in the face of uncertainty.

Second, we had been negotiating with Ron to speak at the April conference, and because of the high value of his time, we gave him the option to accept up to March 1st. When I phoned him to get his decision, I discovered that he had an attractive offer to speak overseas, but he was not eager to do any public speaking due to the potential exposure to the coronavirus. I pointed out that we didn’t need to get his protons and neutrons down to San Jose but could just do it all with electrons and photons. So, he agreed, for which we are deeply thankful.

Third, information began to come in that reduced the uncertainty; speakers began to contact us saying that their organizations would not allow them to travel. This information had tremendous value. First, it involved some prominent speakers whose absence would diminish the value of the conference, and second, we knew that like the first few kernels of popcorn that go off in the microwave, many others would follow. This removed alternative 1: “Do nothing and hope for the best.”

Fourth, alternative 2, to offer the entire two-day conference online, had its own uncertainties and a big potential downside if it did not result in a satisfactory experience. Our final decision was to kick off a series of webinars on the Value of Information with Ron during the week of April 20 and postpone the live conference. More details on the webinars will follow shortly.

Fifth, we expect much of the uncertainty surrounding the epidemic to be resolved in the next couple of months. The value of that information will be that we can make firmer plans for the in-person conference, which we hope to host in the fall.

© Copyright 2020 Sam L Savage

Limbic Analytics from ProbabilityManagement.org RSS

A Combination of Coronavirus and the Flaw of Averages Can Drive You Nuts

by Sam L. Savage

All infectious disease epidemics start life with exponential growth. For example, suppose that each person infected with a disease in the first month infects one other person by the second month. Then those two people will infect two more, and so on, and the number infected will double with each time period. This exponential growth obviously can’t go on forever because eventually you run out of people, or at least susceptible people. So, in the end, the total number infected over time resembles an S curve as shown in Figure 1 below.

Figure 1: The Number of People Infected Over Time given “Average” assumptions.

A model like this assumes that you know the initial infection or growth rate. This is referred to as the Reproductive Ratio or R₀ (R naught) by epidemiologists and it would have been 2 in the example above. That is, in each time period the total number infected is 2 times the number in the previous period. Of course, R₀ can’t be known with certainty, so a statistical estimate is used. This results in both an “Average” value of R₀ and a standard error reflecting its uncertainty, the latter of which is usually flushed down the toilet. This leads to a classic case of the Flaw of Averages, also known as Jensen’s Inequality.

When someone brought this problem to my attention during the Ebola scare, I built a Monte Carlo simulation, available on our Models page, which reflects the uncertainty in R₀ as shown in Figure 2.

Figure 2 - Simulated Disease Trajectories Given the Uncertainty in R0

The uncertainty in a parameter in a model such as R₀ is often referred to as Model Risk. That is, each of the paths above is a potential model of the epidemic, like parallel universes in a Rick and Morty cartoon. The risk is that we don’t know which one is correct, so it makes sense to pick the “Average” of all 1,000 paths, which The Flaw of Averages tells us will be different from the path associated with the “Average” R₀ in Figure 1. When I ran the simulation, I was surprised to see that the average path is systematically different from the path of the “Average” R₀, regardless of disease, as shown in Figure 3.

Figure 3 – Path of the Average R0 vs Average Path

For all infectious diseases, the flawed path associated with the average growth rate systematically underestimates the severity early in the epidemic (before month 12 in the above example) and overestimates the severity later in the epidemic (after month 12). For this example, in month 9, you expected 5% of the population to be infected, but on average you will observe 10%. Then at month 18, you expected 35% but only observed 30%. Although a single case does not win a statistical argument, the Ebola epidemic of 2014-2015 fits the bill perfectly. It started out with fears of “We’re all going to die! We’re all going to die!” and ended up with the development of effective medications, and the realization that “they did not have many cases left to test it on.”

Remember, this says that if you average over all epidemics, you will underestimate the early growth and overestimate the late growth. And although the uncertainty in R₀ is only one of many, it seems unwise to leave any systematic error in calculations with this much social impact.

As to the model, like all nonlinear difference equations, this one can go chaotic for some input values. This model was inspired by by James Gleick’s book “Chaos: Making a New Science.” 

For a chaotic time, download the model and set the parameters to the values suggested.

© Copyright, Sam L. Savage 2020

Limbic Analytics from ProbabilityManagement.org RSS