Original Content

Reviewing Decisions Made in the Face of Uncertainty & Post-Hoc Risk Analysis

Reviewing our response to covid is vital for several reasons, not least of which is learning lessons for the next time we face similar circumstances. I see a lot of good lessons being discussed in the scientific community, but most of the public discourse trying to come to terms with our response is abysmal. One repeating pattern is holding us back from progress and contributing to the already deep divide in our politics while encouraging paranoid and conspiratorial thinking.

The conditions we faced throughout the pandemic (and even now) are uncertain. That is to say, we knew very little, and the little we did know was low confidence. Our knowledge and confidence continue to improve, which is great, but it can complicate reviewing our public-health decisions. This sort of progression makes it appealing to find and prop up the people who bet everything on a premature conclusion that happened to be correct. This is a poor way to review our risk analysis and can lead to extremely harmful conclusions and misplaced trust.

A few basic concepts in risk analysis:

If a product, an action, or a policy has a suspected risk of causing harm to the public or to the environment, protective action should be supported before there is complete scientific proof of a risk. (Precautionary Principle)

Suppose preventive measure X has a 1/10 chance of preventing unwanted outcome Y. It can be worth doing X if Y is severe enough relative to the cost of X, even if we know X is unlikely to prevent Y. (This is often confused with the Precautionary Principle, and although they often go hand-in-hand, this concept involves no uncertainty.)

When we review decisions made in the face of uncertainty, we must limit the analysis to the facts known at the time. This means that our analysis can find that the decision was correct while the action, in hindsight, was not.
This is an essential category for reviewing prior decisions when uncertainty is involved.

An extreme example: a gambling addict announces to his family that it’s his lucky night. He announces to his family that he will put their savings on 35 black, that he will win, and that they will never need to worry about money again. Though nobody knows the outcome, his family urges him to rethink his plan because it’s a bad decision. The addict gambles and wins. Our analysis needs to reflect that while he was technically correct, the best risk analysis at the time would have discouraged the gambling.
(I’m looking at you, Great Barrington Declaration, and you didn’t even get a chance to gamble with the lives of our country’s most vulnerable population.)
Two things to keep in mind as we ponder COVID:

  1. The snails-pace of bureaucracy. We shouldn’t be happy about this or even accept it. Still, we need to acknowledge it so that we don’t attribute to malice that which is adequately explained by incompetence (or bureaucracy).
  2. Preventive measures are not true/false. They don’t work or fail. They reside on a spectrum of efficacy.
    So, without doing a complete analysis, let’s rule out some of the more outlandish analyses and post-hoc claims.

Mask Encouragement / Mandates

Cloth masks don’t work.

If you think that covering your cough works, you must concede that cloth masks work. We all encourage covering coughs.
Medical masks are better; we always knew this, but they were not practical for public health recommendations and requirements.


Vaccine Encouragement / Mandates

Public health officials and pharmaceutical companies lied about what vaccines were capable of.

When the vaccine was first released, it was measured against the original strain of the virus. Many of the claims made were based on this comparison, and their optimism regarding reducing illness and spread was warranted.
The evolution of new variants obviously complicated this, but the vaccine’s efficacy is often understated to compensate for the perceived incorrect (but warranted) statements made relative to the original variant.

Lockdowns

They lied in order to further their agenda.

“They” and “agenda” are rarely, if ever, defined. That’s a red flag.
People are upset about the state of the economy. I get it. Times are tough.
However, there seems to be a strong urge to take mortality and illness statistics that have benefited from the preventive measures, treatments, a more mild dominant variant, and generally functional health services gained through lockdowns and try to force them back through a counterfactual in which we didn’t impose lockdowns. It just doesn’t work.

School Closures

School closures were unwarranted.

This one is tricky because I think it passed through “making the right decision at the time” to “we should have known better”. Both were true at different points, but the pattern seems to be pushing the line for “we should have known better” back way too far, even going so far as to question our decision to close schools at all, which is foolish. The risk of opening schools when they could have been a primary vector would have been catastrophic.

Leave a Comment