Models are NOT Meant to be Accurate

Scott Adams made a point in yesterday’s video that I’d like to share. You can watch the video here, but the point he made is simple enough to convey in a few words: Models (like the IHME model CDC is using to forecast the coronavirus in America) are NOT meant to be accurate, they are meant to useful — typically in a way to convince others to take some action or another.

Adams bases his observation on his corporate experience before becoming a cartoonist, but I can attest to the truth of it based on my own years of experience as a corporate proposal manager. Every proposal contains a financial plan of which the primary exhibit is a spreadsheet that forecasts every foreseeable cost and expense to the customer. The thing is, everyone who works on the financial plan knows that it will have almost no connection to reality. You could even say that it’s a lie, except that would be naive.

It is not a lie to illustrate your thinking as the offeror. In fact, the point of doing so is to convince the customer that you know how to deliver on the promises the customer cares about.

Nor is the proposal financial plan merely an intellectual exercise. Before any work on the model begins, the capture team — typically senior and executive management — specify a “price to win.” The financial plan shows how the program will operate at that price. Thus, if the customer likes your price, the financial plan gives them the incentive to award the program to you.

The entire point centers on that one word, incentive. It is the same with IHME’s modeling of the coronavirus. The model doesn’t tell you anything about reality. It isn’t meant to. Its purpose is to incentivise you to take the experts’ advice on the pandemic.

You can look at that cynically, if you want. You might suspect and resent that you are being manipulated, for example. But how naive would that be?

You don’t actually need the IHME model to understand coronavirus or any other pandemic. That information is widely available elsewhere. You only need the IHME model to understand the proposal being made to you.

All that said, one of my concerns is that the proposal to continue universal social distancing until there is no trace of Covid-19 in America is a bad one on its face.

18 thoughts on “Models are NOT Meant to be Accurate

  1. @Roberts
    In my past life as the CFO or a publicly traded company and having prepared and signed my name to financial projections (models) used in the formal prospectuses for public offerings, I would now be in jail had I followed the standards that you say you used to make inside the corporation proposals.

    The statement . . .
    “The thing is, everyone who works on the financial plan knows that it will have almost no connection to reality. ”

    If that is true or even possibly true it would (a) have to be disclosed in a prospectus and (b) would result in nobody making an investment.

    In short, Adams – as reported by you – is wrong. Models are supposed to be BOTH useful AND reflect reality to the best ability of the modeler, i.e., “accurate.”

    But, since no one actually knows the future, the key to understanding the model and its usefulness is similar to that for financial projections made in a prospectus – very specific identification of ALL significant assumptions.

    Liked by 2 people

    1. RE: “I would now be in jail had I followed the standards that you say you used to make inside the corporation proposals.”

      I doubt it. The formal prospectus for a public offering is not a proposal.

      Given your experience, you need to grasp that once the customer accepts a proposal, the offeror becomes indemnified with respect to the contents of the offer. That is, a contract is issued, and that contract becomes the measure of performance.

      Like

      1. When I worked in one of the local shipyards in the Planning Department, it was common that once the repair work began, other issues were discovered that were not part of the initial package. Spec writers (such as myself) were required to either change the job scope or write completely new specification to repair the discovered work.

        Just a real life example of how planning and modeling change as more information becomes available.

        Liked by 1 person

      2. @Roberts

        Best you not offer “facts” about things you are not familiar with.

        There actually ARE serious criminal penalties hanging over people who issue a formal prospectus for a public offering. Lies or imaginary financial projections WILL land you in jail.

        As for the issuer being indemnified if a prospectus later proves fraudulent or substantially not “accurate,” that too is not correct. I personally was involved in a transaction where the junior partner to the deal became the senior partner because of the threat of litigation over representations in the form of financial projects were not “accurate.”

        It is obvious that rather than admit error you are now wandering off into a different areas. It may be OK in some contexts to advance a “proposal” with a financial model that is not “accurate” but I will be damned if I know what that context might be. You seem to be speaking from the point of view of a Navy contractor which has gotten away with low ball estimates. Unethical and border-line illegal because Adams is wrong – models are supposed to be both useful AND accurate.

        Liked by 2 people

        1. RE: “As for the issuer being indemnified if a prospectus later proves fraudulent or substantially not ‘accurate,’ that too is not correct.”

          I didn’t say that, and wouldn’t have because I do in fact know that a financial prospectus must meet legal requirements. So I’ll make the same point a different way: the financial plan in a proposal is not a prospectus. It is not a prediction. It is an explanation of the bid price.

          The ethical/legal issues you raise simply don’t apply here. You are, frankly, having another Don Quixote moment.

          Like

          1. @Roberts

            I am not raising ethical or legal issues per se. I am pointing out that the idea that models or projections need not be “accurate” and are not expected to be is nonsense. I would add that if they are not “accurate” then their being “useful” to persuade others is a harmful illusion at best or deliberate fraud at worse.

            I did not waste my time listening to Scott Adams again – I think he is a damned fool – but am relying on your characterization of the point he was making. Maybe you got it wrong. I dunno.

            Liked by 1 person

          2. RE: “I dunno.”

            No, you don’t. Let’s review: You didn’t watch the video, so you don’t know what it says. You thought proposals are comparable to financial statements, which shows you don’t know that topic, either.

            Your commentary is truly a waste of time.

            Like

          3. @Roberts

            If you say it was wrong to rely on your reporting of what he had to say, I cannot argue with you. I don’t intend to double check your reporting for the reason that I stated. But, for the sake of good order, I will repeat again that models, financial forecasts, estimates, projections etc. are supposed to be BOTH “accurate” and “useful” and any claim to the contrary – whether yours or Adams’s – is self-evident nonsense.

            Liked by 1 person

          4. RE: “any claim to the contrary – whether yours or Adams’s – is self-evident nonsense.”

            Suit yourself. But now that have established that you speak without knowing, so what?

            Like

  2. In one of the links in an earlier discussion on whether the IHME model failed after being revised, there was an observation about models.

    The crux was simple. Like hurricane tracking, a single model is not as accurate as multiple ones. The consensus of several different tracks is the projection for the cone of uncertainty. Outliers are usually just that.

    This was likened to the COVID19 modeling.

    Both types, storm and pandemic, have many variables, but also both benefit from data. The more data, the better.

    Liked by 1 person

    1. RE: “The more data, the better.”

      That’s fine as far as it goes, as long as you remember not to confuse models with truth.

      Like

      1. Actuarial projections work very well for insurance companies. The truth is no one can predict when anyone one person will die, but with enough data, the truth is that life expectancies will still hover around 80 years for enough people to make insurance lucrative.

        Liked by 2 people

      2. RE: “Actuarial projections work very well for insurance companies.”

        Only until they don’t. That’s why many insurance policies contain exemptions for epidemics.

        Like

      3. The standard for models is “accuracy” not “truth.” More data generally means greater “accuracy” which is means a better approximation of the truth which is as of yet unknown. No body actually confuses these two ideas.

        Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s