Regretful

About five years ago, I did a “regret analysis” on whether or not we should remove Saddam Hussein:

From a “minimax” standpoint, the current course is the lowest-cost one.

Of course, some would argue that this is too simplistic an analysis, because (among numerous other reasons) it doesn’t take into acccount the probabilities of the various scenarios being true, which, if you had them, you could multiply them by the costs to get expected values.

Of course, the problem with that approach here is that, if the cost estimates are wild-ass guesses, the probabilities would be even more so. How much confidence could we have in the output of such an analysis?

What we’re dealing with here is not risk, in which the probabilities can be reliably quantified, but uncertainty, in which they cannot.

As an example, a thirty percent chance of rain represents risk. “It might rain, or it might not, but we have no idea what the probability is” constitutes uncertainty. It’s much easier to decide whether or not to take an umbrella in the first circumstance than the second.

For this reason, economists have come up with a more sophisticated technique for decision making in the absence of probabilities of outcomes. Rather than simply looking for the lowest cost, they instead try to minimize how bad you’ll feel if you make the wrong decision–they minimize “regret.”

It’s based on the notion that when you make a decision, you shouldn’t compare it to some unattainable ideal of zero cost–you should compare it to the best decision you could have possibly made.

Take a simple case–do you take an umbrella when it rains, or not?

Consider a generic cost matrix:

State 1 State 2 Max
3 4 4
1 5 5

It looks like we can minimize our maximum cost by choosing action 1, since four is less than five. But is that really the right decision?

Let’s derive a “regret” matrix from it. This is done by finding the minimum cost for any state, and subtracting each cell of that state from it. The minimum cost for state one is 1, so the column would be three minus one for the first row and one minus one (or zero) for the second row. That makes intuitive sense, since if you made the right decision for that state, you’ll have no regrets. The regret matrix for the example cost matrix is shown below:

State 1 State 2 Max
2 0 2
0 1 1

Note now that if we want to minimize regret, we should actually choose action 2. Note also that this is independent of the relative probabilities of the two states.

NASA is confronted with exactly this kind of uncertainty with the vibration issue on the Ares 1. They don’t know how big the problem is, and have no way of quantifying it with current knowledge. Thus, they’re going to spend billions on getting to an initial flight test, and hope that they don’t find out that they’ll have to spend additional billions (not currently budgeted) to fix the problem, or give up completely and go to a new design (with more billions not currently budgeted), whereas if they knew now that it was intractable, they could stop wasting money on it and move directly (so to speak) on to a different concept.

Now, I don’t have access to the program data to properly fill out the cost matrix, so the following numbers are pulled out of an orifice, but hopefully not the nether one–my WAGs are better than those of many with such things.

Let’s keep it simple, with two courses of action, and three states.

One course is to abandon the concept now (noting that there are other reasons to do this than only the vibration problem–that’s just the latest issue). The other is to continue forward with NASA’s current plan.

The three states are:

  1. There is no problem–the frequencies of the solid don’t resonate with the vehicle structure, and don’t present any hazard to upper stage, crew vehicle, or crew
  2. There is a problem, and it will take a lot of time and money to mitigate it with dampers, shifting mass around, beefing up structure, etc.
  3. There is a problem and it’s not mitigatable, because the measures that would be required to do so would make it too heavy to deliver the required payload to orbit.

This provides us with six cells in the matrices, which have three columns (corresponding to states of reality) and two rows (the potential courses of action).

First let’s consider Row 1: NASA’s current plan.

Option 1: There is no problem. That is the hope (but as military planners will tell you, hope is not a plan).

What is the cost? Nothing. Or rather, the cost is what they expect to spend on the program if it’s nominal. Let’s call it a billion, on the assumption that this is what it will cost to get us to the flight test (if anyone has a better number, let me know).

Option 2: There is a problem, but it involves major changes to the vehicle design to compensate for it.

Let’s say (to be kind) that it costs a year in schedule (what’s the value of that?) and an additional billion dollars in development costs. Let’s be generous again, and say that the year delay (in terms of “gap”) is only an additional couple of billion. So the cost is the billion it takes to get there, a billion to fix plus the two billion for the delay–a total of four billion.

Option 3: The problem is intractable. There is no way to build the vehicle in such a way that it can deliver the required payload into orbit without shaking itself and/or the payload apart.

Now the cost is the billion dollars to get to flight test, plus a new design, almost from scratch, and about three years lost. Let’s say that the new vehicle is a two billion dollar program, relative to what NASA would have spent to complete Ares 1 post flight test. If the gap costs two billion a year, then we have a total of nine billion dollars cost in this worst case.

OK, now for Row 2–scrapping the concept now and getting a head start on a design that will work. The cost is the same in all three cases. It’s the cost of developing the new vehicle relative to expected expenditures on the Ares from here forward, plus, say, a two-year addition to the gap. Call it seven billion.

So the cost matrix looks like this:
COST MATRIX

No Problem Fixable Problem Insoluble Problem

Max
1 4 9 9
7 7 7 7

So the minimax solution, based on the cost matrix alone, is to switch now. It all depends on what you think the likelihood is that the problem will be intractable. We don’t know that that is, so let’s look at the regret matrix.

REGRET MATRIX

No Problem Fixable Problem Insoluble Problem

Max
0 0 2 2
6 3 0 6

Now, the course of minimum regret is to move forward. Regret is zero if there is no problem or it’s fixable, and a max of two billion if they have to start over, whereas they risk a six billion dollar regret by giving up now.

So, at least a cursory analysis would indicate that NASA’s approach makes sense, but I could be way off on the numbers. In addition, I’m not counting all of the less tangible costs of having to switch gears after a flight-test failure. Any thoughts from anyone else? Am I missing something?

8 thoughts on “Regretful”

  1. The initial examples of a cost matrix and a regret matrix would be easier to understand for those new to the concepts if you would flesh out the umbrella example — show where the costs come from (why “3”?), and give the matrices hadnon-generic labels (in English) throughout, instead of “state 1”, “state 2”, “rained” “didn’t rain”, and “brought an umbrella” and “didn’t bring an unbrella”, etc. I know the initial examples are from an earlier posting of yours, but you could edit the current entry. It would also be interesting to revisit the Iraq matrices in a more accessible format.

  2. Yeah, I’ve got a thought: public-choice theory. Which approach will maximize NASA’s budget and headcount for the greatest number of years? They’ll benefit from an insoluble, or greatly protracted, vibration issue.

  3. You have the cost of continuing to purchase Soyuz vehicles for ISS. The potential brain drain loss (not necessarily bad for US, if brains going to private industry) from NASA as years go by without operational missions.

  4. Seriously, how much would it cost to get an existing SRB, build a mock-up upperstage and capsule and fire it off from Kennedy? Would that answer some of these questions?

  5. What an interesting post, Rand. Thanks!

    I think your regret analysis is certainly the way a political animal like NASA will make its decision. In the political arena you get hammered only when you make such a wrong decision that you must quit and start over. Otherwise, if you can possibly bull ahead and pull off the mission goal, however much higher than expected the cost, you’ll generally survive. Cf., Bush, Jr, George and the Iraq war, or Stalin, J. and the Eastern Front in WW2.

    That is very different from the accounting in the private sector, in a competitive economy, where the expectation value of the cost is paramount, and you can easily lose if your costs are too high, even when you reach your mission goals. Cf. NeXT computers, the Multics operating system, Betamax, laserdiscs, EV1.

    So a larger point from your analysis could be quantifiable insight into the basic nature of the “accounting” rules that define success, and which explain decision-making, in the public versus private sector. Clearly some decisions are better made using the public/political rules, and some are better made using the private business rules. It would be interesting to know whether it’s possible to be more precise about which decisions are which. For one thing, that very question plagues much current political debate.

  6. In the political arena you get hammered only when you make such a wrong decision that you must quit and start over.

    That doesn’t seem to be true for NASA. Each time they fail to build a Shuttle replacement and have to start over, they get more money than they did before. When they had to quit on X-33, they got more money for SLI. When they had to quit on that, they got even more money for VSE/ESAS.

  7. Seriously, how much would it cost to get an existing SRB, build a mock-up upperstage and capsule and fire it off from Kennedy?

    An existing SRB wouldn’t be of any value, because they plan to use a five-segment booster for the Ares, which would have a much different resonant frequency. Also, with only a mock upper stage, the vehicle would have no roll control.

  8. OK, now for Row 2–scrapping the concept now and getting a head start on a design that will work. The cost is the same in all three cases. It’s the cost of developing the new vehicle relative to expected expenditures on the Ares from here forward, plus, say, a two-year addition to the gap. Call it seven billion.

    You also need to account for the possibility that the new vehicle design could run into problems.

    The safest solution would be to have more than one vehicle so there’s a backup if anything goes wrong.

    The strange thing about ESAS is that Griffin says NASA can’t afford to fail yet his architecture has multiple single-point failure modes. Ares I, Ares V, CEV, the lunar transfer stage, the lunar lander — there are no backups for any of these. If the development of any one component fails, NASA will never reach the Moon, and if any of these components fails after it enters service, the entire program will have to stand down. Which could be catastrophic if NASA’s already established a base on the Moon that depends on this non-redundant supply train.

Comments are closed.