Regretful

About five years ago, I did a "regret analysis" on whether or not we should remove Saddam Hussein:

From a "minimax" standpoint, the current course is the lowest-cost one.

Of course, some would argue that this is too simplistic an analysis, because (among numerous other reasons) it doesn't take into acccount the probabilities of the various scenarios being true, which, if you had them, you could multiply them by the costs to get expected values.

Of course, the problem with that approach here is that, if the cost estimates are wild-ass guesses, the probabilities would be even more so. How much confidence could we have in the output of such an analysis?

What we're dealing with here is not risk, in which the probabilities can be reliably quantified, but uncertainty, in which they cannot.

As an example, a thirty percent chance of rain represents risk. "It might rain, or it might not, but we have no idea what the probability is" constitutes uncertainty. It's much easier to decide whether or not to take an umbrella in the first circumstance than the second.

For this reason, economists have come up with a more sophisticated technique for decision making in the absence of probabilities of outcomes. Rather than simply looking for the lowest cost, they instead try to minimize how bad you'll feel if you make the wrong decision--they minimize "regret."

It's based on the notion that when you make a decision, you shouldn't compare it to some unattainable ideal of zero cost--you should compare it to the best decision you could have possibly made.

Take a simple case--do you take an umbrella when it rains, or not?

Consider a generic cost matrix:

State 1	State 2	Max
3	4	4
1	5	5

It looks like we can minimize our maximum cost by choosing action 1, since four is less than five. But is that really the right decision?

Let's derive a "regret" matrix from it. This is done by finding the minimum cost for any state, and subtracting each cell of that state from it. The minimum cost for state one is 1, so the column would be three minus one for the first row and one minus one (or zero) for the second row. That makes intuitive sense, since if you made the right decision for that state, you'll have no regrets. The regret matrix for the example cost matrix is shown below:

State 1	State 2	Max
2	0	2
0	1	1

Note now that if we want to minimize regret, we should actually choose action 2. Note also that this is independent of the relative probabilities of the two states.

NASA is confronted with exactly this kind of uncertainty with the vibration issue on the Ares 1. They don't know how big the problem is, and have no way of quantifying it with current knowledge. Thus, they're going to spend billions on getting to an initial flight test, and hope that they don't find out that they'll have to spend additional billions (not currently budgeted) to fix the problem, or give up completely and go to a new design (with more billions not currently budgeted), whereas if they knew now that it was intractable, they could stop wasting money on it and move directly (so to speak) on to a different concept.

Now, I don't have access to the program data to properly fill out the cost matrix, so the following numbers are pulled out of an orifice, but hopefully not the nether one--my WAGs are better than those of many with such things.

Let's keep it simple, with two courses of action, and three states.

One course is to abandon the concept now (noting that there are other reasons to do this than only the vibration problem--that's just the latest issue). The other is to continue forward with NASA's current plan.

The three states are:

1. There is no problem--the frequencies of the solid don't resonate with the vehicle structure, and don't present any hazard to upper stage, crew vehicle, or crew
   
2. There is a problem, and it will take a lot of time and money to mitigate it with dampers, shifting mass around, beefing up structure, etc.
   
3. There is a problem and it's not mitigatable, because the measures that would be required to do so would make it too heavy to deliver the required payload to orbit.
   

This provides us with six cells in the matrices, which have three columns (corresponding to states of reality) and two rows (the potential courses of action).

First let's consider Row 1: NASA's current plan.

Option 1: There is no problem. That is the hope (but as military planners will tell you, hope is not a plan).

What is the cost? Nothing. Or rather, the cost is what they expect to spend on the program if it's nominal. Let's call it a billion, on the assumption that this is what it will cost to get us to the flight test (if anyone has a better number, let me know).

Option 2: There is a problem, but it involves major changes to the vehicle design to compensate for it.

Let's say (to be kind) that it costs a year in schedule (what's the value of that?) and an additional billion dollars in development costs. Let's be generous again, and say that the year delay (in terms of "gap") is only an additional couple of billion. So the cost is the billion it takes to get there, a billion to fix plus the two billion for the delay--a total of four billion.

Option 3: The problem is intractable. There is no way to build the vehicle in such a way that it can deliver the required payload into orbit without shaking itself and/or the payload apart.

Now the cost is the billion dollars to get to flight test, plus a new design, almost from scratch, and about three years lost. Let's say that the new vehicle is a two billion dollar program, relative to what NASA would have spent to complete Ares 1 post flight test. If the gap costs two billion a year, then we have a total of nine billion dollars cost in this worst case.

OK, now for Row 2--scrapping the concept now and getting a head start on a design that will work. The cost is the same in all three cases. It's the cost of developing the new vehicle relative to expected expenditures on the Ares from here forward, plus, say, a two-year addition to the gap. Call it seven billion.

So the cost matrix looks like this:
COST MATRIX

So the minimax solution, based on the cost matrix alone, is to switch now. It all depends on what you think the likelihood is that the problem will be intractable. We don't know that that is, so let's look at the regret matrix.

REGRET MATRIX

Now, the course of minimum regret is to move forward. Regret is zero if there is no problem or it's fixable, and a max of two billion if they have to start over, whereas they risk a six billion dollar regret by giving up now.

So, at least a cursory analysis would indicate that NASA's approach makes sense, but I could be way off on the numbers. In addition, I'm not counting all of the less tangible costs of having to switch gears after a flight-test failure. Any thoughts from anyone else? Am I missing something?