Henry Spencer got it right (no big surprise):

The gap between engine cut off and staging was 1.5 seconds – which was fine for the ablatively cooled engine on Flight 2. But on Flight 3, with the regeneratively cooled engine, there was some residual thrust after engine shut down and this caused the first stage to be pushed back toward the second stage after separation and there was a recontact between the stages.

One of the big mistakes that people make in writing requirements is not writing proper verification statements for them. One of my rules, that I came to late in life, is to not allow a requirement to be accepted unless it has an accompanying verification statement (i.e., how you verify that the requirement has been satisfied). If you can’t write a verification statement for it, it’s not a valid requirement. The other reason is that verification is where most of the cost of a program comes from. Test is very expensive. If you can come up with ways to verify early on that don’t require it (inspection, demonstration, analysis), you can control and estimate costs much better.

One of the key elements of a proper verification statement is the environment. It’s not enough to say, “Verify, by test, that engine thrust is less than TBD Nt TBD seconds after engine shutdown.” It has to be “Verify, by test, in vacuum, that engine thrust is less than TBD Nt TBD seconds after engine shutdown.”

AMROC had a similar problem on SET-1 back in 1989, because the propulsion system testing had all taken place in the desert at Edwards, and the actual launch occurred in the humid October weather of Vandenberg, at the coast. The LOX valve iced up. The vehicle ended up catching fire and fell over and burned on the pad. There was no explosion, but it was a launch failure.

This is why systems engineering processes were developed. I’d be curious to know what kind of SE processes SpaceX had in place. And what they’ll have in place in the future…

[Late evening update]

Here’s the official statement from Elon Musk:

Timing is Everything

On August 2nd, Falcon 1 executed a picture perfect first stage flight, ultimately reaching an altitude of 217 km, but encountered a problem just after stage separation that prevented the second stage from reaching orbit. At this point, we are certain as to the origin of the problem. Four methods of analysis – vehicle inertial measurement, chamber pressure, onboard video and a simple physics free body calculation – all give the same answer.

The problem arose due to the longer thrust decay transient of our new Merlin 1C regeneratively cooled engine, as compared to the prior flight that used our old Merlin 1A ablatively cooled engine. Unlike the ablative engine, the regen engine had unburned fuel in the cooling channels and manifold that combined with a small amount of residual oxygen to produce a small thrust that was just enough to overcome the stage separation pusher impulse.

We were aware of and had allowed for a thrust transient, but did not expect it to last that long. As it turned out, a very small increase in the time between commanding main engine shutdown and stage separation would have been enough to save the mission.

The question then is why didn’t we catch this issue? Unfortunately, the engine chamber pressure is so low for this transient thrust — only about 10 psi — that it barely registered on our ground test stand in Texas where ambient pressure is 14.5 psi. However, in vacuum that 10 psi chamber pressure produced enough thrust to cause the first stage to recontact the second stage.

It looks like we may have flight four on the launch pad as soon as next month. The long gap between flight two and three was mainly due to the Merlin 1C regen engine development, but there are no technology upgrades between flight three and four.

Good Things About This Flight

• Merlin 1C and overall first stage performance was excellent
• The stage separation system worked properly, in that all bolts fired and the pneumatic pushers delivered the correct impulse
• Second stage ignited and achieved nominal chamber pressure
• Fairing separated correctly
• We discovered this transient problem on Falcon 1 rather than Falcon 9
• Rocket stages were integrated, rolled out and launched in seven days
• Neither the near miss potential failures of flight two nor any new ones
  were present

The only untested portion of flight is whether or not we have solved the main problem of flight two, where the control system coupled with the slosh modes of the liquid oxygen tank. Given the addition of slosh baffles and significant improvements to the control logic, I feel confident that this will not be an issue for the upcoming flight four.

Elon

About SpaceX:

If the problem is confirmed to be a simple and easily fixed design flaw, they may not launch again “tomorrow” but I wouldn’t be too surprised if there was another flight within a couple of months.

…Lost in the hubbub over the flight failure was the fact that once again they were able to do a quick resumption of the launch procedure after a hot-fire abort. This sort of robustness in the launch operations and the use of small crews are crucial factors in lowering the cost of launch.

And as noted, the new Merlin apparently performed well. Had it not, that would have been a real setback for both Falcon 1 and 9.

[Update after lunch, Pacific Time]

Henry Spencer has more thoughts, with some history.

It seems quite likely that it was caused by the new engine–that’s the only thing that changed between the last flight and this one, and Henry points out a couple potential plausible scenarios for that.

That doesn’t mean that there’s anything wrong with the engine–it just means that the overall vehicle design and operations have to account for the new characteristics.

It’s looking like there was a second-stage problem, either separation, or ignition (or both, since one could cause the other).

As I said last night, this is obviously a disappointment to the SpaceX team. Particularly since they had previously had a flight where this wasn’t a problem, so in a sense it was two steps forward, one step back. I think that at this point, almost anyone is going to be pretty leery of putting a payload on the vehicle until it’s had at least one successful flight. Is it the end? Despite what Elon said a long time ago about three strikes, it’s hard to see it now. He’s fully invested now, both financially and (I would imagine) emotionally, and he’s not going to come this far just to give up, particularly when tantalized by his previous almost-success on the second flight.

They’ll go through the telemetry, figure out as best they can what happened, and try again, and hopefully soon. In a sense, as someone noted in comments in the earlier post, Falcon 1 is really a test program for the bigger vehicles, though they should get an operational small launcher out of it as well.

As always, this points up the problem with expendable vehicles. They are very expensive to flight test, so you can’t afford to do very many, and every flight is a first flight, so you can’t wring bugs out of a vehicle with incremental testing. And it’s a lot harder to figure out what went wrong because you generally don’t get much debris to analyze (the first flight that failed off the pad was a rare exception)–you have to dig through electronic entrails. And NASA, of course, in its cargo-cult determination to redo Apollo, is taking exactly the same expensive and unreliable approach.

And just checking now, I see that Clark is having similar thoughts to mine.

Once the problem that caused this failure is determined, I would suggest that SpaceX just bite the bullet and allocate 2 or 3 Falcon I vehicles for test flights and fly them within a relatively short period, say six months.

This would represent a $20M-$30M investment but until the Falcon I is flying reliably, SpaceX will find it very difficult to get any more commercial or government payload contracts and it won’t have any chance of getting COTS D (ISS crew transport) funding. The Falcon 9 is a completely different vehicle but the Falcon I is what currently defines the company’s ability, or inability, to deliver what it says it can.

Anyway, best of luck to them in the future, but they know that they need more than just luck.

[Update a couple minutes later]

I see that Elon has a statement, which confirms my suspicion above:

There should be absolutely zero question that SpaceX will prevail in reaching orbit and demonstrating reliable space transport. For my part, I will never give up and I mean never.

That’s the kind of attitude you have to have, even if eventually, you do in fact have to give up. I hope he won’t have to.

Also note Clark’s comment at the end of the post, that SpaceX is following in the tradition of all expendable staged launch vehicles in its failure modes, though they do seem to be getting the avionics right.