Henry Spencer got it right (no big surprise):

The gap between engine cut off and staging was 1.5 seconds – which was fine for the ablatively cooled engine on Flight 2. But on Flight 3, with the regeneratively cooled engine, there was some residual thrust after engine shut down and this caused the first stage to be pushed back toward the second stage after separation and there was a recontact between the stages.

One of the big mistakes that people make in writing requirements is not writing proper verification statements for them. One of my rules, that I came to late in life, is to not allow a requirement to be accepted unless it has an accompanying verification statement (i.e., how you verify that the requirement has been satisfied). If you can’t write a verification statement for it, it’s not a valid requirement. The other reason is that verification is where most of the cost of a program comes from. Test is very expensive. If you can come up with ways to verify early on that don’t require it (inspection, demonstration, analysis), you can control and estimate costs much better.

One of the key elements of a proper verification statement is the environment. It’s not enough to say, “Verify, by test, that engine thrust is less than TBD Nt TBD seconds after engine shutdown.” It has to be “Verify, by test, in vacuum, that engine thrust is less than TBD Nt TBD seconds after engine shutdown.”

AMROC had a similar problem on SET-1 back in 1989, because the propulsion system testing had all taken place in the desert at Edwards, and the actual launch occurred in the humid October weather of Vandenberg, at the coast. The LOX valve iced up. The vehicle ended up catching fire and fell over and burned on the pad. There was no explosion, but it was a launch failure.

This is why systems engineering processes were developed. I’d be curious to know what kind of SE processes SpaceX had in place. And what they’ll have in place in the future…

[Late evening update]

Here’s the official statement from Elon Musk:

Timing is Everything

On August 2nd, Falcon 1 executed a picture perfect first stage flight, ultimately reaching an altitude of 217 km, but encountered a problem just after stage separation that prevented the second stage from reaching orbit. At this point, we are certain as to the origin of the problem. Four methods of analysis – vehicle inertial measurement, chamber pressure, onboard video and a simple physics free body calculation – all give the same answer.

The problem arose due to the longer thrust decay transient of our new Merlin 1C regeneratively cooled engine, as compared to the prior flight that used our old Merlin 1A ablatively cooled engine. Unlike the ablative engine, the regen engine had unburned fuel in the cooling channels and manifold that combined with a small amount of residual oxygen to produce a small thrust that was just enough to overcome the stage separation pusher impulse.

We were aware of and had allowed for a thrust transient, but did not expect it to last that long. As it turned out, a very small increase in the time between commanding main engine shutdown and stage separation would have been enough to save the mission.

The question then is why didn’t we catch this issue? Unfortunately, the engine chamber pressure is so low for this transient thrust — only about 10 psi — that it barely registered on our ground test stand in Texas where ambient pressure is 14.5 psi. However, in vacuum that 10 psi chamber pressure produced enough thrust to cause the first stage to recontact the second stage.

It looks like we may have flight four on the launch pad as soon as next month. The long gap between flight two and three was mainly due to the Merlin 1C regen engine development, but there are no technology upgrades between flight three and four.

Good Things About This Flight

• Merlin 1C and overall first stage performance was excellent
• The stage separation system worked properly, in that all bolts fired and the pneumatic pushers delivered the correct impulse
• Second stage ignited and achieved nominal chamber pressure
• Fairing separated correctly
• We discovered this transient problem on Falcon 1 rather than Falcon 9
• Rocket stages were integrated, rolled out and launched in seven days
• Neither the near miss potential failures of flight two nor any new ones
  were present

The only untested portion of flight is whether or not we have solved the main problem of flight two, where the control system coupled with the slosh modes of the liquid oxygen tank. Given the addition of slosh baffles and significant improvements to the control logic, I feel confident that this will not be an issue for the upcoming flight four.

Elon

About SpaceX:

If the problem is confirmed to be a simple and easily fixed design flaw, they may not launch again “tomorrow” but I wouldn’t be too surprised if there was another flight within a couple of months.

…Lost in the hubbub over the flight failure was the fact that once again they were able to do a quick resumption of the launch procedure after a hot-fire abort. This sort of robustness in the launch operations and the use of small crews are crucial factors in lowering the cost of launch.

And as noted, the new Merlin apparently performed well. Had it not, that would have been a real setback for both Falcon 1 and 9.

[Update after lunch, Pacific Time]

Henry Spencer has more thoughts, with some history.

It seems quite likely that it was caused by the new engine–that’s the only thing that changed between the last flight and this one, and Henry points out a couple potential plausible scenarios for that.

That doesn’t mean that there’s anything wrong with the engine–it just means that the overall vehicle design and operations have to account for the new characteristics.