Henry Spencer got it right (no big surprise):

The gap between engine cut off and staging was 1.5 seconds – which was fine for the ablatively cooled engine on Flight 2. But on Flight 3, with the regeneratively cooled engine, there was some residual thrust after engine shut down and this caused the first stage to be pushed back toward the second stage after separation and there was a recontact between the stages.

One of the big mistakes that people make in writing requirements is not writing proper verification statements for them. One of my rules, that I came to late in life, is to not allow a requirement to be accepted unless it has an accompanying verification statement (i.e., how you verify that the requirement has been satisfied). If you can’t write a verification statement for it, it’s not a valid requirement. The other reason is that verification is where most of the cost of a program comes from. Test is very expensive. If you can come up with ways to verify early on that don’t require it (inspection, demonstration, analysis), you can control and estimate costs much better.

One of the key elements of a proper verification statement is the environment. It’s not enough to say, “Verify, by test, that engine thrust is less than TBD Nt TBD seconds after engine shutdown.” It has to be “Verify, by test, in vacuum, that engine thrust is less than TBD Nt TBD seconds after engine shutdown.”

AMROC had a similar problem on SET-1 back in 1989, because the propulsion system testing had all taken place in the desert at Edwards, and the actual launch occurred in the humid October weather of Vandenberg, at the coast. The LOX valve iced up. The vehicle ended up catching fire and fell over and burned on the pad. There was no explosion, but it was a launch failure.

This is why systems engineering processes were developed. I’d be curious to know what kind of SE processes SpaceX had in place. And what they’ll have in place in the future…

[Late evening update]

Here’s the official statement from Elon Musk:

Timing is Everything

On August 2nd, Falcon 1 executed a picture perfect first stage flight, ultimately reaching an altitude of 217 km, but encountered a problem just after stage separation that prevented the second stage from reaching orbit. At this point, we are certain as to the origin of the problem. Four methods of analysis – vehicle inertial measurement, chamber pressure, onboard video and a simple physics free body calculation – all give the same answer.

The problem arose due to the longer thrust decay transient of our new Merlin 1C regeneratively cooled engine, as compared to the prior flight that used our old Merlin 1A ablatively cooled engine. Unlike the ablative engine, the regen engine had unburned fuel in the cooling channels and manifold that combined with a small amount of residual oxygen to produce a small thrust that was just enough to overcome the stage separation pusher impulse.

We were aware of and had allowed for a thrust transient, but did not expect it to last that long. As it turned out, a very small increase in the time between commanding main engine shutdown and stage separation would have been enough to save the mission.

The question then is why didn’t we catch this issue? Unfortunately, the engine chamber pressure is so low for this transient thrust — only about 10 psi — that it barely registered on our ground test stand in Texas where ambient pressure is 14.5 psi. However, in vacuum that 10 psi chamber pressure produced enough thrust to cause the first stage to recontact the second stage.

It looks like we may have flight four on the launch pad as soon as next month. The long gap between flight two and three was mainly due to the Merlin 1C regen engine development, but there are no technology upgrades between flight three and four.

Good Things About This Flight

• Merlin 1C and overall first stage performance was excellent
• The stage separation system worked properly, in that all bolts fired and the pneumatic pushers delivered the correct impulse
• Second stage ignited and achieved nominal chamber pressure
• Fairing separated correctly
• We discovered this transient problem on Falcon 1 rather than Falcon 9
• Rocket stages were integrated, rolled out and launched in seven days
• Neither the near miss potential failures of flight two nor any new ones
  were present

The only untested portion of flight is whether or not we have solved the main problem of flight two, where the control system coupled with the slosh modes of the liquid oxygen tank. Given the addition of slosh baffles and significant improvements to the control logic, I feel confident that this will not be an issue for the upcoming flight four.

Elon

About SpaceX:

If the problem is confirmed to be a simple and easily fixed design flaw, they may not launch again “tomorrow” but I wouldn’t be too surprised if there was another flight within a couple of months.

…Lost in the hubbub over the flight failure was the fact that once again they were able to do a quick resumption of the launch procedure after a hot-fire abort. This sort of robustness in the launch operations and the use of small crews are crucial factors in lowering the cost of launch.

And as noted, the new Merlin apparently performed well. Had it not, that would have been a real setback for both Falcon 1 and 9.

[Update after lunch, Pacific Time]

Henry Spencer has more thoughts, with some history.

It seems quite likely that it was caused by the new engine–that’s the only thing that changed between the last flight and this one, and Henry points out a couple potential plausible scenarios for that.

That doesn’t mean that there’s anything wrong with the engine–it just means that the overall vehicle design and operations have to account for the new characteristics.

It’s looking like there was a second-stage problem, either separation, or ignition (or both, since one could cause the other).

As I said last night, this is obviously a disappointment to the SpaceX team. Particularly since they had previously had a flight where this wasn’t a problem, so in a sense it was two steps forward, one step back. I think that at this point, almost anyone is going to be pretty leery of putting a payload on the vehicle until it’s had at least one successful flight. Is it the end? Despite what Elon said a long time ago about three strikes, it’s hard to see it now. He’s fully invested now, both financially and (I would imagine) emotionally, and he’s not going to come this far just to give up, particularly when tantalized by his previous almost-success on the second flight.

They’ll go through the telemetry, figure out as best they can what happened, and try again, and hopefully soon. In a sense, as someone noted in comments in the earlier post, Falcon 1 is really a test program for the bigger vehicles, though they should get an operational small launcher out of it as well.

As always, this points up the problem with expendable vehicles. They are very expensive to flight test, so you can’t afford to do very many, and every flight is a first flight, so you can’t wring bugs out of a vehicle with incremental testing. And it’s a lot harder to figure out what went wrong because you generally don’t get much debris to analyze (the first flight that failed off the pad was a rare exception)–you have to dig through electronic entrails. And NASA, of course, in its cargo-cult determination to redo Apollo, is taking exactly the same expensive and unreliable approach.

And just checking now, I see that Clark is having similar thoughts to mine.

Once the problem that caused this failure is determined, I would suggest that SpaceX just bite the bullet and allocate 2 or 3 Falcon I vehicles for test flights and fly them within a relatively short period, say six months.

This would represent a $20M-$30M investment but until the Falcon I is flying reliably, SpaceX will find it very difficult to get any more commercial or government payload contracts and it won’t have any chance of getting COTS D (ISS crew transport) funding. The Falcon 9 is a completely different vehicle but the Falcon I is what currently defines the company’s ability, or inability, to deliver what it says it can.

Anyway, best of luck to them in the future, but they know that they need more than just luck.

[Update a couple minutes later]

I see that Elon has a statement, which confirms my suspicion above:

There should be absolutely zero question that SpaceX will prevail in reaching orbit and demonstrating reliable space transport. For my part, I will never give up and I mean never.

That’s the kind of attitude you have to have, even if eventually, you do in fact have to give up. I hope he won’t have to.

Also note Clark’s comment at the end of the post, that SpaceX is following in the tradition of all expendable staged launch vehicles in its failure modes, though they do seem to be getting the avionics right.

Falcon 1 goes up at 4 PM Pacific Time. That’s 7 PM for me, and we already have tickets purchased for Dark Knight, so I guess we’ll miss it, if it goes on schedule. I’ll have to watch the replay.

[Update at 10:30 PM EDT]

Back from the movie, which was very good. Ledger can certainly expect a posthumous Oscar nomination.

There have been launch delays, but they’re currently reloading fuel after having drained it (there was apparently concern that it was getting too cold during other delays) and are now expecting a launch at 11 PM EDT (8 PM Pacific), in almost exactly half an hour.

[Update a couple minutes later]

They must plan for an 8:05 liftoff, based on the count I just heard. T-32 and counting at 7:33. Weather is green, though there’s some cloud cover.

[Update about ten till the hour]

There must be a delay or something on the web feed, because they’re still saying it will be an 8 PM PDT launch, even though their count makes it come out three or four minutes after that. I wonder if there will be a transmission delay on the launch itself of a couple minutes? If so it won’t quite be live, but it will be close enough.

[Update shortly after scheduled launch time]

They had a (literally) last-minute abort. The window closes in an hour, and I doubt they can turn it around that fast, since they still have to look at the data to figure out what happened. Better luck tomorrow.

[Update a couple minutes later]

That was fast. Now they’re saying they think they may be able to recycle from T-10, so it still may be on tonight.

[10:30 EDT update]

Now they’re at T-7 and counting again.

[Update shortly after launch]

Uh oh. Sounds like strike three. The picture was lost at about 35 km altitude and a thousand meters/sec. They announced an “anomaly.” That doesn’t sound good. The last update on the site was that it was about to enter inertial guidance (not clear what they were doing prior to that). Did something go wrong with an IMU, or some other part of the GN&C?

Fortunately, you’re allowed more than three strikes in this game. It has to be a huge disappointment, though, unless the anomaly was merely a loss of signal, and the vehicle’s doing all right. The webcast is over, though. I think that I’d assume that the news is bad.