And how much should it cost? Over at my Pajamas Media piece this weekend, frequent TTM commenter “bbbeard” comments:
SpaceX has a launch record of 3 complete failures and two successes. What is disturbing about the SpaceX failures is that they hinged on relatively major oversights. Take the Demo2 flight, for example. SpaceX’s post-flight analysis showed that incorrect propellant utilization parameters were uploaded into the engine computer, a textbook case of sloppy configuration control. There was a recontact during staging, which initiated a slosh event — that was not mitigated because the LOX tank had no baffles. These are the kind of rookie mistakes that get you labeled as a “hobbyist”. It will take more than two successful flights to show that Elon Musk’s company has outgrown its hobbyist mentality and is ready to tackle human spaceflight.
Safety is the elephant in the foyer that you have not addressed. STS has suffered two launch failures in 132 missions (counting Columbia’s foam strike as a launch failure) — and what no one in NewSpace seems able to admit is that that loss rate is unacceptable. You can deny all you want that NASA is up to the job of designing a vehicle significantly safer than STS, but it is a fact that Ares is being designed to tough and unprecedented requirements for loss of crew rates — and Atlas and Delta never were. You claim Atlas has an “unbroken string of many dozens of successful flights” but by my count only 20 of the 21 flights of Atlas V have been successful — and that is an unacceptable loss rate. Only 2 out 3 Delta IV-Heavy flights have been successful — and that is an unacceptable loss rate.
Unlike SpaceX, the engineers at Boeing and Lockheed are the best in the business. But they were never directed to make Atlas and Delta reliable enough for human spaceflight. Using those platforms as human launch vehicles would be a step backward from STS safety levels, which are already unacceptably high.
What your argument boils down to is that you, Rand Simberg, think that the extra reliability that Ares aspires to is not worth the price tag. You may be right, you may be wrong. But why won’t you explain that that is your argument, instead of simplistically blaming NASA for poor cost control?
Man, there’s a lot to unpack there. I don’t know if I have time to deal with it right now, but let me at least lay out the issues. One is what an “acceptable” level of safety is (particularly relative to the reliability required to deliver a satellite worth a billion dollars). Another is how it is achieved. A third is how much it should cost to do so. A fourth is how much someone who had pretty much the same experience as other “professionals” in developing rockets for the first time can be said to be a “hobbyist.” (I would note as an aside that I don’t intrinsically accept “hobbyist” and “amateur” as pejoratives vis a vis “professionals” — many amateurs and hobbyists can be better than professionals — they just don’t choose to do it for a living. Space historian Henry Spencer comes to mind. I don’t think that there is anyone on the planet who is more familiar with both space history and space technology than Henry, but it’s not his day job.)
Anyway, I’m trying to figure out how to earn a living myself, so have at it in comments for now. I may weigh in later.
2 failures in 132 missions. Probability of failure 1 in 66. STS 25 (Challenger) Probability of failure 1 in 25. So, Shuttle is currently more than twice as “reliable” as it was in the early years.
All that SpaceX has to do is get better than 1 failure in 66 manned flights and it will be better than anything NASA has achieved since Mercury and Gemini.
I know this is a very simplistic approach to a complex problem but when the numbers are boiled down to a couple of facts then we begin to see that the new guys don’t have to be worried about being at least as good – if not better – than NASA. NASA has Manned Spacecraft Design Documents and they will be followed and possibly improved upon by the commercial people.
As for Boeing and Lockheed having better engineers – it just is not true. What probably is true is that SpaceX engineers get to engineer rather than attend interminable meetings and turn into bureaucrats along the way.
Two thoughts:
1) Some of these sample sizes are too small to use statistically. 21 launches is at the borderline – 2 or 3 is useless.
2) New development always has a higher error / failure rate then running an existing program.
Thanks for the recognition, Rand. 😉
Can someone explain to me why Falcon I’s LOX tank had no slosh baffles? I’m not focusing on this to be argumentative — I honestly don’t know, and my contacts at NASA have been unable to garner any information about this issue. I tried contacting SpaceX directly but got the expected PR runaround and was never put in touch with any technical folks….
BBB
Can someone explain to me why Falcon I’s LOX tank had no slosh baffles?
Likely because they add weight and cost, and they didn’t think that they needed them prior to the flight in which the problems occurred. There are two ways to find out — to simulate (and hope that the sim is accurate), or to fly. Their costs are sufficiently low that they are learning by flying (one of the reason that I have no problems with failed test flights). The Falcon 1 was their “training rocket,” and they’ve incorporated a lot of lessons learned into Falcon 9.
Bernard,
There is some discussion regarding this issue in the updates archives on the Spacex website, in the March 27, 2007 update posting:
http://www.spacex.com/updates_archive.php?page=0107-0707
SpaceX has a launch record of 3 complete failures and two successes.
The entire point of the F1 was to learn by doing. It has been a fantastic success. They have close to a thousand people that have benefited from this approach. They are learning fast.
rookie mistakes
Definition of rookie: Everybody when they start out. They are rapidly leaving the rookie stage behind.
Unlike SpaceX, the engineers at Boeing and Lockheed are the best in the business.
ROFLMAO. My stepfather was a supervisor at Boeing in Seattle since the 1960s and moved to Lockheed in Palmdale later. I assure you, lots of rookie mistakes are made in those two organizations as well.
The Merlin is a pretty good accomplishment for a bunch of rookies. The list is long and growing.
I agree completely with Rand’s comment above.
Thanks, Rand; don’t mind if I do.
First, let’s be clear on two distinct definitions of “failure” which bbbeard quite unreasonably conflates. The Challenger and Columbia disasters were loss of crew (LOC) failures. These are not directly comparable to loss of mission (LOM) failures. Apollo 13 was an LOM failure. Key difference? Presence or absence of corpses.
As pointed out, Atlas, Delta and SpaceX have all had LOM failures. Would these also have been LOC failures had there been human crew aboard? This would depend upon what assumptions one makes about design of crew vehicle and launch escape systems (LES). Mercury and Apollo vehicles used discardable tower-tractor LES’s. Ares I/Constellation was to do likewise. Gemini used ejection seats. Shuttles had ejection seats for pilot and co-pilot on, I believe, the first two missions, but no LES of any kind thereafter.
It’s worth noting that even LES-equipped NASA manned vehicles did not have LES capability all the way to orbit. I don’t know what the maximum speed and altitude limits for ejection seats were for Gemini and early Shuttle missions, but the Mercury and Apollo tower LES systems had to be discarded fairly early in the second stage burns of their respective vehicles.
SpaceX’s first LOM was a low-altitude first stage failure. A classic tower-tractor LES would have prevented such an LOM failure from also being an LOC failure had there been a manned vehicle in the picture. The next two Falcon I LOM’s were due to failures at much higher altitudes, but a classic LES would probably still have saved the day had any humans been at risk, especially since the failures involved no explosions. I have no knowledge of the details of the few LOM failures that have struck the Atlas and Delta series boosters, but I suspect that even a classic NASA-style LES would have prevented all or most of them from also being LOC failures had there been people and not transponders at risk.
It is my current understanding that the LES SpaceX intends for its Dragon vehicle is integral and non-discardable, providing a rapid separation from booster capability all the way from ignition to orbit; a capability never previously built into any manned NASA vehicle. I don’t think it’s at all a stretch to assert that, on paper, SpaceX’s Dragon is safer than any NASA design, including Ares I/Constellation.
Thanks for the link, John. This seems to be the key graf at a quick read:
These seem to me to be things you’d only find during flight, and wouldn’t necessarily show up in a sim. The only thing that concerns me is that if they hadn’t discovered the recontact/slosh issue on that flight, they might have had a successful flight, and not realized the need for baffles for many flights until it happened to bite them again.
In addition to the two LOC failures that NASA had with the Shuttles, they had some LOM failures as well. For example, STS-80 was a partial success (deployed and retrieved a couple scientific satellites) but the planned spacewalks were canceled when the airlock door couldn’t be opened. It turns out the airlock door was jammed by a loose screw. Was that a rookie mistake? Likewise, some of the early satellite deployments (pre Challenger) failed because the PAM didn’t ignite. A subsequent mission was able to retrieve two satellites and return them to Earth but you have to consider the original launch a failure.
The first Delta IV Heavy achieved orbit but it was lower than desired. Two microsatellites were released too low and burned in while the test payload ended up in a lower than desired orbit. You can certainly count that as a failure although if it had’ve been a live payload, it might’ve been able to be boosted to the proper orbit using the satellite’s own propulsion system. It might be more accurate to describe this as a partial failure. Had it been carrying a human crew, they would’ve survived. IIRC, the problem was due to a faulty propellant sensor causing the booster’s first stage engines to shut down prematurely. The problem was fixed and two live (and expensive) payloads were launched successfully.
The Atlas V also had a partial failure in 2007 when the Centaur upper stage shut down prematurely, leaving the NRO payload in a lower than desired orbit. That problem has also been fixed.
The key thing in these EELV failures as well as those by SpaceX is that they identified the causes and corrected them. Before Challenger, more than one Shuttle launch showed leakage around the SRB joints but the problem wasn’t fixed until a crew was lost. Likewise, foam impacts were a fairly common happenstance before Columbia but NASA didn’t seem too concerned. In that regard, NASA seems more reactive than proactive to flight risks than the commercial guys. After all, why should they care? How many NASA managers were fired following the Challenger and Columbia accidents? It’s not like there was any accountability or anything.
Another issue to add to the ones Mr. Simberg posed: what, exactly, is our basis for thinking that Ares I will be more reliable than the Falcon 9? Or EELVs, or the Shuttle?
The answer, as far as I can tell, seems to be “Projections made by people whose jobs depend on there being an Ares I program”.
Hmmmmmm…
My beef (if you will) with the whole safety issue is the fundamental assumption that any LOC event is unacceptable. If you apply that same concept to your daily life, you would never leave the house – too dangerous.
I don’t believe that it’s possible to completely eliminate LOC events in any stage of manned spaceflight. Does that mean we shouldn’t try? Of course not. It means we should take reasonable and prudent precautions – not obsess about it.
If safety trumps everything, why do HSF at all?
Dick-
Pretty good summary of LES needs. The primary need for an LES, IIRC, is the near zero/zero (speed/altitude) abort. According to the Apollo 11 flight transcript (http://history.nasa.gov/ap11fj/01launch.htm), the tower was jettisoned right after stage 1 separation, at 3:17. I’ve never heard discussions of ‘black zones’ in Apollo, but it was my understanding that, at that altitude, second (or third) stage engines could be shut down and the next stage (third stage or service module) would move the capsule away.
I’m quite excited to see SpaceX’s approach using their capsule fuel. Cutting down on the parasitic weight of the LES is a good step in the right direction.
Just saw Ironman 2 which was not a particularly good movie. It had two cameos… one of Elon and another that looked like the Hawthorne facility.
Thanks, John, good to know you’re lurking here…. I had previously read the page you linked. It left me wondering what tools SpaceX used to do their trajectory sims, why they thought those tools were adequate, what real-world check cases they validated the code against, what scale-model tests were conducted, what they thought their viscous damping was, how many dispersed simulation cases they ran, what coverage at what consumer risk… as you know we wrestle with these issues ourselves, constantly, and not always successfully. But at least we’re aware of the issues.
Rand wrote: These seem to me to be things you’d only find during flight, and wouldn’t necessarily show up in a sim.
Well, beg to differ, at least on the slosh part of the story. An enormous amount of research on propellant slosh was conducted in the 1950’s and 60’s, initially prompted by high-profile losses of early launch vehicles, later by the necessity to develop design guidance for new vehicle programs. Check NASA SP-8031, CR 406, SP-8009, and of course SP-106.
Case in point: we have known for a long time that purely lateral slosh (i.e. side-to-side excitation) can produce rotary motion of the propellant, which can couple to rotary modes of the control system. We have also known for a long time that ring baffles almost completely eliminate this swirling mode. It seems to me that SpaceX was either unaware of this phenomenon, or chose to ignore it, and generally relied on incompetently crafted simulations to design Falcon I (so, yeah, it wouldn’t necessarily “show up in a sim” — garbage in, garbage out). It’s hard to tell for sure from the writeups I’ve seen, including the page John linked.
Anyway, a flight test seems like a really lazy and expensive way to avoid a literature search. And this is just one issue that led to one failure — how much does SpaceX not know about designing launch vehicles?
Shutdown transients are tricky to model. In hindsight SpaceX had not done enough ground testing to characterize the range of behaviors to be incorporated into the control system design criteria. Do we have confidence that they now have? Have they mastered the physics of the Merlin shutdown transient, or have they just been lucky for the last couple of flights? What if the Demo 2 transient was not an extraordinary event, but simply a “1.5 sigma” excursion? Do we even know if the tail of the distribution is exponential [which includes Gaussian] — or could it be a fat-tailed distribution?
I don’t want to make too strong a case here; I think there is much to admire about the strategy of rapid development and iterative low-cost flight test. I remarked on an earlier thread here about Ares I-X that an actual recontact failure would have been tremendously informative. Instead the actual flight fell within the envelope of pre-flight simulations, so in a sense the flight test changed nothing and merely confirmed our models without moving the limits of our ignorance.
But judging from what I’ve seen, it is foolhardy to assume that SpaceX is done with launch failures, that there were only three things they needed to fix before achieving acceptable launch reliability. Maybe I’m wrong. It they reach 50 launches without another failure I will buy you a Diet Coke.
BBB
larry j wrote: It turns out the airlock door was jammed by a loose screw. Was that a rookie mistake?
It seems pretty clear to me that complex systems suffer failures due to small mechanical details, well, all the time. A number of incidents on airlines have occurred due to improper maintenance of O-rings. An early F-16 crash was caused by the mistaken deletion of an anti-rotation pin from a backup engine control transfer valve. It’s really difficult to engineer these kinds of failures out of complex systems.
But… “gee, we loaded the wrong software into the engine control”? How long have aerospace engineers known that you have to have affirmative configuration controls on engine and vehicle software — that you can’t just walk up to an aerospace vehicle and upload any damn software you want? How about, “gee, we didn’t know we had to provide damping in the propellant tanks or the vehicle may become unstable at some point in the trajectory”?
So when I write about “rookie mistakes”, I don’t mean, “How many parts could be misinstalled or left out accidentally”, I mean, “How many deliberate design choices have been made on Falcon 1 and 9 in ignorance of the accumulated wisdom of 50 years of spaceflight?”
G Clark wrote: My beef (if you will) with the whole safety issue is the fundamental assumption that any LOC event is unacceptable. If you apply that same concept to your daily life, you would never leave the house – too dangerous.
It’s not true that we’re assuming that any LOC event is unacceptable. Ares I has an explicit requirement not-to-exceed 1 LOC in X launches, where X is a number much greater than 100. The design system explicitly recognizes that perfect safety is not only unattainable but uneconomical. You need to rethink your objection to this strategy.
Jeff Dougherty wrote: Another issue to add to the ones Mr. Simberg posed: what, exactly, is our basis for thinking that Ares I will be more reliable than the Falcon 9?
That’s a really good question. I don’t want to stray into sensitive subject matter, but I will say that the Ares design process is structured around trajectory Monte Carlo, and that design criteria for every piece of hardware are tied to the extreme values of the predictions of those flight simulation ensembles. And the process is iterative: from early trajectory MC, initial designs are formulated, changes are fed back into the TMC, new predictions are made, resign criteria are revised, etc. From my limited viewpoint, the process is extremely thorough and as mathematically (i.e. statistically) correct as we can make it.
I can’t say what the Falcon 9 team has done, except that they have obviously taken a different road to enlightenment.
BBB
It doesn’t take a “rookie” company to make design choices that don’t work out. How many airliners have we seem crash because of poor design margins, or bad designs (Air France Flight 447 pitot tube).
People will die going to & from space – it’s just a question of when. People will also die in space. Testing is supposed to find the obvious errors, and hopefully hint at the more hidden ones.
For SpaceX, they are contracted for 12 COTS deliveries with their Falcon 9 and Dragon. This will give them a much higher degree of confidence in their design if they choose to upgrade to crew services.
Also, having worked for “startups” and old guard aerospace companies, I can tell you that startups have their choice of the best people. All things being equal, would you rather work for an exciting new company that could change the way we access space, or for an old established aerospace company. I can almost guarantee you that SpaceX does not have the highest pay, and that they work long hours, and that they have 1,000 resumes for each open position they advertise.
Check NASA SP-8031, CR 406, SP-8009, and of course SP-106.
Heheh, I did that and ended up at http://www.bbbeard.org. I see you work at ARES Corporation. Hmm, who else do we know who has an association with ARES? 😉
bbbeard said:
“Well, beg to differ, at least on the slosh part of the story.”
I can’t find the exact reference but I do remember Musk at the time saying explicitly that they were aware of slosh as a potential issue and believed that their software was sufficient to damp any slosh. The contact between the second stage nozzle and the interstage was just too big for the software to overcome.
This prompted them to go back to adding baffle even though the software would be sufficient in normal cases. The return to baffles was more to add redundancy than because they believed the software slosh mitigation was inadequate.
They also discussed throttling down the merlin just before MECO as a way of damping the thrust transient, though whether they went ahead with this or not I don’t know as throttling down would be at the cost of some loss of engine life. An issue when you’re planning to reuse the vehicle.
The main point is there was a lot of work done in the wash up of flight 2 and reading the incident reports about it on the SpaceX web site makes it perfectly clear that the issues were worked through in considerable detail.
It’s why everyone has test flights.
The important point is they’ve learnt from their mistakes.
It was flights 1, 2 and 3 that failed. Not 4 and not 5.
http://www.spacex.com/F1-DemoFlight2-Flight-Review.pdf
is the full document
If LOC is accepted, then why do people make such a big deal of crew safety?
I’m not saying (and have never believed) we should just send crews up in any old thing. Obviously, there must be certain minimum standards. I want those standards to be uniformly applied – if it’s good enough for NASA, it’s good enough for LockMart, Boeing, Orbital, SpaceX, & etc.
I object to the idea that a Powerpoint rocket is superior to something that is already (or about to be) flying. I object to the idea that just because it’s relatively inexpensive, it is in some way inferior. I object to the idea that just because they don’t have a twenty-or-more-year flight heritage, they must be rookies. I object to even the very thought that some officious bureaucratic twit might attempt to force a double standard on the commercial providers.
(‘I’m mad as Hell, and I’m not going to take it any longer!’)
Now I’m going to go take my blood pressure meds.
That’s a really good question. I don’t want to stray into sensitive subject matter, but I will say that the Ares design process is structured around trajectory Monte Carlo, and that design criteria for every piece of hardware are tied to the extreme values of the predictions of those flight simulation ensembles. And the process is iterative: from early trajectory MC, initial designs are formulated, changes are fed back into the TMC, new predictions are made, resign criteria are revised, etc. From my limited viewpoint, the process is extremely thorough and as mathematically (i.e. statistically) correct as we can make it.
OTOH, SpaceX uses a process that we know works. I’ve heard other people make similar extravagant claims about this modeling approach and it seems to be the foundation of the claim that the Shuttle’s SRBs have a failure rate of something like 1 in 3700 rather than a factor of ten worse. All I know is that the SRBs have a demonstrated failure rate ten times worse, have only launched or been tested somewhere around 300 times, nobody is modeling the whole process including manufacture and launch prep, and that there’s no penalty for failure, if the modelers should turn out to be very wrong.
If LOC is accepted, then why do people make such a big deal of crew safety?
Well, to paraphrase an old joke, we know what we are, we’re just haggling about the price.
BBB
But… “gee, we loaded the wrong software into the engine control”? How long have aerospace engineers known that you have to have affirmative configuration controls on engine and vehicle software — that you can’t just walk up to an aerospace vehicle and upload any damn software you want? How about, “gee, we didn’t know we had to provide damping in the propellant tanks or the vehicle may become unstable at some point in the trajectory”?
If you’re holding up NASA as the standard for non-rookie organizations, you’ll have to look at the times when they’ve also had software issues with the Shuttle. I don’t have my copy of “Computer Related Risks” handy at the moment but NASA, despite being a SEI Level 5 organization, has had some software errors as well. I remember reading many years ago about one in particular that was discovered after the Shuttle had flown several times. This particular glitch was discovered in simulation where it was possible for the software to effectively prevent the crew from jettisoning the ET, leading to a LOC accident.
Also, Lockheed-Martin is far from a rookie launch company but they also had a very expensive LOM incident. The payload was flight 3 of the Milstar communications satellite (worth about a billion dollars).
1999 April 30 – USA 143 – Launch Site: Cape Canaveral. Launch Complex: LC40. Launch Vehicle: Titan. FAILURE: Centaur software programming error. Perigee: 1,097 km (681 mi). Apogee: 5,149 km (3,199 mi). Inclination: 28.20 deg.
The Titan core vehicle operated correctly, but a software error in the Centaur stage resulted in all three planned burns being made at the wrong times, during the first orbit instead of over a six hour period. The three burns planned to place Milstar successively in a 170 x 190 km parking orbit, a geostationary transfer orbit, and finally geosynchronous orbit. Instead, at 19:00 GMT, several hours before the scheduled third burn, Milstar separated into a useless 740 km x 5000 km orbit. Milstar-2 F1 was the first upgraded Milstar with an extra Medium Data Rate payload with a higher throughput. The payload included EHF (44 GHz), SHF (20 GHz) and UHF communications transponders and satellite-to-satellite crosslinks, with narrow beams to avoid jamming.
But judging from what I’ve seen, it is foolhardy to assume that SpaceX is done with launch failures, that there were only three things they needed to fix before achieving acceptable launch reliability.
Did I miss where someone assumed that? SpaceX is certainly making no such claim.
Elon has asserted just the opposite. Stating if the first F9 fails they are already building more and will continue despite failures.