Is this comment from the article true:
“… kerosene is heavier than hydrogen for the same amount of energy, ….”
Having trouble believing it from a strictly chemical sense. I could believe “more Isp” or “more delV” per pound, but total energy?
Since carbon is heavier than hydrogen it seems reasonable that there would be more bonds per gram.
No, I’m flat wrong. On a per-mass basis it’s just ridiculous. Per mole is another story – but pointless. Latent heat of vaporization not going to cover that large a difference.
Rand: you may already know this, but if you look at particular categories of your blog (like “Space”), you don’t see active links in the stories. This post, for instance, just says “SpaceX Thoughts …from Ric Locke” with no html link to Ric’s article.
I only bookmark the Space category, but that means that following your links always requires an extra click.
Actually, I wasn’t aware of that. Not much I can do about it without hacking WordPress.
Ric sez: According to the European Space Agency, the Soyuz launch vehicle is the most frequently used and most reliable launch vehicle in the world. Folks, that’s what you call a “track record”. It makes sense to build on and improve what’s been demonstrated to work, no?
The Soyuz has about a 3% launch failure rate. This is actually better than the average for all launch vehicles, but not better than Shuttle, which is half of that, counting Columbia as a launch failure. By Ric’s reasoning, we should be evolving the Shuttle.
Ah, but the Shuttle is more expensive than Soyuz, no doubt. And ARES would have continued that tradition, intending on pushing failure rates well below a percent (1/500 was the design requirement) but with higher launch cost.
And that’s one source of my irritation with SpaceX fans: you are willing to tolerate a 3% failure rate if it will lower launch costs. You don’t see the value in cutting failure rate in half if it triples the cost. You have a magical belief that if you just fly crappy rockets long enough with low cost as the primary objective, they will get better and better as you fix the things that break them, and that eventually the failure rate will be low enough to be acceptable. Well, look, the Russians have over a 1000 Soyuz launches, and the empirical failure rate is still over a percent (if you want to ignore earlier failures). Meanwhile, Shuttle is being retired in large part because 1.5% is too high.
It’s a gross oversimplfication to say that Shuttle is being retired because its failure rate is too high. For instance, if we had a couple dozen orbiters, and the ops costs were an order of magnitude lower, it would probably still be flying, even with its current reliability.
I understand completely that it is your position that launch costs are the sine qua non of launch vehicle viability, but if you actually listen to the people who make the decisions, you will find that safety is a primary consideration. Whenever Mike Griffin was asked why we couldn’t just continue operating the Shuttle, his response had to do with safety, not launch costs. And you’re well aware of that, since that is a constant refrain in your criticism of NASA.
Mike Griffin did not make the decision to cancel the Shuttle.
Fly crappy rockets long enough? Who said anything about wanting to fly crappy rockets? bbbeard, do you think Falcon 9 is a “crappy rocket”?
And that’s one source of my irritation with SpaceX fans: you are willing to tolerate a 3% failure rate if it will lower launch costs. You don’t see the value in cutting failure rate in half if it triples the cost. You have a magical belief that if you just fly crappy rockets long enough with low cost as the primary objective, they will get better and better as you fix the things that break them, and that eventually the failure rate will be low enough to be acceptable.
What a nice set of strawmen you’ve assembled there. Tell us, who among us SpaceX fans are saying any of those things you just claimed? Anyone? Anyone? Who among us has said we’d be willing to tolerate a 3% loss rate?
SpaceX is a tightly vertically integrated company. They produce just about everything on their boosters and capsule. As a SpaceX employee told me yesterday, “If there’s a problem, we know it’s our fault. There is no finger pointing among different subcontractors. It’s our fault so we move quickly to fix it.” Their track record to date proves that is true. Furthermore, they’ve built features into their boosters to increase reliability because they know that losing a lot of rockets isn’t a good business strategy to woo future customers. Yes, they lost 3 Falcon 1 rockets in a row. In each case, the cause of the problem was quickly determined and fixed. None of those problems have been repeated and they’ve had 4 successful launches in a row (two each of the Falcon 1 and Falcon 9). They have a long way to go before they establish a definitive track record but they’re working on it.
Consider their base Falcon 9. It was designed to be able to survive losing an engine right at liftoff and still fly the mission. Both the Atlas V and Delta IV core designs are single engined. That means any loss of an engine (or even significant loss of thrust) will result in loss of mission and payload. The result of those design choices is that ULA has to use very high quality and very expensive engines. SpaceX, by way of contrast, built their engines to be simple, reliable, and inexpensive to manufacture. According to that SpaceX employee I was talking to, they also designed them to be reusable from 20-40 times even after being exposed to salt water. If they can solve their stage recovery problems, that will further lower their costs.
Wait, my belief that people become more successful with more practice is magical? I would have called it “obvious”, certainly, “supported by all available data in a wide variety of fields”, I suppose, and I’d even have been egotistical enough to say “rational”, but magical too? Neat! From now on I’m going to call everything I write a “spell”.
@Rand: yes, but Griffin was central to the decision to keep it dead when Congress wanted to extend the life of the Shuttle.
@Cecil: Who said anything about wanting to fly crappy rockets? — That is what I infer from Ric Locke’s blog post, which is the topic of this thread. Ric’s post is in part a paean to the workhorse of the Soviet rocket fleet. He quotes a journalist quoting ESA quoting Russia that the Voskhod/Molniya/Soyuz is “the most reliable launch vehicle in the world” [sic]. Ric thinks so much of this reasoning that he wants SpaceX to emulate Russian design practice. You fill in the blanks.
@Larry: Who among us has said we’d be willing to tolerate a 3% loss rate? — Again, Ric has held up the 1700-flight, 3% loss rate of the Soyuz as a paradigm for rocket development. Except that he left out the “3%” part. Because it sounds, you know, unacceptable.
“we move quickly to fix it…. they lost 3 Falcon 1 rockets in a row. In each case, the cause of the problem was quickly determined and fixed. “ — Have they considered designing the rocket correctly in the first place? I’ve mentioned this before and still have not gotten an answer: why was Falcon I not designed with LOX tank baffles in the first place? We’ve known for almost 50 years that you have to suppress propellant slosh. How did SpaceX design and launch without this knowledge? How many more “lessons learned” will they have to relearn?
None of those problems have been repeated and they’ve had 4 successful launches in a row (two each of the Falcon 1 and Falcon 9). They have a long way to go before they establish a definitive track record but they’re working on it. — IIRC Falcon I flights 2 and 3 were both failures that involved recontact during staging. I would like to know what process they went through to design the staging system. Is this another case of “let’s fly and see what breaks”? In the case of flight 3, thrust tailoff has long been known to be a major source of concern for staging. How much consideration did SpaceX give to this? How did they process the Merlin engine test data and convert it to requirements for the staging system? I have no confidence that SpaceX knows what they are doing. Right now their rockets are “software quality”, not “aerospace quality”, which is to say, they’re cheap, they work most of the time, and they are undergoing a continuous process of upgrading. Which befits the founder of PayPal.
four successful launches in a row — Granted, that is way better than four losses in a row, which is what we saw the last time someone put 30 engines on the first stage of a launch vehicle. But you have to understand that an engineer will look at “0 losses in 2 flights” and get out the binomial tables. It turns out that with zero losses in two flights you have 50% confidence that the true failure rate is less than… drumroll… 29.3%. Now, that’s better than the 79% failure rate at 50% confidence that we see with the Taurus XL (2 failures out of 3). And while any successful launch is a reason to pop the champagne corks, I’m not ready to buy yet.
Just for reference, call me when the Falcon 9 gets to seven launches without failure. That’s when the 50% confidence empirical failure rate drops below the 10% mark, which is approximately the historical failure rate. And I will buy each of you a Coca-cola if Falcon 9 makes it to 34 launches with no failures, when its 50% confidence failure rate drops below Shuttle’s.
@Roy: Wait, my belief that people become more successful with more practice is magical? — The question isn’t whether the design gets better as it is exercised, it’s whether it gets better enough, and whether that process is better than designing the rocket correcly in the first place. The Soyuz has been evolving for four decades and has been launched 1700 times. When will it evolve to a rocket with less than 1% failure rate?
bbbeard, rockets will fail, a capsule design gives the crew a better chance of survival than the shuttle design so if spacex can get their failure rate down to 1-2% or better, and you’ve offered no evidence to the contrary, given the choice, I’d rather be launched in a dragon than the shuttle, even ignoring the extra cost of the shuttle.
Have they considered designing the rocket correctly in the first place?
Oh, you mean like the Delta III that had a 33% success rate? It was “designed right” by professional engineers and it still had failures. So did the Ariene V and most other booster families in the history of spaceflight. There is such a thing as a learning curve with any new design.
IIRC Falcon I flights 2 and 3 were both failures that involved recontact during staging.
IIRC, Flight 2’s first stage brushed against the second stage’s nozzle at separation. However, the coning that lead to premature upper stage engine shut-down didn’t being until minutes later.
On Flight 3, there was definite bumping between the stages at separation. This was caused by an unexpected level of residual thrust from the first stage engine. On that flight, they switched from a radiantly cooled Merlin to a regeneratively cooled design. They had no facilities to test the residual thrust so they didn’t anticipate it. The only fix required for that was to wait a few additional seconds after first stage engine shut-down before separating the stages. It worked perfectly on all subsequent flights.
Time will tell about SpaceX’s safety record. The Shuttle had a seemingly perfect safety record until the Challenger accident. No manned spacecraft had a heat shield failure until Columbia. Those accidents happened despite billions being spent on R&D and additional billions being spent each year for Shuttle operations. Sometimes, spending a lot of money on something only results in an expensive rhing, not necessarily a safe or good thing. You don’t always get what you pay for. Conversely, an inexpensive thing designed and built well may be more reliable than an expensive thing. Guess which car ends up in the repair shop more often, a Honda Civic or a Ferrari?
@Andrew: if spacex can get their failure rate down to 1-2% or better, and you’ve offered no evidence to the contrary…
@Larry: There is such a thing as a learning curve with any new design.
Note the latter includes all seven SpaceX launches. It might be fairer to have separate Falcon 1 and Falcon 9 curves. I’ll leave that as an exercise.
It might be fairer to have separate Falcon 1 and Falcon 9 curves.
Ya think?
@Larry: Conversely, an inexpensive thing designed and built well may be more reliable than an expensive thing. — I think the point is whether you make safety a design requirement, and whether you have the institutional capability to meet that requirement. A Volvo S40 is safer than the Mustang GT that I drive, despite being cheaper, in large part because they emphasize safety over performance, and have for a long time. SpaceX claims to have high reliability (not safety per se, as they are unmanned) as a design goal but so far it is not showing up in the numbers or in their design practices.
@Rand: Ya think? — I’m just being fair, as always. ;-)…. I think there is an argument either way. I think the Falcon I failures were the byproduct of an misguided design process. It is not clear at all that the design process has been fixed, even as individual design decisions have been corrected. Did a different group of engineers design the Falcon 9? Did they use a predictive, rather than reactive, design process? If so, there is an argument for separating the two datasets. On the other hand, both rockets use the same engine, and presumably some other hardware, so the launch failure probabilities are not independent no matter what argument you make.
SpaceX claims to have high reliability (not safety per se, as they are unmanned) as a design goal but so far it is not showing up in the numbers or in their design practices.
The same can be said for NASA. When they were building the Shuttles back in the 1970s, they claimed it would be extremely safe (IIRC, something like a LOC of 1:1000). Despite all the best design practices and many billions of dollars, they lost two vehicles and 14 astronauts in less than 130 launches. It doesn’t exactly sound like they met their goals, does it? So, why are you holding a private company to higher standards than what NASA was able to achieve?
@Larry: So, why are you holding a private company to higher standards than what NASA was able to achieve?
Because it’s not 1975 anymore.
SpaceX seems to be heading down a path of rediscovering all the possible ways one can fail a launch. They seem charmingly devoid of any institutional memory. It doesn’t bother them that the mistakes they’ve made could have been avoided because we’ve known about these failure modes for decades. Shuttle is at 1-2% right now, and that’s not good enough to keep flying people beyond a couple of more flights. SpaceX is at 43%, or 0%, depending on who’s being asked — the numbers are too small to judge. But the way they’ve failed, and their stated design philosophy, which brings to mind Mickey Rooney saying “Let’s put on a show!”, brings up serious questions.
But they’re cheap, I’ll give them that.
And I don’t mean to give NASA a pass. There are some serious process issues at NASA — including a rapidly diminishing institutional memory! But this thread is ostensibly about Ric Locke’s gushing encomium for SpaceX. If you want to change topics, just say the magic words (“You’re right, Bernard! Hey, want to talk about NASA?”) 😉
SpaceX seems to be heading down a path of rediscovering all the possible ways one can fail a launch.
That seems very unlikely. I think you’re being hyperbolic.
They seem charmingly devoid of any institutional memory. It doesn’t bother them that the mistakes they’ve made could have been avoided because we’ve known about these failure modes for decades.
Are you unaware that the first thing that they did before initiating design activities was to hire Futron to do a study for them on exactly this topic?
@Rand: I think you’re being hyperbolic.
And you, sir, are being elliptical!
Are you unaware that the first thing that they did before initiating design activities was to hire Futron to do a study for them on exactly this topic?
Yes, I’ve read the Futron study before. Are you aware that the study deals only with subsystem failures and says nothing about my main point — that the design process at SpaceX is defective?
More to the point, are you aware that the Futron study only covers the two decades prior to 2004? Do you understand that prior to 1984 there were multiple launch failures due to slosh problems? That we figured out how to mitigate this risk? That the lessons learned from those losses and the research that followed are the main reason slosh-related failures are less frequent now? Well, they were until SpaceX came along and Futron decided that launch vehicle history started in 1984! Can you spot the root cause of the Demo 2 failure?
In any case, are you seriously suggesting that a four-page white paper from a “decision management solution” think tank is a substitute for a proactive design process?
In any case, are you seriously suggesting that a four-page white paper from a “decision management solution” think tank is a substitute for a proactive design process?
No, but then, the notion that the “design process is defective” is your opinion, not a fact.
Did they use a predictive, rather than reactive, design process?
All design is predictive; All fixes are reactive. There is such a thing as diminishing returns. The safest rocket is in a museum.
I’d be happy to take the place of that cheese wheel on the next flight.
No amount of memory is going to help you anticipate all future issues. Also, if they added all the safety feature of every rocket (isn’t that your suggestion???) no amount of thrust would get it off the ground.
They’re now planning a three stage rocket where two of the stages get held before launch (with abort potential) and only one engine isn’t.
Dragon will have over a dozen missions before any people go up. NASA would go on second launch (who are the amateurs?)
There is such a thing as remembering the wrong lessons.
@ken: All design is predictive
Yes, but some predictions are better than others. Predictions based on test and analysis with a sound statistical characterization of uncertainties are better than formulas picked out of an undergraduate textbook. Predictions based on enumeration and re-examination of known failure modes is better than design based on a willful disregard of risks and a forlorn hope that ignorance will be not fatal.
Also, if they added all the safety feature of every rocket (isn’t that your suggestion???) no amount of thrust would get it off the ground.
The standard of practice is to designate acceptable levels of risk and to design to them using carefully selected measures of statistical significance. Is there any evidence that SpaceX takes this approach? I would say, not hardly… because the available evidence suggests they barely know what the problems are.
No amount of memory is going to help you anticipate all future issues.
Right. So let’s not even try.
We’re at the point in aerospace history that we think we can design manned launch vehicles with smaller than 1% failure rates (with design requirements tighter than that). But if we choose to throw away everything we’ve learned in 50 years of space travel (or at least in the first 25 years of space travel, if you put your stock in the Futron approach), I can state with some confidence that we won’t get there.
NASA would go on second launch
Not really the second launch, but too soon, in my view as well. Like I said, if you want to change topics, just say the magic words (“You’re right, Bernard! Hey, want to talk about NASA?”)
…design based on a willful disregard of risks and a forlorn hope that ignorance will be not fatal.
…choose to throw away everything we’ve learned in 50 years of space travel…
Bernard, I see that you continue your unsubstantiated hyperbole.
@Rand: Bernard, I see that you continue your unsubstantiated hyperbole.
Okay, I admit it, I’m being hyperbolic. They’re only willing to throw away half of what we’ve learned.
But I keep returning to the substantiation and you keep ignoring it. Why were there no baffles in the LOX tank of Falcon 1? Why was the Merlin thrust tailoff not designed for? What are the grounds for believing that a 27-engine first stage is reliable enough for critical missions?
Oh, and — did SpaceX not know you have to tumble boosters if you want to recover them?
Why were there no baffles in the LOX tank of Falcon 1?
I’m going to hazard a guess that a single individual overrode his engineers who told him they were needed, and that he has since learned not to do that. Just a guess, though.
Why was the Merlin thrust tailoff not designed for?
Because they didn’t know that the tailoff of the different (regenerative) nozzle would be different, because no ground-test rig would have the senstivity to tell them what it was…? Sorry, but sometimes, you can only learn things in actual flight. And even when you learn it that hard way, it’s still a hell of a lot cheaper than analysis, which may or not be valid, and months of delay.
What are the grounds for believing that a 27-engine first stage is reliable enough for critical missions?
What are the grounds for believing that it isn’t enough, given engine-out capability? I would point out that they are going to quickly gain a lot of data on engine reliability from that design, given how many engines they’ll be testing each flight.
…did SpaceX not know you have to tumble boosters if you want to recover them?
Why do you think that a) that is a truism and b) they don’t know that?
Do you have ESP, or a ouija board?
@Rand: I’m going to hazard a guess
Seriously, why don’t you ask Elon next time you have a friendly moment? I’m still in the dark after several years of inquiry. What you said would be my guess, too, but if it were the case, it would be a sharp indictment of their design process. Which is probably why we still don’t know.
Because they didn’t know that the tailoff of the different (regenerative) nozzle would be different, because no ground-test rig would have the senstivity to tell them what it was…? Sorry, but sometimes, you can only learn things in actual flight. And even when you learn it that hard way, it’s still a hell of a lot cheaper than analysis, which may or not be valid, and months of delay.
Did they attempt to do any modeling? What kind of dispersion assumptions did they use? If I had to hazard a guess, I would suspect that the modeling they did, if any, either didn’t correct for vacuum conditions, or did not use appropriately dispersed extreme values. But it doesn’t matter — what you are endorsing is precisely the “let’s fly and see what breaks” attitude. It is the cheaper and quicker approach. The question is whether it gives you a system with the reliability you claim to want.
For sure, there are some things you can only learn by flight test — that’s why we do flight test. I’ll tell you stories over beer next time you’re in Rocket City. But it’s just a waste if you don’t build in those mitigations we already know about.
Why do you think that a) that is a truism and b) they don’t know that?
I think it’s a truism because (a) NASA spent a lot of time designing the booster recovery protocol for the Shuttle SRBs (and it seems like SpaceX decided they could ignore all that), (b) I have been peripherally involved in the trajectory Monte Carlo for Constellation, and (c ) I was directly involved in the staging team for Ares I.
But I’m always looking for enlightenment — are there in fact any boosters that are recovered successfully without tumbling? And how many boosters has SpaceX recovered successfully? Just sayin’.
Do you have ESP, or a ouija board?
No, just the usual rumor mill ;-). SpaceX is not exactly forthcoming about their failures, though I do appreciate the bits they have allowed out. Do you have anything you would like to share?
@Rand: What are the grounds for believing that it isn’t enough, given engine-out capability?
Well, my grounds are that I’ve actually run some numbers. Suppose that Falcon Heavy only needs 26 out of 27 engines to be operating, that is, it has one-engine-out tolerance. (You’ve argued that it should be able to tolerate three failures, based on 1 for a Falcon 9, but as I’ve pointed out before, the fault tolerance is not scalable that way.)
The crucial question is: what is the failure probability of the Merlin engine? If you’ve got some data, I’ll be glad to look at it, but for the moment just suppose it has the same reliability as the RS-68. In that case the k-out-of-N failure rate is around 6e-4… which would be great! Now suppose that the Merlin has the same failure rate as the SSME. Then the k-out-of-N is 8e-4… still great! But suppose the Merlin has the same failure rate as the RL-10. Then the k-out-of-N failure jumps to over 4%! That would be trouble. Of course, you can claim just about whatever you want at this stage for the Merlin failure rate. You can even claim that Falcon Heavy can withstand 2 failures. But I would say that I’m concerned whether anyone at SpaceX has even looked at this issue, or run trajectories, or done anything beyond sketching up a marketing brochure… gosh, what am I saying? They’re probably already cutting metal, analysis be damned!
The ‘restrained at ignition’ bit should make the comparison not-apples-to-apples though.
An aborted launch due to non-catastrophic engine failure shouldn’t end up with payload/crew loss.
What is the failure rate of the RL10 (or the Merlin) after the ignition period? That is: What fraction of the quoted 4% failure rate is ‘failure to light’ or some other ignition problem?
if it were the case, it would be a sharp indictment of their design process.
It would be a sharp indictment of their design process at the time. It tells us nothing about their current one.
@Al: Two of the RL-10 failures involved a failure in the brazing of the combustion chamber, leading to a breach in the chamber and loss of mission. I suppose you could argue that those failures would be caught during the ignition sequence, but I’m not sure that is physically guaranteed. The third was a turbopump failure at T+361 seconds.
Just to be clear, the 4% is the failure rate for 27 RL-10-equivalents with 1-engine-out tolerance. A single RL-10 has a lower failure rate.
Maybe the Merlin will be fine in a 26-out-of-27 configuration. My point is that we have no confidence that anyone has even examined this issue.
For that matter, does the cross-feed introduce new risks? One suspects that separate pumps have to be involved, since the pressure boundary conditions are so different for engine feed and cross-feed. Does the cross-feed affect drain and anti-vortex performance? Ullage pressurization? Does it have implications for ullage pressure collapse? Does it interact with the pogo suppression system? (Does the Falcon 9/Heavy even have a pogo system?) How much is the dispersion in cross-feed pump performance and does that affect the flight propellant reserve?
Are there any risks associated with the 27 engine plumes? It is rumored that plume instability in the 30-engine Soviet N1 caused unforeseen roll control problems. Again, it will probably be fine, but is anyone looking at this?
Inquiring minds want to know….
It would be a sharp indictment of their design process at the time. It tells us nothing about their current one.
That’s fair. But what is their current process? I really haven’t detected any more thoughtfulness or foresight in their approach. I have heard rumors that they are getting better advice now, but what I’m hearing is mostly third hand….
@bbbeard:
Two points. My first point is that whether or not reduced cost for reduced reliability is a good trade depends on the payload. If I’m hauling off-the-shelf commodities (propellants, victuals, etc.), I can live with a reliability of 50%, if the price is right.
My second point is that there are different ways to get high reliability. One way is to design a complicated system with lots of catastrophic failure modes, and pay an army of engineers to wring their hands over each and every possible failure. But if you can design fewer catastrophic failure modes into the system to begin with, you get high reliability without the standing army. That’s why Maxwell Hunter was so insistent on continuous intact abort capability. It’s not clear that high reliability necessarily has to be the enemy of low cost.
@Peter: My second point is that there are different ways to get high reliability. One way is to design a complicated system with lots of catastrophic failure modes, and pay an army of engineers to wring their hands over each and every possible failure. But if you can design fewer catastrophic failure modes into the system to begin with, you get high reliability without the standing army.
Either way, you need to demonstrate that you have a good handle on launch vehicle risks and risk mitigations.
And you also have to be careful where this leads you. Prominent among the reasons for the Ares I configuration was that it had (a) only two stages, and (b) only one engine per stage. What’s more, a Shuttle SRB, with “proven high reliability” (oops, sorry about that O-ring mishap!), was selected for the first stage. What could be simpler or more reliable? 😉 But in that seemingly straightforward decision process, Dr. Horowitz was never bothered that in the entire history of space flight, no one had ever put a cryo upper stage on top of a single inline solid motor. I suspect it will be a long time before anyone tries this again. Not that there is any law of physics that forbids this configuration (in fact we have a nifty solution to the thrust oscillation issue) — it’s just that… the devil is in the details.
bbbeard’s arguement began with the assertion that Falcon 1 flt 2 lacked baffles and this shows a lack of respect for lessons learnt over the years. This is essentially a false premis as a reading of the SpaceX updates notes will quickly shows.
SpaceX was aware of slosh as an issue. They didn’t have baffles, but did impliment a software based anti-slosh program. Whatever the weight of baffles a program beats it hands down. It seemed a smart decision at the time. It saves weight and still mitigates slosh.
However the the contact between the engine bell of the second stage and the interstage imparted a slosh greater than the program was designed to cope with. Now SpaceX could have fixed the problem by adjusting the parameters of the anti-slosh program, but with the 20-20 vision of hindsight SpaceX decided not to rely on just the software but to also re-introduce baffles. The grounds being that you can’t have too many back up systems.
This was the real lesson learnt that day.
But the underlying reason flight 2 failed had little to do with slosh. there was a software error in the first stage engine instructions. The engine was designed to run lean early and rich later in the flight (or the other way around. I always get them mixed up.) Anyway the lines of code were reversed. The result was the engine didn’t perform as well as expected. Staging was at a lower altitude than expected and that caused the stage 2 engine bell to contact the interstage imparting the slosh.
Improved software control before flight was the real fix here.
The lesson we should all take from this is that it’s not enough to assume SpaceX, or anyone else, don’t know what they’re doing. Check your facts before you start throwing around accusations of incompitence.
Re: SpaceX and the Mysterious Case of the Missing Slosh Baffles
Pure speculation on my part, but I think in the much younger and smaller SpaceX where this decision was taken, structures engineering probably didn’t get enough attention or respect vs. engine engineering. Engines are sexy. Structures aren’t. In a new startup, everybody tends to gravitate to the sexy, high-profile stuff, not the seemingly dull, meat-and-potatoes stuff. From what I’ve read of the early SpaceX they were all about the engines.
Re: the engineering wisdom of using lots of engines instead of one or two
As noted by others, if your engine fails and it’s the only one you’ve got, you’re having a bad day. If you’ve got eight others (or 26) – not so much. F9 and FH were designed from the get-go to get their jobs done even with a zero-altitude engine failure. This is more effective reliability (i.e., does the job get done) than is deliverable with any design in which engine performance perfection or near-perfection is mandatory for mission accomplishment. Elon came from the computing world. In computing, parity checks and in-line error-correcting codes are ubiquitous. Computer engineers have also learned to design in near-bulletproof reliability in critical systems by employing multiple fallible components in clever ways so that single component failures do not become system failures – e.g., RAID storage arrays.
Re: “let’s fly and see what breaks” being “the cheaper and quicker approach”
What a crushing indictment of analytical methods! Considering the millions it costs to put even a Falcon I on the pad and light it, why isn’t it trivially possible – after decades of development and with the strong wind of Moore’s Law at their backs – for the numerical/statistical analysis community to model the relevant reality at a competitive price? Or is this inability confined only to the city limits of Huntsville, AL?
What a crushing indictment of analytical methods!
Have you ever heard the phrase “paralysis by analysis.” Some people are so in love with analysis that they can never make a decision. Computer analysis can only take you so far. Ask the folks at XCor about the relative value of computer analysis to wind tunnel testing for their Lynx design.
Ultimately, you have to conduct flight test to find out what works and what doesn’t. Boeing did tons of analysis in their 787 design. They’re still doing hundreds of test flights to prove the analysis was good and to fix the things that analysis got wrong, such as that electrical system issue that grounded the test fleet for weeks.
what you are endorsing is precisely the “let’s fly and see what breaks” attitude. It is the cheaper and quicker approach. The question is whether it gives you a system with the reliability you claim to want.
Quicker and cheaper are good qualities, especially if you are not sacrificing reliability to get them. We don’t know everything that goes on inside the company but we do get glimpses. I see a lot of analysis going on including predictive. The results so far seem to bear this out. There is never going to be enough analysis to satisfy everyone. What they have shown is capability and cost that most would have argued (and did) isn’t possible (but now they can’t.) In the future, they will have failures including loss of life because zero and a hundred never exist in any real life probabilities.
Using the “fly it and see what breaks” approach, Lockheed’s Skunk Works went from blank paper to flying the Blackbird (A-12 Oxcart version) in about 3 years.
Using the “let’s analyze it to death” approach, the F-22 took over 20 years to become operational and that’s with massive overruns and other associated problems.
When “fly it and see what breaks” saves both time and money, it’s a senseable approach to “paralysis by analysis.”
@Fred: SpaceX was aware of slosh as an issue. They didn’t have baffles, but did impl[e]ment a software based anti-slosh program. Whatever the weight of baffles a program beats it hands down. It seemed a smart decision at the time. It saves weight and still mitigates slosh.
By “software-based” I assume you mean scheduling gains and lags in the vehicle flight control system. Software is very light but it also doesn’t provide any dissipation. What is almost always the case is that you need some kind of physical dissipation in the tank to make the vehicle stable throughout the trajectory. Because the g level and fluid level and mass properties are changing throughout the flight, there is usually a crossover of structural, slosh, and flight control frequencies at some point.
But the real question is: what process did they go through to make this decision? What gave them confidence this approach would work? Did they do any scale model testing? Did their trajectory analysis include dispersions about the nominal damping characteristic? Did they even formulate the nominal damping characteristics? Did they even do trajectory dispersion? Did they do a literature search?
Did the people who designed the mods to the flight control software understand the fluid dynamics? Did they understand, e.g., that lateral and rotary slosh modes are coupled, and that their software thus had to mitigate rotary slosh as well as lateral slosh?
And once the slosh mitigation strategy was formulated, what kind of review did it undergo? Did they seek the advice of anyone familiar with launch vehicle fluid dynamics? Was it really just one guy or gal who approved this?
I’m asking these questions because I don’t know the facts. I am ignorant. Maybe SpaceX did all these things and more. But it sure doesn’t show. And it sure doesn’t jibe with all the happy talk about learning by flight test.
The grounds being that you can’t have too many back up systems. This was the real lesson learn[ed] that day.
Well, I hope that is not the lesson they learned. What I hope they learned is that subsystem design has to be tolerant of faults in other subsystems. SpaceX made a (justifiably) big deal out of their niobium nozzle bell, which can withstand impact far better than the carbon-carbon bell of the J-2X. It seems like a good design choice to me, even given the tradeoffs. But did anyone tell the “slosh programming team” their system had to be able to mitigate a slosh wave set up by staging recontact? If not, then it doesn’t really matter if your nozzle bell remains intact or not, because you lose control of the vehicle a few seconds later anyway. Message: if there is a vehicle requirement, like “the vehicle will withstand recontact during staging without loss of mission” then you have to pass it along to all the subsystems that are affected.
That’s a far cry from “you can’t have too many backup systems”.
there was a software error in the first stage engine instructions. The engine was designed to run lean early and rich later in the flight (or the other way around. I always get them mixed up.) Anyway the lines of code were reversed.
The industry has also learned a lot about software control over the years, too. It would appear that SpaceX’s approach also tosses that body of knowledge overboard — or at least encourages unacceptable cutting of corners.
Before any software is loaded onto a modern military aircraft, for example, it is put through a test suite on real hardware in a systems integration lab that is designed to test literally every single line of code. I’m familiar with the SIL for the F-16. They have real hardware versions of every black box on the F-16, real radars, and (tested and approved) simulators for subsystems like missiles and the engine. Once the software passes the SIL, you can’t touch it. There is a protocol then for distributing and installing the software on the correct platforms. Why do we do it that way? It’s not just because we’re anal and the government pays us to do this — it’s because experience has shown that people die unnecessarily if you don’t do it this way.
Maybe the big picture here is that if SpaceX is just a satellite launching company, they can afford to cut corners all they want in the name of cost reduction. I’m sure there are plenty of companies who would accept even a 10 or 20 percent risk of loss if they can get their hardware into space for a tenth of the cost. It’s just hardware; no one cares but the insurance company. But IMHO they have a lot of explaining to do if they want to launch American astronauts into space. And it also means that comparisons to Ares, or even Soyuz, are just hot air.
No amount of testing can prove a complex piece of software is completely error free. Despite huge sums of money being spent on military software development and testing, errors still exist. For example, the first time the Air Force tried to deploy the F-22 to Japan, the navigation computers failed when the plane crossed the International Date Line. Errors have existed in Shuttle software. Errors have happened in mission critical flight control software. Errors happen. We have to deal with them as best we can.
@Dick: As noted by others, if your engine fails and it’s the only one you’ve got, you’re having a bad day. If you’ve got eight others (or 26) – not so much.
It’s slightly more complicated than that.
For each (k,N) for k-out-of-N reliability (e.g. 8 out of 9 for Falcon 9, hypothetically 26 out of 27 for Falcon Heavy), there is a “crossover” single-engine failure rate above which the redundancy only hurts you and below which the redundancy helps — and may be worth implementing if the weight, etc. trades work out.
For (k,N) = (8,9), this crossover reliability is pfail=3.2%. In other words, if the Merlin engine failure rate were above 3.2%, the (8,9) scheme only hurts you (because now you have essentially nine times as many engine failures, plus significant dual failures). But if pfail < 3.2%, you have reduced the overall failure rate, by an amount that is maximum somewhere in between 0 and 3.2% and close to half the crossover rate, i.e. about 1.6%. Below pfail=1.6% it still benefits you, reliability-wise, to have the (8,9) redundancy, but you have diminishing returns as the single-engine failure rate goes to zero. For example, with an RS-68, with about 0.1% failure rate, you can reduce the overall failure rate to practically zero by bundling nine of them together and relying on only eight, but you have to weigh the weight and cost and vehicle integration impacts against the value of eliminating that tenth-of-a-percent LOM. If the Merlin engine has a failure rate more like the RL-10, i.e. over 1%, then it likely makes sense to have redundancy because 1% LOM due just to the engine is unacceptable. If the Merlin engine failure rate exceed 3.2% (unlikely), then the redundancy is actually hurting you. And you would be advised to work on the engine reliability rather than just bundling more engines together.
So whether (8,9) makes sense for Falcon 9 depends in part on the other requirements and in particular on the failure rate of the Merlin engine. My guess would be that it is reasonable to expect the Merlin to fall between RS-68 and RL-10 and it's likely to be a reasonable trade to bundle them in an (8,9) configuration.
But what about Falcon Heavy? Again, it's just a working hypothesis that the Heavy is one-engine-out tolerant (and not 2- or 3-out tolerant). But what happens to these numbers?
We look at the (26,27) scheme. It turns out that the crossover probability is pfail=0.30%. Above this level, the redundancy actually increases the overall failure rate. Between 0 and 0.3%, the benefit is positive, but negligible at 0% and 0.3%.
So here's the question: what is the failure rate for a single Merlin engine? If it's greater than 0.3%, you have only hurt yourself, reliability-wise, by bundling 27 of them together even if you can tolerate a failure. If it's better than an RS-68, you benefit by bundling, but only by a small amount.
So these are the preliminary calculations you would do to start a reliability analysis. The preliminary model assumes binary choices (engine good / engine fail), independent probabilities, firm assumptions about k, etc. The next steps relax some of these assumptions — what if some two-engine-out scenarios are tolerable (e.g. one each in the left and right boosters is probably okay, but not two in the middle)? What if some single-engine failures destroy the vehicle? What do we find out if we estimate an uncertainty for the engine failure rate and test the sensitivity to our assumptions (i.e. do a PRA)?
My point is that "more engines is always more reliable" is untrue, and the decision about engine configurations can be made on a rational basis.
@Dick: What a crushing indictment of analytical methods! Considering the millions it costs to put even a Falcon I on the pad and light it, why isn’t it trivially possible – after decades of development and with the strong wind of Moore’s Law at their backs – for the numerical/statistical analysis community to model the relevant reality at a competitive price? Or is this inability confined only to the city limits of Huntsville, AL?
Not sure where you’re going with this, but it is not my position that the choices are dimming the lights with computer models vs flight test. The point is that you need to have validated models for the relevant pieces of your system that you are trying to design. Testing is essential, in particular subscale or full-scale ground testing and possibly subscale flight testing to lead to full-scale flight test. The testing, analysis, simulation, and design processes have to be informed by what we have learned in 50 (or is it 70?) years of launch vehicle development.
FWIW I don’t care where you work, Huntsville, Huntington Beach, Beijing, or Timbuktu, nobody has the capability to do stem-to-stern simulation of all the “relevant reality” of a launch vehicle. We’re not even close… Moore’s law notwithstanding.
@Larry: Computer analysis can only take you so far.
One of my pet peeves is the headlong rush to incorporate CFD and FEM into the design process without adequate validation, or even understanding of the underlying physics. I can’t count how many arguments I’ve had with CFD folks who think the purpose of testing is to prove their codes, who then go on to claim they “design” components with CFD when all they are doing is modeling their guesses at the correct design parameters.
IMHO it is only through the correlation of testing, analysis, and simulation that we come to a validated understanding of the physics of any particular system on the launch vehicle, be it the physics of a pogo suppressor, the dynamics of a stage separation system, the plume forces of a retrorocket, the effectiveness of an anti-vortex device mounted on a tank drain, the impact resistance of a nozzle bell, and on and on. It’s a painstaking process.
@Larry: Errors happen. We have to deal with them as best we can.
If your point is that the imperfectibility of software is an excuse not to exercise effective software control, then I demur.
Is this comment from the article true:
“… kerosene is heavier than hydrogen for the same amount of energy, ….”
Having trouble believing it from a strictly chemical sense. I could believe “more Isp” or “more delV” per pound, but total energy?
Since carbon is heavier than hydrogen it seems reasonable that there would be more bonds per gram.
No, I’m flat wrong. On a per-mass basis it’s just ridiculous. Per mole is another story – but pointless. Latent heat of vaporization not going to cover that large a difference.
Rand: you may already know this, but if you look at particular categories of your blog (like “Space”), you don’t see active links in the stories. This post, for instance, just says “SpaceX Thoughts …from Ric Locke” with no html link to Ric’s article.
I only bookmark the Space category, but that means that following your links always requires an extra click.
Actually, I wasn’t aware of that. Not much I can do about it without hacking WordPress.
Ric sez: According to the European Space Agency, the Soyuz launch vehicle is the most frequently used and most reliable launch vehicle in the world. Folks, that’s what you call a “track record”. It makes sense to build on and improve what’s been demonstrated to work, no?
The Soyuz has about a 3% launch failure rate. This is actually better than the average for all launch vehicles, but not better than Shuttle, which is half of that, counting Columbia as a launch failure. By Ric’s reasoning, we should be evolving the Shuttle.
Ah, but the Shuttle is more expensive than Soyuz, no doubt. And ARES would have continued that tradition, intending on pushing failure rates well below a percent (1/500 was the design requirement) but with higher launch cost.
And that’s one source of my irritation with SpaceX fans: you are willing to tolerate a 3% failure rate if it will lower launch costs. You don’t see the value in cutting failure rate in half if it triples the cost. You have a magical belief that if you just fly crappy rockets long enough with low cost as the primary objective, they will get better and better as you fix the things that break them, and that eventually the failure rate will be low enough to be acceptable. Well, look, the Russians have over a 1000 Soyuz launches, and the empirical failure rate is still over a percent (if you want to ignore earlier failures). Meanwhile, Shuttle is being retired in large part because 1.5% is too high.
It’s a gross oversimplfication to say that Shuttle is being retired because its failure rate is too high. For instance, if we had a couple dozen orbiters, and the ops costs were an order of magnitude lower, it would probably still be flying, even with its current reliability.
I understand completely that it is your position that launch costs are the sine qua non of launch vehicle viability, but if you actually listen to the people who make the decisions, you will find that safety is a primary consideration. Whenever Mike Griffin was asked why we couldn’t just continue operating the Shuttle, his response had to do with safety, not launch costs. And you’re well aware of that, since that is a constant refrain in your criticism of NASA.
Mike Griffin did not make the decision to cancel the Shuttle.
Fly crappy rockets long enough? Who said anything about wanting to fly crappy rockets? bbbeard, do you think Falcon 9 is a “crappy rocket”?
And that’s one source of my irritation with SpaceX fans: you are willing to tolerate a 3% failure rate if it will lower launch costs. You don’t see the value in cutting failure rate in half if it triples the cost. You have a magical belief that if you just fly crappy rockets long enough with low cost as the primary objective, they will get better and better as you fix the things that break them, and that eventually the failure rate will be low enough to be acceptable.
What a nice set of strawmen you’ve assembled there. Tell us, who among us SpaceX fans are saying any of those things you just claimed? Anyone? Anyone? Who among us has said we’d be willing to tolerate a 3% loss rate?
SpaceX is a tightly vertically integrated company. They produce just about everything on their boosters and capsule. As a SpaceX employee told me yesterday, “If there’s a problem, we know it’s our fault. There is no finger pointing among different subcontractors. It’s our fault so we move quickly to fix it.” Their track record to date proves that is true. Furthermore, they’ve built features into their boosters to increase reliability because they know that losing a lot of rockets isn’t a good business strategy to woo future customers. Yes, they lost 3 Falcon 1 rockets in a row. In each case, the cause of the problem was quickly determined and fixed. None of those problems have been repeated and they’ve had 4 successful launches in a row (two each of the Falcon 1 and Falcon 9). They have a long way to go before they establish a definitive track record but they’re working on it.
Consider their base Falcon 9. It was designed to be able to survive losing an engine right at liftoff and still fly the mission. Both the Atlas V and Delta IV core designs are single engined. That means any loss of an engine (or even significant loss of thrust) will result in loss of mission and payload. The result of those design choices is that ULA has to use very high quality and very expensive engines. SpaceX, by way of contrast, built their engines to be simple, reliable, and inexpensive to manufacture. According to that SpaceX employee I was talking to, they also designed them to be reusable from 20-40 times even after being exposed to salt water. If they can solve their stage recovery problems, that will further lower their costs.
Wait, my belief that people become more successful with more practice is magical? I would have called it “obvious”, certainly, “supported by all available data in a wide variety of fields”, I suppose, and I’d even have been egotistical enough to say “rational”, but magical too? Neat! From now on I’m going to call everything I write a “spell”.
@Rand: yes, but Griffin was central to the decision to keep it dead when Congress wanted to extend the life of the Shuttle.
@Cecil: Who said anything about wanting to fly crappy rockets? — That is what I infer from Ric Locke’s blog post, which is the topic of this thread. Ric’s post is in part a paean to the workhorse of the Soviet rocket fleet. He quotes a journalist quoting ESA quoting Russia that the Voskhod/Molniya/Soyuz is “the most reliable launch vehicle in the world” [sic]. Ric thinks so much of this reasoning that he wants SpaceX to emulate Russian design practice. You fill in the blanks.
@Larry: Who among us has said we’d be willing to tolerate a 3% loss rate? — Again, Ric has held up the 1700-flight, 3% loss rate of the Soyuz as a paradigm for rocket development. Except that he left out the “3%” part. Because it sounds, you know, unacceptable.
“we move quickly to fix it…. they lost 3 Falcon 1 rockets in a row. In each case, the cause of the problem was quickly determined and fixed. “ — Have they considered designing the rocket correctly in the first place? I’ve mentioned this before and still have not gotten an answer: why was Falcon I not designed with LOX tank baffles in the first place? We’ve known for almost 50 years that you have to suppress propellant slosh. How did SpaceX design and launch without this knowledge? How many more “lessons learned” will they have to relearn?
None of those problems have been repeated and they’ve had 4 successful launches in a row (two each of the Falcon 1 and Falcon 9). They have a long way to go before they establish a definitive track record but they’re working on it. — IIRC Falcon I flights 2 and 3 were both failures that involved recontact during staging. I would like to know what process they went through to design the staging system. Is this another case of “let’s fly and see what breaks”? In the case of flight 3, thrust tailoff has long been known to be a major source of concern for staging. How much consideration did SpaceX give to this? How did they process the Merlin engine test data and convert it to requirements for the staging system? I have no confidence that SpaceX knows what they are doing. Right now their rockets are “software quality”, not “aerospace quality”, which is to say, they’re cheap, they work most of the time, and they are undergoing a continuous process of upgrading. Which befits the founder of PayPal.
four successful launches in a row — Granted, that is way better than four losses in a row, which is what we saw the last time someone put 30 engines on the first stage of a launch vehicle. But you have to understand that an engineer will look at “0 losses in 2 flights” and get out the binomial tables. It turns out that with zero losses in two flights you have 50% confidence that the true failure rate is less than… drumroll… 29.3%. Now, that’s better than the 79% failure rate at 50% confidence that we see with the Taurus XL (2 failures out of 3). And while any successful launch is a reason to pop the champagne corks, I’m not ready to buy yet.
Just for reference, call me when the Falcon 9 gets to seven launches without failure. That’s when the 50% confidence empirical failure rate drops below the 10% mark, which is approximately the historical failure rate. And I will buy each of you a Coca-cola if Falcon 9 makes it to 34 launches with no failures, when its 50% confidence failure rate drops below Shuttle’s.
@Roy: Wait, my belief that people become more successful with more practice is magical? — The question isn’t whether the design gets better as it is exercised, it’s whether it gets better enough, and whether that process is better than designing the rocket correcly in the first place. The Soyuz has been evolving for four decades and has been launched 1700 times. When will it evolve to a rocket with less than 1% failure rate?
bbbeard, rockets will fail, a capsule design gives the crew a better chance of survival than the shuttle design so if spacex can get their failure rate down to 1-2% or better, and you’ve offered no evidence to the contrary, given the choice, I’d rather be launched in a dragon than the shuttle, even ignoring the extra cost of the shuttle.
Have they considered designing the rocket correctly in the first place?
Oh, you mean like the Delta III that had a 33% success rate? It was “designed right” by professional engineers and it still had failures. So did the Ariene V and most other booster families in the history of spaceflight. There is such a thing as a learning curve with any new design.
IIRC Falcon I flights 2 and 3 were both failures that involved recontact during staging.
IIRC, Flight 2’s first stage brushed against the second stage’s nozzle at separation. However, the coning that lead to premature upper stage engine shut-down didn’t being until minutes later.
On Flight 3, there was definite bumping between the stages at separation. This was caused by an unexpected level of residual thrust from the first stage engine. On that flight, they switched from a radiantly cooled Merlin to a regeneratively cooled design. They had no facilities to test the residual thrust so they didn’t anticipate it. The only fix required for that was to wait a few additional seconds after first stage engine shut-down before separating the stages. It worked perfectly on all subsequent flights.
Time will tell about SpaceX’s safety record. The Shuttle had a seemingly perfect safety record until the Challenger accident. No manned spacecraft had a heat shield failure until Columbia. Those accidents happened despite billions being spent on R&D and additional billions being spent each year for Shuttle operations. Sometimes, spending a lot of money on something only results in an expensive rhing, not necessarily a safe or good thing. You don’t always get what you pay for. Conversely, an inexpensive thing designed and built well may be more reliable than an expensive thing. Guess which car ends up in the repair shop more often, a Honda Civic or a Ferrari?
@Andrew: if spacex can get their failure rate down to 1-2% or better, and you’ve offered no evidence to the contrary…
@Larry: There is such a thing as a learning curve with any new design.
Here you go:
Soyuz Learning Curve
SpaceX Learning Curve
Note the latter includes all seven SpaceX launches. It might be fairer to have separate Falcon 1 and Falcon 9 curves. I’ll leave that as an exercise.
It might be fairer to have separate Falcon 1 and Falcon 9 curves.
Ya think?
@Larry: Conversely, an inexpensive thing designed and built well may be more reliable than an expensive thing. — I think the point is whether you make safety a design requirement, and whether you have the institutional capability to meet that requirement. A Volvo S40 is safer than the Mustang GT that I drive, despite being cheaper, in large part because they emphasize safety over performance, and have for a long time. SpaceX claims to have high reliability (not safety per se, as they are unmanned) as a design goal but so far it is not showing up in the numbers or in their design practices.
@Rand: Ya think? — I’m just being fair, as always. ;-)…. I think there is an argument either way. I think the Falcon I failures were the byproduct of an misguided design process. It is not clear at all that the design process has been fixed, even as individual design decisions have been corrected. Did a different group of engineers design the Falcon 9? Did they use a predictive, rather than reactive, design process? If so, there is an argument for separating the two datasets. On the other hand, both rockets use the same engine, and presumably some other hardware, so the launch failure probabilities are not independent no matter what argument you make.
SpaceX claims to have high reliability (not safety per se, as they are unmanned) as a design goal but so far it is not showing up in the numbers or in their design practices.
The same can be said for NASA. When they were building the Shuttles back in the 1970s, they claimed it would be extremely safe (IIRC, something like a LOC of 1:1000). Despite all the best design practices and many billions of dollars, they lost two vehicles and 14 astronauts in less than 130 launches. It doesn’t exactly sound like they met their goals, does it? So, why are you holding a private company to higher standards than what NASA was able to achieve?
@Larry: So, why are you holding a private company to higher standards than what NASA was able to achieve?
Because it’s not 1975 anymore.
SpaceX seems to be heading down a path of rediscovering all the possible ways one can fail a launch. They seem charmingly devoid of any institutional memory. It doesn’t bother them that the mistakes they’ve made could have been avoided because we’ve known about these failure modes for decades. Shuttle is at 1-2% right now, and that’s not good enough to keep flying people beyond a couple of more flights. SpaceX is at 43%, or 0%, depending on who’s being asked — the numbers are too small to judge. But the way they’ve failed, and their stated design philosophy, which brings to mind Mickey Rooney saying “Let’s put on a show!”, brings up serious questions.
But they’re cheap, I’ll give them that.
And I don’t mean to give NASA a pass. There are some serious process issues at NASA — including a rapidly diminishing institutional memory! But this thread is ostensibly about Ric Locke’s gushing encomium for SpaceX. If you want to change topics, just say the magic words (“You’re right, Bernard! Hey, want to talk about NASA?”) 😉
SpaceX seems to be heading down a path of rediscovering all the possible ways one can fail a launch.
That seems very unlikely. I think you’re being hyperbolic.
They seem charmingly devoid of any institutional memory. It doesn’t bother them that the mistakes they’ve made could have been avoided because we’ve known about these failure modes for decades.
Are you unaware that the first thing that they did before initiating design activities was to hire Futron to do a study for them on exactly this topic?
@Rand: I think you’re being hyperbolic.
And you, sir, are being elliptical!
Are you unaware that the first thing that they did before initiating design activities was to hire Futron to do a study for them on exactly this topic?
Yes, I’ve read the Futron study before. Are you aware that the study deals only with subsystem failures and says nothing about my main point — that the design process at SpaceX is defective?
More to the point, are you aware that the Futron study only covers the two decades prior to 2004? Do you understand that prior to 1984 there were multiple launch failures due to slosh problems? That we figured out how to mitigate this risk? That the lessons learned from those losses and the research that followed are the main reason slosh-related failures are less frequent now? Well, they were until SpaceX came along and Futron decided that launch vehicle history started in 1984! Can you spot the root cause of the Demo 2 failure?
In any case, are you seriously suggesting that a four-page white paper from a “decision management solution” think tank is a substitute for a proactive design process?
In any case, are you seriously suggesting that a four-page white paper from a “decision management solution” think tank is a substitute for a proactive design process?
No, but then, the notion that the “design process is defective” is your opinion, not a fact.
Did they use a predictive, rather than reactive, design process?
All design is predictive; All fixes are reactive. There is such a thing as diminishing returns. The safest rocket is in a museum.
I’d be happy to take the place of that cheese wheel on the next flight.
No amount of memory is going to help you anticipate all future issues. Also, if they added all the safety feature of every rocket (isn’t that your suggestion???) no amount of thrust would get it off the ground.
They’re now planning a three stage rocket where two of the stages get held before launch (with abort potential) and only one engine isn’t.
Dragon will have over a dozen missions before any people go up. NASA would go on second launch (who are the amateurs?)
There is such a thing as remembering the wrong lessons.
@ken: All design is predictive
Yes, but some predictions are better than others. Predictions based on test and analysis with a sound statistical characterization of uncertainties are better than formulas picked out of an undergraduate textbook. Predictions based on enumeration and re-examination of known failure modes is better than design based on a willful disregard of risks and a forlorn hope that ignorance will be not fatal.
Also, if they added all the safety feature of every rocket (isn’t that your suggestion???) no amount of thrust would get it off the ground.
The standard of practice is to designate acceptable levels of risk and to design to them using carefully selected measures of statistical significance. Is there any evidence that SpaceX takes this approach? I would say, not hardly… because the available evidence suggests they barely know what the problems are.
No amount of memory is going to help you anticipate all future issues.
Right. So let’s not even try.
We’re at the point in aerospace history that we think we can design manned launch vehicles with smaller than 1% failure rates (with design requirements tighter than that). But if we choose to throw away everything we’ve learned in 50 years of space travel (or at least in the first 25 years of space travel, if you put your stock in the Futron approach), I can state with some confidence that we won’t get there.
NASA would go on second launch
Not really the second launch, but too soon, in my view as well. Like I said, if you want to change topics, just say the magic words (“You’re right, Bernard! Hey, want to talk about NASA?”)
…design based on a willful disregard of risks and a forlorn hope that ignorance will be not fatal.
…choose to throw away everything we’ve learned in 50 years of space travel…
Bernard, I see that you continue your unsubstantiated hyperbole.
@Rand: Bernard, I see that you continue your unsubstantiated hyperbole.
Okay, I admit it, I’m being hyperbolic. They’re only willing to throw away half of what we’ve learned.
But I keep returning to the substantiation and you keep ignoring it. Why were there no baffles in the LOX tank of Falcon 1? Why was the Merlin thrust tailoff not designed for? What are the grounds for believing that a 27-engine first stage is reliable enough for critical missions?
Oh, and — did SpaceX not know you have to tumble boosters if you want to recover them?
Why were there no baffles in the LOX tank of Falcon 1?
I’m going to hazard a guess that a single individual overrode his engineers who told him they were needed, and that he has since learned not to do that. Just a guess, though.
Why was the Merlin thrust tailoff not designed for?
Because they didn’t know that the tailoff of the different (regenerative) nozzle would be different, because no ground-test rig would have the senstivity to tell them what it was…? Sorry, but sometimes, you can only learn things in actual flight. And even when you learn it that hard way, it’s still a hell of a lot cheaper than analysis, which may or not be valid, and months of delay.
What are the grounds for believing that a 27-engine first stage is reliable enough for critical missions?
What are the grounds for believing that it isn’t enough, given engine-out capability? I would point out that they are going to quickly gain a lot of data on engine reliability from that design, given how many engines they’ll be testing each flight.
…did SpaceX not know you have to tumble boosters if you want to recover them?
Why do you think that a) that is a truism and b) they don’t know that?
Do you have ESP, or a ouija board?
@Rand: I’m going to hazard a guess
Seriously, why don’t you ask Elon next time you have a friendly moment? I’m still in the dark after several years of inquiry. What you said would be my guess, too, but if it were the case, it would be a sharp indictment of their design process. Which is probably why we still don’t know.
Because they didn’t know that the tailoff of the different (regenerative) nozzle would be different, because no ground-test rig would have the senstivity to tell them what it was…? Sorry, but sometimes, you can only learn things in actual flight. And even when you learn it that hard way, it’s still a hell of a lot cheaper than analysis, which may or not be valid, and months of delay.
Did they attempt to do any modeling? What kind of dispersion assumptions did they use? If I had to hazard a guess, I would suspect that the modeling they did, if any, either didn’t correct for vacuum conditions, or did not use appropriately dispersed extreme values. But it doesn’t matter — what you are endorsing is precisely the “let’s fly and see what breaks” attitude. It is the cheaper and quicker approach. The question is whether it gives you a system with the reliability you claim to want.
For sure, there are some things you can only learn by flight test — that’s why we do flight test. I’ll tell you stories over beer next time you’re in Rocket City. But it’s just a waste if you don’t build in those mitigations we already know about.
Why do you think that a) that is a truism and b) they don’t know that?
I think it’s a truism because (a) NASA spent a lot of time designing the booster recovery protocol for the Shuttle SRBs (and it seems like SpaceX decided they could ignore all that), (b) I have been peripherally involved in the trajectory Monte Carlo for Constellation, and (c ) I was directly involved in the staging team for Ares I.
But I’m always looking for enlightenment — are there in fact any boosters that are recovered successfully without tumbling? And how many boosters has SpaceX recovered successfully? Just sayin’.
Do you have ESP, or a ouija board?
No, just the usual rumor mill ;-). SpaceX is not exactly forthcoming about their failures, though I do appreciate the bits they have allowed out. Do you have anything you would like to share?
@Rand: What are the grounds for believing that it isn’t enough, given engine-out capability?
Well, my grounds are that I’ve actually run some numbers. Suppose that Falcon Heavy only needs 26 out of 27 engines to be operating, that is, it has one-engine-out tolerance. (You’ve argued that it should be able to tolerate three failures, based on 1 for a Falcon 9, but as I’ve pointed out before, the fault tolerance is not scalable that way.)
The crucial question is: what is the failure probability of the Merlin engine? If you’ve got some data, I’ll be glad to look at it, but for the moment just suppose it has the same reliability as the RS-68. In that case the k-out-of-N failure rate is around 6e-4… which would be great! Now suppose that the Merlin has the same failure rate as the SSME. Then the k-out-of-N is 8e-4… still great! But suppose the Merlin has the same failure rate as the RL-10. Then the k-out-of-N failure jumps to over 4%! That would be trouble. Of course, you can claim just about whatever you want at this stage for the Merlin failure rate. You can even claim that Falcon Heavy can withstand 2 failures. But I would say that I’m concerned whether anyone at SpaceX has even looked at this issue, or run trajectories, or done anything beyond sketching up a marketing brochure… gosh, what am I saying? They’re probably already cutting metal, analysis be damned!
The ‘restrained at ignition’ bit should make the comparison not-apples-to-apples though.
An aborted launch due to non-catastrophic engine failure shouldn’t end up with payload/crew loss.
What is the failure rate of the RL10 (or the Merlin) after the ignition period? That is: What fraction of the quoted 4% failure rate is ‘failure to light’ or some other ignition problem?
if it were the case, it would be a sharp indictment of their design process.
It would be a sharp indictment of their design process at the time. It tells us nothing about their current one.
@Al: Two of the RL-10 failures involved a failure in the brazing of the combustion chamber, leading to a breach in the chamber and loss of mission. I suppose you could argue that those failures would be caught during the ignition sequence, but I’m not sure that is physically guaranteed. The third was a turbopump failure at T+361 seconds.
Just to be clear, the 4% is the failure rate for 27 RL-10-equivalents with 1-engine-out tolerance. A single RL-10 has a lower failure rate.
Maybe the Merlin will be fine in a 26-out-of-27 configuration. My point is that we have no confidence that anyone has even examined this issue.
For that matter, does the cross-feed introduce new risks? One suspects that separate pumps have to be involved, since the pressure boundary conditions are so different for engine feed and cross-feed. Does the cross-feed affect drain and anti-vortex performance? Ullage pressurization? Does it have implications for ullage pressure collapse? Does it interact with the pogo suppression system? (Does the Falcon 9/Heavy even have a pogo system?) How much is the dispersion in cross-feed pump performance and does that affect the flight propellant reserve?
Are there any risks associated with the 27 engine plumes? It is rumored that plume instability in the 30-engine Soviet N1 caused unforeseen roll control problems. Again, it will probably be fine, but is anyone looking at this?
Inquiring minds want to know….
It would be a sharp indictment of their design process at the time. It tells us nothing about their current one.
That’s fair. But what is their current process? I really haven’t detected any more thoughtfulness or foresight in their approach. I have heard rumors that they are getting better advice now, but what I’m hearing is mostly third hand….
@bbbeard:
Two points. My first point is that whether or not reduced cost for reduced reliability is a good trade depends on the payload. If I’m hauling off-the-shelf commodities (propellants, victuals, etc.), I can live with a reliability of 50%, if the price is right.
My second point is that there are different ways to get high reliability. One way is to design a complicated system with lots of catastrophic failure modes, and pay an army of engineers to wring their hands over each and every possible failure. But if you can design fewer catastrophic failure modes into the system to begin with, you get high reliability without the standing army. That’s why Maxwell Hunter was so insistent on continuous intact abort capability. It’s not clear that high reliability necessarily has to be the enemy of low cost.
@Peter: My second point is that there are different ways to get high reliability. One way is to design a complicated system with lots of catastrophic failure modes, and pay an army of engineers to wring their hands over each and every possible failure. But if you can design fewer catastrophic failure modes into the system to begin with, you get high reliability without the standing army.
Either way, you need to demonstrate that you have a good handle on launch vehicle risks and risk mitigations.
And you also have to be careful where this leads you. Prominent among the reasons for the Ares I configuration was that it had (a) only two stages, and (b) only one engine per stage. What’s more, a Shuttle SRB, with “proven high reliability” (oops, sorry about that O-ring mishap!), was selected for the first stage. What could be simpler or more reliable? 😉 But in that seemingly straightforward decision process, Dr. Horowitz was never bothered that in the entire history of space flight, no one had ever put a cryo upper stage on top of a single inline solid motor. I suspect it will be a long time before anyone tries this again. Not that there is any law of physics that forbids this configuration (in fact we have a nifty solution to the thrust oscillation issue) — it’s just that… the devil is in the details.
bbbeard’s arguement began with the assertion that Falcon 1 flt 2 lacked baffles and this shows a lack of respect for lessons learnt over the years. This is essentially a false premis as a reading of the SpaceX updates notes will quickly shows.
SpaceX was aware of slosh as an issue. They didn’t have baffles, but did impliment a software based anti-slosh program. Whatever the weight of baffles a program beats it hands down. It seemed a smart decision at the time. It saves weight and still mitigates slosh.
However the the contact between the engine bell of the second stage and the interstage imparted a slosh greater than the program was designed to cope with. Now SpaceX could have fixed the problem by adjusting the parameters of the anti-slosh program, but with the 20-20 vision of hindsight SpaceX decided not to rely on just the software but to also re-introduce baffles. The grounds being that you can’t have too many back up systems.
This was the real lesson learnt that day.
But the underlying reason flight 2 failed had little to do with slosh. there was a software error in the first stage engine instructions. The engine was designed to run lean early and rich later in the flight (or the other way around. I always get them mixed up.) Anyway the lines of code were reversed. The result was the engine didn’t perform as well as expected. Staging was at a lower altitude than expected and that caused the stage 2 engine bell to contact the interstage imparting the slosh.
Improved software control before flight was the real fix here.
The lesson we should all take from this is that it’s not enough to assume SpaceX, or anyone else, don’t know what they’re doing. Check your facts before you start throwing around accusations of incompitence.
Re: SpaceX and the Mysterious Case of the Missing Slosh Baffles
Pure speculation on my part, but I think in the much younger and smaller SpaceX where this decision was taken, structures engineering probably didn’t get enough attention or respect vs. engine engineering. Engines are sexy. Structures aren’t. In a new startup, everybody tends to gravitate to the sexy, high-profile stuff, not the seemingly dull, meat-and-potatoes stuff. From what I’ve read of the early SpaceX they were all about the engines.
Re: the engineering wisdom of using lots of engines instead of one or two
As noted by others, if your engine fails and it’s the only one you’ve got, you’re having a bad day. If you’ve got eight others (or 26) – not so much. F9 and FH were designed from the get-go to get their jobs done even with a zero-altitude engine failure. This is more effective reliability (i.e., does the job get done) than is deliverable with any design in which engine performance perfection or near-perfection is mandatory for mission accomplishment. Elon came from the computing world. In computing, parity checks and in-line error-correcting codes are ubiquitous. Computer engineers have also learned to design in near-bulletproof reliability in critical systems by employing multiple fallible components in clever ways so that single component failures do not become system failures – e.g., RAID storage arrays.
Re: “let’s fly and see what breaks” being “the cheaper and quicker approach”
What a crushing indictment of analytical methods! Considering the millions it costs to put even a Falcon I on the pad and light it, why isn’t it trivially possible – after decades of development and with the strong wind of Moore’s Law at their backs – for the numerical/statistical analysis community to model the relevant reality at a competitive price? Or is this inability confined only to the city limits of Huntsville, AL?
What a crushing indictment of analytical methods!
Have you ever heard the phrase “paralysis by analysis.” Some people are so in love with analysis that they can never make a decision. Computer analysis can only take you so far. Ask the folks at XCor about the relative value of computer analysis to wind tunnel testing for their Lynx design.
Ultimately, you have to conduct flight test to find out what works and what doesn’t. Boeing did tons of analysis in their 787 design. They’re still doing hundreds of test flights to prove the analysis was good and to fix the things that analysis got wrong, such as that electrical system issue that grounded the test fleet for weeks.
what you are endorsing is precisely the “let’s fly and see what breaks” attitude. It is the cheaper and quicker approach. The question is whether it gives you a system with the reliability you claim to want.
Quicker and cheaper are good qualities, especially if you are not sacrificing reliability to get them. We don’t know everything that goes on inside the company but we do get glimpses. I see a lot of analysis going on including predictive. The results so far seem to bear this out. There is never going to be enough analysis to satisfy everyone. What they have shown is capability and cost that most would have argued (and did) isn’t possible (but now they can’t.) In the future, they will have failures including loss of life because zero and a hundred never exist in any real life probabilities.
Using the “fly it and see what breaks” approach, Lockheed’s Skunk Works went from blank paper to flying the Blackbird (A-12 Oxcart version) in about 3 years.
Using the “let’s analyze it to death” approach, the F-22 took over 20 years to become operational and that’s with massive overruns and other associated problems.
When “fly it and see what breaks” saves both time and money, it’s a senseable approach to “paralysis by analysis.”
@Fred: SpaceX was aware of slosh as an issue. They didn’t have baffles, but did impl[e]ment a software based anti-slosh program. Whatever the weight of baffles a program beats it hands down. It seemed a smart decision at the time. It saves weight and still mitigates slosh.
By “software-based” I assume you mean scheduling gains and lags in the vehicle flight control system. Software is very light but it also doesn’t provide any dissipation. What is almost always the case is that you need some kind of physical dissipation in the tank to make the vehicle stable throughout the trajectory. Because the g level and fluid level and mass properties are changing throughout the flight, there is usually a crossover of structural, slosh, and flight control frequencies at some point.
But the real question is: what process did they go through to make this decision? What gave them confidence this approach would work? Did they do any scale model testing? Did their trajectory analysis include dispersions about the nominal damping characteristic? Did they even formulate the nominal damping characteristics? Did they even do trajectory dispersion? Did they do a literature search?
Did the people who designed the mods to the flight control software understand the fluid dynamics? Did they understand, e.g., that lateral and rotary slosh modes are coupled, and that their software thus had to mitigate rotary slosh as well as lateral slosh?
And once the slosh mitigation strategy was formulated, what kind of review did it undergo? Did they seek the advice of anyone familiar with launch vehicle fluid dynamics? Was it really just one guy or gal who approved this?
I’m asking these questions because I don’t know the facts. I am ignorant. Maybe SpaceX did all these things and more. But it sure doesn’t show. And it sure doesn’t jibe with all the happy talk about learning by flight test.
The grounds being that you can’t have too many back up systems. This was the real lesson learn[ed] that day.
Well, I hope that is not the lesson they learned. What I hope they learned is that subsystem design has to be tolerant of faults in other subsystems. SpaceX made a (justifiably) big deal out of their niobium nozzle bell, which can withstand impact far better than the carbon-carbon bell of the J-2X. It seems like a good design choice to me, even given the tradeoffs. But did anyone tell the “slosh programming team” their system had to be able to mitigate a slosh wave set up by staging recontact? If not, then it doesn’t really matter if your nozzle bell remains intact or not, because you lose control of the vehicle a few seconds later anyway. Message: if there is a vehicle requirement, like “the vehicle will withstand recontact during staging without loss of mission” then you have to pass it along to all the subsystems that are affected.
That’s a far cry from “you can’t have too many backup systems”.
there was a software error in the first stage engine instructions. The engine was designed to run lean early and rich later in the flight (or the other way around. I always get them mixed up.) Anyway the lines of code were reversed.
The industry has also learned a lot about software control over the years, too. It would appear that SpaceX’s approach also tosses that body of knowledge overboard — or at least encourages unacceptable cutting of corners.
Before any software is loaded onto a modern military aircraft, for example, it is put through a test suite on real hardware in a systems integration lab that is designed to test literally every single line of code. I’m familiar with the SIL for the F-16. They have real hardware versions of every black box on the F-16, real radars, and (tested and approved) simulators for subsystems like missiles and the engine. Once the software passes the SIL, you can’t touch it. There is a protocol then for distributing and installing the software on the correct platforms. Why do we do it that way? It’s not just because we’re anal and the government pays us to do this — it’s because experience has shown that people die unnecessarily if you don’t do it this way.
Maybe the big picture here is that if SpaceX is just a satellite launching company, they can afford to cut corners all they want in the name of cost reduction. I’m sure there are plenty of companies who would accept even a 10 or 20 percent risk of loss if they can get their hardware into space for a tenth of the cost. It’s just hardware; no one cares but the insurance company. But IMHO they have a lot of explaining to do if they want to launch American astronauts into space. And it also means that comparisons to Ares, or even Soyuz, are just hot air.
No amount of testing can prove a complex piece of software is completely error free. Despite huge sums of money being spent on military software development and testing, errors still exist. For example, the first time the Air Force tried to deploy the F-22 to Japan, the navigation computers failed when the plane crossed the International Date Line. Errors have existed in Shuttle software. Errors have happened in mission critical flight control software. Errors happen. We have to deal with them as best we can.
@Dick: As noted by others, if your engine fails and it’s the only one you’ve got, you’re having a bad day. If you’ve got eight others (or 26) – not so much.
It’s slightly more complicated than that.
For each (k,N) for k-out-of-N reliability (e.g. 8 out of 9 for Falcon 9, hypothetically 26 out of 27 for Falcon Heavy), there is a “crossover” single-engine failure rate above which the redundancy only hurts you and below which the redundancy helps — and may be worth implementing if the weight, etc. trades work out.
For (k,N) = (8,9), this crossover reliability is pfail=3.2%. In other words, if the Merlin engine failure rate were above 3.2%, the (8,9) scheme only hurts you (because now you have essentially nine times as many engine failures, plus significant dual failures). But if pfail < 3.2%, you have reduced the overall failure rate, by an amount that is maximum somewhere in between 0 and 3.2% and close to half the crossover rate, i.e. about 1.6%. Below pfail=1.6% it still benefits you, reliability-wise, to have the (8,9) redundancy, but you have diminishing returns as the single-engine failure rate goes to zero. For example, with an RS-68, with about 0.1% failure rate, you can reduce the overall failure rate to practically zero by bundling nine of them together and relying on only eight, but you have to weigh the weight and cost and vehicle integration impacts against the value of eliminating that tenth-of-a-percent LOM. If the Merlin engine has a failure rate more like the RL-10, i.e. over 1%, then it likely makes sense to have redundancy because 1% LOM due just to the engine is unacceptable. If the Merlin engine failure rate exceed 3.2% (unlikely), then the redundancy is actually hurting you. And you would be advised to work on the engine reliability rather than just bundling more engines together.
So whether (8,9) makes sense for Falcon 9 depends in part on the other requirements and in particular on the failure rate of the Merlin engine. My guess would be that it is reasonable to expect the Merlin to fall between RS-68 and RL-10 and it's likely to be a reasonable trade to bundle them in an (8,9) configuration.
But what about Falcon Heavy? Again, it's just a working hypothesis that the Heavy is one-engine-out tolerant (and not 2- or 3-out tolerant). But what happens to these numbers?
We look at the (26,27) scheme. It turns out that the crossover probability is pfail=0.30%. Above this level, the redundancy actually increases the overall failure rate. Between 0 and 0.3%, the benefit is positive, but negligible at 0% and 0.3%.
So here's the question: what is the failure rate for a single Merlin engine? If it's greater than 0.3%, you have only hurt yourself, reliability-wise, by bundling 27 of them together even if you can tolerate a failure. If it's better than an RS-68, you benefit by bundling, but only by a small amount.
So these are the preliminary calculations you would do to start a reliability analysis. The preliminary model assumes binary choices (engine good / engine fail), independent probabilities, firm assumptions about k, etc. The next steps relax some of these assumptions — what if some two-engine-out scenarios are tolerable (e.g. one each in the left and right boosters is probably okay, but not two in the middle)? What if some single-engine failures destroy the vehicle? What do we find out if we estimate an uncertainty for the engine failure rate and test the sensitivity to our assumptions (i.e. do a PRA)?
My point is that "more engines is always more reliable" is untrue, and the decision about engine configurations can be made on a rational basis.
@Dick: What a crushing indictment of analytical methods! Considering the millions it costs to put even a Falcon I on the pad and light it, why isn’t it trivially possible – after decades of development and with the strong wind of Moore’s Law at their backs – for the numerical/statistical analysis community to model the relevant reality at a competitive price? Or is this inability confined only to the city limits of Huntsville, AL?
Not sure where you’re going with this, but it is not my position that the choices are dimming the lights with computer models vs flight test. The point is that you need to have validated models for the relevant pieces of your system that you are trying to design. Testing is essential, in particular subscale or full-scale ground testing and possibly subscale flight testing to lead to full-scale flight test. The testing, analysis, simulation, and design processes have to be informed by what we have learned in 50 (or is it 70?) years of launch vehicle development.
FWIW I don’t care where you work, Huntsville, Huntington Beach, Beijing, or Timbuktu, nobody has the capability to do stem-to-stern simulation of all the “relevant reality” of a launch vehicle. We’re not even close… Moore’s law notwithstanding.
@Larry: Computer analysis can only take you so far.
One of my pet peeves is the headlong rush to incorporate CFD and FEM into the design process without adequate validation, or even understanding of the underlying physics. I can’t count how many arguments I’ve had with CFD folks who think the purpose of testing is to prove their codes, who then go on to claim they “design” components with CFD when all they are doing is modeling their guesses at the correct design parameters.
IMHO it is only through the correlation of testing, analysis, and simulation that we come to a validated understanding of the physics of any particular system on the launch vehicle, be it the physics of a pogo suppressor, the dynamics of a stage separation system, the plume forces of a retrorocket, the effectiveness of an anti-vortex device mounted on a tank drain, the impact resistance of a nozzle bell, and on and on. It’s a painstaking process.
@Larry: Errors happen. We have to deal with them as best we can.
If your point is that the imperfectibility of software is an excuse not to exercise effective software control, then I demur.