Who watches the watchmen on software testing? SpaceX's control issue might have been found with better testing, but the test case writer didn't start with a big enough perturbation for the problem to appear. It's also not clear that the tester software is sufficiently good to tease out problems with the control software. That's especially true if the same people are writing the control software and the tester software.

The rest of the entry reads like technobabble from a movie like Failsafe. Nevertheless, this is the $64 billion question that can make SpaceX another of Musk's successes or ground his Mars colonization plans altogether.

There are ways to manage to get high fidelity to desired specifications. One is to have independent testing from designers. Another is to have test plans that are vetted by a second independent verifier. A third is to have multiple independent testers.

Testers can boil the ocean seeking test scenarios. Tests need to hit all regimes that are likely to be encountered, but need to do so economically. A good choice is a fractional factorial design that tests all the regimes for each variable, but not every variable cross every other variable. Deciding what needs to be tested is as important as passing the tests chosen.

It's still a problem if testers are testing the wrong model. If the control software and the test software both have the same error, then there will be a false negative in testing even if every possible scenario is tested.

One thing to do is test the testers by introducing errors into the design on purpose and seeing if the testers can find them. This can give a hint about how many unknown errors there are depending on how many known errors are not found through testing.

This is what I do in my day job at Optimal Auctions for our auction software that has been used to buy and sell over $100 billion in cost of goods sold.

I asked SpaceX this question when I toured SpaceX before their first two launches. I expressed confidence that they were getting this better after their first launch.

I don't see much change in culture with the release of their latest flight review that Rand noted today. Their current culture and methodology may be enough to get them to orbit. With only 8 anomalies they detected with only one fatal, they are in good shape. Actually flying hardware (or in my case holding an auction) can give additional confidence that the test plan accurately models the flight hardware. If they do succeed, flawless results are great for their business but they create new problems; they can also reduce vigilance by the testers.

For testing success, remember Andy Grove's dictum: "Only the paranoid survive."

Musk escalates the value of his approach succeeding to the difference between being a space superpower and taking a back seat to China and India.

This illustrates the limitation of using software to test software. I wonder in the grand scheme of costs how this would have turned out with more hardware testing.

The problem with the error seeding model, of course, is the assumption that the errors you seed in are the same kind of errors that your programmers are making.

It almost seems better if the hardware is cheap enought to simply let it fly and fail and let your people learn operationally as opposed to sitting and executing computer models ad infintium.

I am not an engineer but if I were Mr. Musk, I would have tried and flown the 1st stage with a dummy upper a time or two first.

So, are Dennis and Mike saying that John Carmack was right after all, and the right approach is to crash dozens of vehicles, fly hundreds?

I am certainly a fan of massive hardware testing. I would bet in hindsight that Elon may feel the same. The problems with the first stage flight would have been solved (and hopefully recovered the stage) if they had done a flight with a dummy upper stage. However, there is something to be said for full up testing like that, if you can take the loss of hardware.

I would hope that the spin would be a little more subdued though. I looked on NASAwatch and saw the post about Elon's flight and compared it with the statement about the NRO flight just below it and the irony meter pegged for a moment.

Expendables are expensive to develop mainly because they can't be tested often enough. That fact seems unlikely to change anytime soon.

Lee

The SpaceX first stage is designed to be recoverable and reusable so that is not a factor in this discussion.

Many years ago I worked in structural dynamics analysis and test at Grumman, were my boss had a saying:

"No one believes an analysis except for the people who ran it, and everyone believes a test except for the people who ran it."

I later found out that was a paraphrase of:

"No one believes an hypothesis except its originator but everyone believes an experiment except the experimenter."
-- W. I. B. Beveridge, _The Art of Scientific Investigation_, 1950.

Although Beveridge was writing about science, there is also art in engineering, and devising and executing good analysis and test is not a simple job.

That's why God made us test engineers.

ic1dtr6pwzt eby9su57p6lhvqz [URL=http://www.929316.com/341358.html] dkimcurppsiz [/URL] 2qg86pld24

q66yeelrlbq66yeelrlb bfd5ctplk1 1186636286