John Carmack has a new post up describing what he thinks is the reason for the Pixel failure a few weeks ago at the LLC.
Our systems run the propellant valves through a set of relays controlled by a watchdog microcontroller, which drives the valves closed if the main computer doesn’t continuously toggle a keep-alive signal bit. We found that with the valves on the bench, we could get interruptions to the current when we tapped on the relays in the electronics box. We believe that acoustic vibrations from the operating rocket engine could cause the contacts on the relay to intermittently lose contact.
A couple points. First, this demonstrates the need to test components in their operating environment. But more importantly, these are the kinds of things that you learn only with a lot of experience and testing, something that NASA can’t afford to do with their vehicles because a) they throw them away after each flight and b) they’re as expensive as hell. That’s why reusable vehicles, and approaches like Armadillo’s and XCOR’s hold the key to both low cost, and reliability, goals that NASA will never achieve with their current approach.
The critical task here is to persuade the check writers (Congress) that they shall benefit from facilitating low cost reliable access to space.
Arguments that such developments will benefit America or humanity should be but shall not be sufficient. Therefore, what needs to be demonstrated is that seeking low cost reliable access to space shall enhance their individual re-election prospects. Good luck with that.
I also assert this is equally true for Democrats and Republicans, hence non-partisan.
Failing that, locating non-taxpayer sources of funding is the route to follow. Which is why Armadillo (and many others deserve our support).
Actually, because of this post, I’m off to buy some Armadillo logo gear (CafePress) for my holiday gift giving needs. That little Armadillo mascot is cute and I have nieces and nephews who will think its cool.
Armadillo wants to solve the problem by using relays with stronger springs rather than going solid-state. I’m not an engineer but I wonder whether this will make trouble for the whole Pixel concept of rocket modules that can be used in any combination. A relay might be strong enough to work in some Pixel configurations (such as stand-alone) but couldn’t it encounter a different level of vibration in a different configuration? In that case, every time we start using a new grouping of Pixels won’t we have new questions about relay reliability?
If you could launch pixel from the middle of knowhere , constrained by physics, then this particular component would not be needed at all.
A safety system whose job is to kill the vehicle as a last resort has to be really reliable.
Bill,
I have little hope that NASA will be “facilitating” low cost access to space if that threatens jobs in Texas or Florida. What I do hope is for an organizational change that shifts how the money is spent. Letting engineering teams have more control over their budgets, membership and goal objectives, with less central command, would be a really big improvement.
You know, there used to be a concept associated with mechanical switches and computers called “debouncing”. You can do it in hardware or software (I think you can find an example in the Apple ][ ROM)
Bruce,
There still is. Anyone using a rotary encoder to count pulses and direction of rotation by the human operator has to do it, to give just one example. You are correct it can be done in hardware, software or both.
Debouncing applies to inputs to your computer. I got the impression that these relays were used to interface a signal output by a microcontroler to a valve. The contact bouncing from normal relay closing apparently isn’t causing trouble. The relay vibrating open despite no change in the control signal is a problem.
Using a solid state replacement to the relay would solve the vibration problem, but might introduce other drawbacks. If a delay in response is acceptable, a capacitor to compensate for momentary interruptions might be an option.
I’m not sure a mechanical relay is the right solution here. It has the advantage of being extremely inexpensive but it is susceptible to vibration. I’d be worried about the welded contact failure mode too. If they’ve got really strong springs holding it closed, then they either need a lot of current to hold the relay in, or some mechanical advantage which again may be susceptible to flight conditions.
Yea, it was fun to see John C. talk about:“For years, my drive to the Id Software office afforded me an almost daily opportunity to run out fourth gear on the service road.”
I was a student at North Mesquite High School right at the height of the original Doom game. There is a service road that runs right in from the school next to LBJ 635 and goes right in front of the Town East Tower. Several times a week, one could spot John roaring down the service road towards the Tower. Rumor had it that the car was not street legal. And in order to keep the police of his back he, “procured” a new set of bullet proof vests for the entire Mesquite police department.
Needless to say, I was/am green with envy.
I think that NASA testing would have discovered the problem, since they do a lot of qualification on every component, like shake testing, vacuum testing, and so on. It just would have cost 100x more, and some really obvious cases would have been missed, like getting a part installed backwards or mixing up SI and imperial units.