Looking a lot like SpaceX is going to get the flag at the ISS this year. And between this, the 737 Max, and SLS, Boeing is having a really bad year.
63 thoughts on “More Boeing Delays”
Comments are closed.
Looking a lot like SpaceX is going to get the flag at the ISS this year. And between this, the 737 Max, and SLS, Boeing is having a really bad year.
Comments are closed.
The FBI has started a criminal investigation into the 737 MAX, and the DoJ is issuing subpoenas. That’s aside from the usual investigations over an air crash or safety issue.
As I mentioned last night, the FAA is going to make Boeing scale back MCAS’s control authority so that it can only add a little bit of down trim, and they’re going to require it to use redundant flight sensors. If adding more sensors to its inputs requires a wiring change, and especially if it requires a change to wiring harnesses, that may keep the 737 MAX on the ground for a while.
Last night I was punching in some spreadsheet numbers on thrust, AoA, and moment arms (by eyeball absent real data) to see how much extra pitch torque the plane might have to cope with, and how much down trim should be required.
Sub-degree trim compensation should probably handle things at low altitudes, so given what they actually implemented, I think they have some serious exposure. It seems more like an afterthought or a band-aid. “If the AOA is high, run the trim motor to see if it helps, and keep trying if it doesn’t.” That does not rise to what I would call “aeronautical engineering”. They should have calculated where the trim should optimally be under the present flight conditions, relative to some trim position start pointing based on loading, and either automatically tracked that position, or moved into it once when the AoA crept up, and then returned to the original trim position once AoA was back within normal cruise ranges.
The MCAS system also automatically advances the throttles to full power and tries to hold them there to avoid a potential stall, but the system was originally required because it was the high power settings that tended to make the aircraft pitch up into a high AoA, which is what the system is supposed to prevent. Apparently part of their fix for the problem was to do more of the thing that causes the problem. Maybe that’s the best answer there is, or maybe the system should leave the throttles alone and just focus on reducing the AoA, so there’s less going on to confuse and distract the pilot.
I’m wondering if their top controls engineering teams were assigned to the 777X and other more glamorous or heinously difficult development projects, whereas yet another 737 upgrade seems a pretty cut-and-dried affair that shouldn’t require the same resources.
Michael Crichton could have written a whole book about this one.
Yeah, but if the angle-of-attack sensor is stuck or otherwise sending a bad signal, can your fancy control theory do any good?
This criminal investigation sounds like a way to get either FAA or Boeing officials to “clam up.” I think it is better to figure out what went wrong and then worry about indictments later. When no one is expecting it.
I wouldn’t think it would cause a problem. What they’re correcting is a slight tendency to pitch up, and they’re correcting it only because the pilot might not notice his angle of attack and make a small correction. So even if the minor tweak is wrong, it still wouldn’t crash the plane.
What the current system can do is intermittently engage and run the trim to the travel limit, resulting in an unrecoverable condition.
What Boeing needs is a way to get these planes in the air by next Wednesday. I can’t believe that significantly changing the software won’t require reworking manuals, pilot training and especially emergency procedures.
How would this MCAS system affect a go-around. Close to the ground, usually bad weather, often treacherous winds, what could go wrong? You don’t need a good pilot when everyting’s clear and calm, and everything’s working right.
At the same time, How did a difference that was so small that it wasn’t necessary to train the pilots require an entirely new system to correct. In other words, they designed a plane and then decided that a pilot couldn’t fly it.
Sorry (but not really surprised) to hear of the CST-100 delay. It’s bad for Boeing and bad for NASA. It also proves how SpaceX continues to deliver faster (as well as less expensive) results than the competition. How much more battering can Old Aerospace withstand?
I have a soft spot for the CST-100 because the capsule reminds me of Boeings original proposal for the CEV project. (I’ve wondered, how much of their CST-100 proposal was Boeing just dusting off work they had done on their CEV?) That Boeing 4.5m diameter CEV capsule seemed so much more practically sized than the elephantine 5m capsule NASA imposed on Orion. Heck I remember when NASA had the CEV capsule sized 5.5m diameter!
(I’ve also wondered how much of the current NASA NRHO lunar rendezvous architecture was inspired by something like that old Boeing CEV proposal, which envisioned using L-1 rendezvous.)
It’s not a good look for Boeing Space, especially given how much more they bid and how much greater their resources were than SpaceX’s.
But if it is true that SpaceX was *always* likely to be the first to reach the finish line, we also have to recognize that the cumbersome way in which NASA has administered the CCtCAP phase of Commercial Crew has slowed down *both* SpaceX and Boeing. Barring that, I think both contractors would be flying crew by now.
Still, while I’m far more of a fan of SpaceX, I’m still rooting for Boeing to get Starliner off the blocks as quickly as possible. The more crew capabilities we have from U.S. soil, the better, and the sooner, the better. Reliance on Roscosmos cannot end soon enough.
Still, while I’m far more of a fan of SpaceX, I’m still rooting for Boeing to get Starliner off the blocks as quickly as possible. The more crew capabilities we have from U.S. soil, the better, and the sooner, the better./i>
I agree. It would be the first time in history that the U.S. has two different crew spacecraft operational at the same time.
What would be even better would be if Dragon 2 was capable of launching on an Atlas V and Starliner was capable of launching on a Falcon 9. Then no matter what goes wrong with a particular piece of hardware, we would have no downtime while the cause is found and corrected.
Sorry, screwed up the close tag there.
Been there, done that. I really miss the old preview feature.
But, italic or not, I’m in complete agreement about the mix-and-match rockets and capsules thing.
Highest-compensated Boeing employees are essentially lobbyists (who bribe government officials), not engineers (who create things)
Years ago I knew the guy who was the Chief Test Pilot for the Royal Australian Air Force during the 1950s and 1960s. He flew and tested the Canberra bomber, the Australian Avon Sabre version of the F-86, the F104, Mirage 3 etc.
He once told me that if the airplane wants to go in the opposite direction to which you are trying to make it go, just go with the airplane because it is trying to tell you something.
Any of the 5 other systems (one manual, 4 automatic) can fail and run the trim motor at any time. Remedy is to hit the stab trim cutout switches. It is a memory item. The Lion Air crew died trying to find the appropriate section in the emergency procedures manual.
I doubt re-wiring will be necessary to make the MCAS look at both sensors. It switches which sensor it uses every power cycle. i.e When you shut down the airplane and start it up again.
Of course that now makes it TWICE as likely that MCAS will fail due to erroneous AoA sensor input. How do you decide which is correct or do you simply shut the system down? Better would be at some lowish ground speed on takeoff, say 50 knots IAS, check the sensors agree and are in the sane range and then choose one to trigger MCAS.
I should add when the MCAS triggers the airplane is about to stall because you are pulling back on the wheel. MCAS will roll in down trim on the stab which will lower the nose. If you are dumb enough to keep pulling back at that point when the nose has started to drop, yes MCAS will try again. I will point out that at the stall the nose will drop on its own in most aircraft if you hold the stick back. If not you need to add an automatic system to make it behave like that. Next step if dumb enough to keep pulling back at or after the stall is the airplane departs from controlled flight and perhaps enters a spin. Good luck recovering from that if you are dumb enough to get that far. I doubt Boeing ever checks spin recovery characteristics on transport category airplanes even if a Third World crew has ever been trained in spin recognition and recovery.
But, but, what if the angle-of-attack vane is reporting a wrong value, you have stall warning blaring and the stick shaker is going like a hardware store paint mixer, and the trim wheels keep rolling forward to make the nose drop, your airspeed indicator is reporting a crazy low speed, and your lying eyes are telling you that you are not nose up?
Yeah, you have a electric trim runaway, and reaching down to flip those switches on the center console should be a drill trained into you, not something to start paging through the book of emergency checklists.
As for the MCAS delivering enormous gobs of downward trim, the story I heard is that the flight-test team found this was needed in some “corner” of the flight envelope, it wasn’t the engineering team deciding this. Furthermore, that corner of the flight envelope should never be encountered, and if you do, the claim is that you need that much correction to get the nose down and recover.
All of this is predicated on the angle-of-attack vane not going stupid. Another theory floated on the “Prune” web site “PPRuNe.org” is that something might be messed up with electric lines connecting both the AoA vane and a pitot tube nearby supplying airspeed data?
SpaceX has certainly had its delays with Dragon 2, but indeed it does now look like Drangon 2 will fly manned first. Actually, if the schedule holds, Dragon 2 will fly manned before CST-100 flies at all.
As for Boeing, it’s indeed having a rough month, with the 737max, SLS, CST-100, etc. But, the good news for them is that the aforementioned issues are diverting attention for other scandals, like the Air Force grounding all Boeing KC-46 tankers because they were delivered with trash inside. (The AF also suspended deliveries)
https://www.reuters.com/article/us-boeing-usaf-tanker/u-s-air-force-grounds-boeings-kc-46-tankers-over-debris-issue-idUSKCN1QI58R
Looks to me like Boeing is batting 4 for 4 in a bad way this month, and also that CST-100 and SLS are the least of its problems.
One of my frequent anti-SpaceX sparring partners over at Space News and Doug Messier’s Parabolic Arc is a former Boeing test pilot who has been openly scornful of SpaceX’s choosing to build StarHopper, and now Starship, out in the open at Boca Chica because of all the windblown sand and other FOD that will get into the rocket. Not at all like the standard aerospace practice of building everything in nice clean giant hangars.
If one smelted all the iron in the Boeing-related irony on display the last couple weeks, one could probably roll enough stainless steel sheet to construct the entire armada of SH-Starships Elon will be building over the next few years.
Well, being scornful of SpaceX is hardly a new posture for Doug.
No, it isn’t. But in this particular case, I was referring to Robert G. Oler, who, along with you and I, is among frequent commenters at Doug’s site and at Space News.
He is a bit unstable. *twirls finger at temple*
Well, he’s a long-time adherent of the cult of Boeing. As the late, great Robert A. Heinlein said, “It’s a pretty serious thing when a man’s religion fails him.”
Robert G. Oler used to “hang around” Rand’s fine site. Mr. Oler (are we allowed to call him that or are we supposed to use a military rank or honorific?) is entertaining to read at first, but his persistence in the online discussion can get old pretty quickly.
Rand, um, pulled the plug by banning them from here. It takes a lot for Rand to do this (thank you Rand for being so patient with me over the years!). You see a lot of people you wish Rand would ban, but it takes a lot to reach that threshold.
Didn’t know that. Perhaps he took the hint anent other comment venues. At Doug’s place, Space News and Jeff Foust’s The Space Review, he’s a bit arrogant, dismissive and cocksure – that seems roughly normative for maybe a quarter of regular commenters. But he’s no Gary Church.
“But he’s no Gary Church.”
For which, we may all be thankful.
He’s also a global warming loon.
Doug, that is. Don’t know about Oler, who seems a little unhinged at times.
Yeah, Doug and I have had a lot of innings of play over Global Warming[tm] and Climate Change[tm]. That’s at least as much a religion as Mr. Oler’s Boeing worship. And its adherents are as devout and unshakable as any Christian fundy I’ve ever met.
I *love* the idea that they are building ships out on the open. Just like the ones that sail the seas of Earth but these will sail the starry black.
The more I think about it the move to stainless steel was brilliant.
Oh yeah! 1930’s-looking pulp magazine cover Starships! The capabilities of these things are going to be epic. It’s almost too much extra that they’re going to look so good while doing it.
When the automatic system designed to keep you flying, like lion air and the Ethiopian airliner did, you shut the faulty system off and fly the rest of the airplane to a landing. Like the lion air crew did to the accident aircraft the day before the accident. If no-one corrects the problem, nor informs the subsequent flight crews of the anomaly, I fail to see how this can be dropped on Boeing’s doorstep. There is a good reason flight training starts in airplanes with few, if any, electric or electronic flight controls.
It’s hardly ever just one thing, always a chain of mistakes and misjudgements.
Switching between two sensors is a lot closer to Russian Roulette than any sort of sane redundancy.
At some point you either have to trust the pilot to fly the airplane or admit that you have an unflyable airplane. I wonder how close they came to crashing during testing before they “fixed” it with software.
How many Spacex/Boeing/Commercial Crew voodoo dolls are hidden in desks at NASA?
The trash is just the latest KC-46 screw up.
There are a lot of layers to the onion. Airliners almost never have a fatal stall without several other things going wrong, and in its entire history, the 737 has only suffered a few fatal accidents initiated by a stall. All of those seem to be due to a serious loss of situational awareness by the pilot, such as pulling up into a 30 degree climb for no reason at all, or due to things like wildly inappropriate procedures during severe wind shear, or trying not to slam into a volcano.
I can’t find any fatal 737 stall where I think MCAS would’ve helped, though there may be some, as the fatal stalls seem to be initiated in just a couple seconds by a badly disoriented pilot. MCAS is a fix for a stall that slowly creeps up. I could see where pilots might inadvertently stall when they’re doing a go-around in bad weather while disoriented, but more commonly they seem to wrongly think they’re wildly nose up and in response fly almost straight into the ground.
So one obvious question is that if all the airlines just disabled the MCAS system entirely, would a 737 MAX ever suffer a fatal stall that wouldn’t have occurred with MCAS enabled, and if so how often and under what conditions? It’s quite possible the answer is “never”, but other answers are also possible.
One aspect of this 737 Max situation that I don’t get is the reliance on a single sensor for AoA data. AoA seems at least as important a datum as airspeed and I think most airliners have multiple pitot tubes.
Worse, these mechanical trailing vane AoA sensors have been implicated in crashes before. I seem to recall at least one case involving an Airbus plane somewhere in Europe where the things were found to freeze at altitude when water from being pressure-washed got past the seals and pooled inside the mechanism.
It strikes me that there ought to be at least three such sensors on an aircraft and that AoA data should be decided by majority vote. Better yet, some alternative sensor type should be employed instead of, or in addition to, these problematical little vanes.
Maybe a lidar capable of tracking dust particles in the air could also compute their angle of incidence relative to the long axis of the plane? Come to think of it, such a thing could also compute airspeed to back up the pitot tubes – which have also been known to freeze up.
Just spitballing here, but it sure looks to me like AoA/airspeed sensor tech is pretty damned old-school compared to the rest of today’s planes.
You could put piezo sensors all over the place and figure out where the airstream was coming from based on their pressures, but that would likely not work if there is icing.
You could have a little pitot tube or slot spinning on a horizontal shaft, that noting where the maximum pressure occurred in the rotational cycle, but that would probably be more complex and less reliable that the vane.
You could stick some little symmetrical object out on a shaft and use force transducers in X and Y to calculate the direction of the airstream. The shaft itself would probably work fine for that.
You could probably use a set of ultrasonic or acoustic transducers (send/receive pairs) to determine it by comparing sound velocity in X, Y, and Z, but that might also be affected by icing.
You could use a tiny camera to look at a piece of yarn.
Perhaps the sampling method is kind of crude because until fairly recently only test aircraft were commonly equipped with an AOA sensor, often one out on a boom.
I thought of some of the same things you did and discarded them for the same reasons. Even the yarn could get wet and freeze to the hull. The camera aperture could frost over.
Even so, I’ve often wondered why airliners don’t have any hull-, wing- or tail-mounted cameras to show such really useful pilot nice-to-knows as “Gee, engine number two isn’t there any more and the wing is on fire!” or “Damn, the vertical stabilizer’s gone!” The only aircraft-mounted cameras on airliners seem to be nose cams on a few types to let the passengers see what takeoff and landing look like on their seat-back screens. Not the optimal use of the technology IMHO.
Lidars would probably need to be in heated transparent blisters that stay warm enough to shed ice – ice really is a bitchy problem for almost everything related to aircraft design isn’t it?
Still, perhaps the cheap lidars being engineered for mass production and use in driverless ground vehicles could be used, though I’m thinking the ability to see dust particles in the air would probably qualify as a bug, rather than a feature, for that particular application.
Suppose you ignored the ice when it occurred, instead substituting a calculated value based on altitude, temperature (and thus air density), airspeed, thrust, and inertial forces (which let you calculate lift). From lift you can trivially back calculate the lift coefficient Cl=Lift/(1/2 rho S V^2), and from Cl and flap settings you can find AoA all the way up to stall by just looking it up for the appropriate range of Reynold’s numbers. It wouldn’t be surprising if the calculation is more accurate than the little vanes they’re using.
The flight state of the aircraft lets you calculate a value for almost any failed instrument, but the software doesn’t seem to do that. Often the equations, if fed a single bad input, produce an impossible condition, such as flying upwards and flat to the airstream, or producing tremendous lift with zero airspeed. It’s shouldn’t be at all hard to filter out the nonsensical flight states to flag the bad sensors, substituting a calculated value and alerting the pilot to the sensor failure.
Maybe such a system would have some trouble with turbulence, but it should work better than nothing.
I’m all in favor of anything that’s better than nothing. Because anything based on a single breakable sensor is going to be nothing when the breakage happens.
That’s why I tend to favor belt and suspenders approaches where the economics are reasonable. Considering the stakes at issue here, I think your suggestion – which seems to be something that could be implemented as an addition to extant flight control software – would likely be a good adjunct to multiple and/or dissimilar sensor suites in allowing an aircraft flight control system an accurate appreciation of its actual orientation. That applies whether said flight control system is more software or a human being.
The claim is that the MAX with its bulbous engines can get into a pitched up position short of a stall where you can’t easily push the nose down.
In the absence of MCAS, could pilots be instructed to start dialing in down trim, either with the wheels besides the seats or with the yoke thumb switch? That is what Chuck Yeager figured out in the X-1 when he started losing elevator effectiveness crossing the sound barrier. Could pushing hard forward on the stick be made to activate down trim?
If the pilots got confused by the blaring stall warning and by the paint mixer, they would start putting in down elevator, which would quickly drop the nose without dialing in so much trim that they can’t pull out of that dive?
So you are right, maybe they could just disconnect MCAS and condition the pilots in simulator drills to use trim to drop the nose if elevators aren’t doing it?
But maybe that is up against that “new type certificate” requirement that Boeing was apparently trying to dodge?
Well, an earlier comment mentioned that in flight test, they found a corner of the envelope where the full down trim might be needed to avoid a stall. But what the fix has done is drastically shrink the safe flight envelope to not include any areas where the MCAS system might kick in (high AoA), because when MCAS kicks in the plane crashes a lot.
I say that because as far as I can tell, in less than five months the fix has crashed more 737’s than creeping stalls have in 51 years. By that simplistic metric (a better would be crashes per flight hour), the fix on the MAX is 140 times worse than the stall problem on the earlier 737’s. So how bad is the stall problem on the MAX, and if it is so bad why doesn’t MCAS have to operate when the flaps are down? The flaps certainly add a pitch-down moment, but the plane can still reach high AoA with flaps and full power.
If the pilot was actually made aware of how close he is to a stall, there is probably no external system action required because he’ll avoid the stall, even if he has to push pretty hard. A stick shaker should make him aware, but those have also had some problems that led to fatal crashes, blamed on poor piloting or insufficient training. On the French Airbus flight from Brazil, blaring cockpit alarms saying “stall” were ignored all the way from cruise altitude to the ocean. Idiot proofing something as complex as an airliner is really hard.
The flight controls could just start making adjustments, but how those should happen has to be well thought out, and the pilot should possibly be in the loop. I think the MCAS problem might be that they kept the pilot out of the loop because they didn’t want to have to retrain pilots about a new loop – that the pilot isn’t in on.
There’s a lot of complexity to these questions.
What Airbus found on some of its crashes is that in response to the stick pusher, the pilot pulled back against it. That’s a human factors problem where the control system is doing something the pilot hasn’t commanded, and the pilot is naturally trying to retain his control. What the system needs to do is make the pilot change what he’s wanting to do. When you push or pull on someone’s arm, their natural reaction is to resist it and apply counter force. A better human-interface system might instead present the pilot with a banana on the instrument panel and try to get him to reach for it, pushing the nose down as he does so. ^_^
So better training? But in human factors engineering, if what you have to train goes against natural reaction, you’ll still have problems, no matter how much skill the pilot has.
Example: Homer is driving his family around the Springfield loop, looking over his shoulder to holler at Bart who keeps poking Maggie, and he has to shout loudly over the constant nattering his car always makes on a busy freeway. His instrument panel starts beeping at him. He stays focused on Bart because the car beeps and natters a lot anyway. Suddenly his steering wheel starts to shake more and more violently and pulls hard to the left. He spins around and goes white knuckles, fighting hard to retain control and stay in his lane. Obviously the front left tire has blown (which what the beeping was probably about), and he madly scans all the flashing warning lights on the instrument panel. Then he slams into the back of a stalled semi and kills everybody. Yet he’s an incredibly experienced driver, as are we all.
His smart car was desperately urging him to switch lanes, using an ever increasing array of actions to get him to do so. Instead it caused him to lock himself into his own lane and stop looking forwards. In that crisis situation, he might have even yanked the car off the road to the right and hit a bridge abutment as he fought to keep the car from doing what the car’s systems were trying to get him to do.
The first action that really got his attention was the shaking and wheel jerk. Upon instantly realizing he was at risk of losing control of the wheel, he reasserted his control of the wheel, his primary reaction, before he tried to figure out if he’s fighting a blown tire, a broken tie-rod, a computer system failure, a power steering failure, sketchy hitchhiker in the passenger seat, or whether the car was trying to make sure he doesn’t miss his off ramp. All those things were possibilities that he was immediately aware of, at some level of consciousness, but working through all those troubleshooting trees was of secondary importance to retaining immediate control of the situation.
You can have software engineers who will design such a control system for a smart car, and who think their ideas were really brilliant and obvious. “If the idiot driver won’t shift to the left lane, we’ll shake the wheel, and if that doesn’t work, we’ll crank it over anyway.” The trouble is that they’re presenting the driver with an ambiguous situation that could have several causes, and they’re simultaneously overloading him with information and kicking in his natural counter-reactions to any unexpected event. Out of seemingly nowhere, the system initiates a fight with the driver for control of the vehicle, and that fight may be far more of a crisis than what the vehicle’s control system was initially trying to prevent.
Human factors and man machine interface design is complex, and often intuition about it is wrong because we often poorly understand ourselves. We often design a bad human interface and compensate with training. Sometimes the bad interfaces are legacies of using the simplest mechanical solution to a controls problem, such as you see on almost all heavy equipment like fork trucks and backhoes which have over a dozen identical looking levers in front of the operator, and faded, peeling stickers with an arrow graphic to hint at what they’re for. Since all the operators have learned on those old systems, even the newer equipment has the same controls. It’s not optimal but it works, and that’s why we all use QWERTY keyboards.
But as equipment gets more and more complicated, and the decision and failure trees keep branching, clever interface people can cause more problems than they solving by not understanding the total man- machine-vehicle environment deeply enough. One of the ultimate goals of a good interface is to always keep the operator’s mental model of his environment in perfect sync with reality. This includes every critical and important system, even ones that may not be under his control but which should impinge on his decision making.
If we had that ideal system, any of us could glance in the cockpit and almost instantly know everything important about the flight state of the aircraft, as if the flight computer beamed all its state data into another organic flight computer in our heads. Obviously we’re a long way from that. You stare at an old cockpit and you think “Wow. That’s a lot of instruments!” You stare at a modern cockpit and you think “Wow. That’s pretty!” There’s no sign of flight state beaming so far.
A pilot still has to scan instrument by instrument, pick out relevant bits of information, and slot those into his own mind’s flight model. Sometimes the sheer amount of discrete information we want to convey is so overwhelming that we’re forced to rethink our approach. This happened earlier after engineers and pilots realized that nobody could really stay on top of over a hundred discrete instruments in the F-84 Thunderjet, so we started automating and simplifying. The amount of information we can gather that a fighter pilot can make use of is so overwhelming that we’re trying a whole new approach with the F-35’s helmet system.
But it doesn’t have to be about displays. A high performance motocross bike has virtually no displays at all and yet the rider maintains an incredible level of situational awareness. For most vehicles, the real world is the best primary indicator of the state of the real world. That’s why driving works so well.
The world has littered our routes with an infinite number of speed and position references that we see through the windows, and we can navigate through them at high speeds with almost inch precision without even thinking about it.
Unfortunately, in an aircraft those constant, data rich, and intuitive visual references tend to be about five or six miles down, underneath some clouds, and air is invisible. Filling the sky with birds would solve that but create far worse problems. So we display everything deemed important as instrument data or graphics, and then use caution lights, flashing alarms, and bells for anything that gets really critically important. MCAS was not one of those things. It stays in the background and keeps dialing in nose-down trim.
Anyway, what the French also found in studying stall incidents was that some pilots first went to full throttle in response to an incipient stall instead of focusing on reducing the AoA. The higher throttle setting caused more positive pitch, making the situation worse. Airbus was going to address that pilot reaction by focusing on more training and different procedures. That also touches on MCAS, since the system automatically advances the throttles to provide full thrust.
Common to all these flight system interventions is that they’re making the aircraft take a bunch of actions that the pilot may not immediately understand, just like my steering wheel hypothetical. The pilot’s initial problem prior to high AoA was probably distraction, disorientation, or a loss of situational awareness, and the flight control system is suddenly presenting the pilot with a constant stream of new problems he has to figure out. “Why did the throttles just advance? Why am I suddenly having to fight the stick? Why are all these alarms suddenly going off?”
There’s not a big floating arrow saying “Here! This is the initial and primary problem, and all the other cacophony is just the side-effect of it.” The human-machine interface should try to elicit an immediate and correct initial response from the pilot by presenting an unambiguous situation, and only then have him pursue a trouble-shooting tree to figure out the root cause of that situation. And sometimes the pilot’s immediate and correct initial response is to ignore what the control system thinks the problem is because the control system is wrong. So forcing the pilot into an unconsidered response might not be at all wise.
A controls programmer is often approaching things from the perspective of their code, in which they know what a the problem is as they pound out the crit_response() subroutine, since a built-in assumption of the code they’re writing is that it is handling a valid and well specified situation, which is why they’re sitting at a desk writing it in the first place. You can be too confident that your code will be running with good inputs and valid variables, and commanding those outputs is so darn easy. Sometimes it’s all you can do because the system isn’t sufficiently complex to possess its own self-doubt.
Depending on what situations you’re imagining in your head, or are in the specs, you code against those situations, and depending on what is not in your head, you can fail to foresee the more complex consequences you’ve just set up. Cause and effect. Cause and effect.
Sometimes your thinking is too narrow. You’ve got that one train of thought about what’s going to be going on as your software executes, working harder and harder to avoid that stall, and you’re not imagining what your system would be doing under completely different circumstances, such as a constant and invalid high AoA input and a pilot who is off trying to troubleshoot his pitot heater.
And, as a controls engineer, often all the critically bad design decisions have already been made. The overall project is almost complete and they’re dropping things in your lap so you can slap some band-aids over the baked-in flaws. Oh, and you only have six months before heads start rolling.
MCAS has some of the hallmarks of that. If the requirement had started in the early MAX design phase, the aeronautical people probably would have generated plots of recommended stab trim settings for different flight regimes. Or they might have further increased the size of the tail (the MAX already has much more tail area than the earlier models).
But instead the change came from flight testing, and the MCAS system is an iterative tweak and check loop. A teenager in a robotics class would come up with that. “If right wall is too close then bump motor left and wait 10”
Most often it’s the band-aids that don’t get sufficiently deep thought about cause and effect and the complex interplay of the man-machine interface, on top of the complexities of vehicle dynamics.
George, don’t take it personally, but it takes longer to read your answer than those two flights lasted. Yes, somewhere they found a problem the MCAS was designed to solve, but it failed. Lion air refused to fix it and sent the plane up again with a known fault. No amount of engineering will prevent this from being a problem. A computer does exactly what it’s programmers think you are telling it to do. If you and the programmers speak different languages, misunderstandings can occur. In a situation like that, you need the Leroy Gibbs solution to computer issues; remove the power until you can sort out the cause.
Sorry about that, and that comment was excerpted from my five page proto-comment!
This is a topic I really get into because it’s a rich subject.
In a more space related vein, I highly recommend Digital Apollo: Human and Machine in Spaceflight.
To lay the groundwork of its primary topic, it goes back to cover stability and control in aircraft, from WW-I to post-Apollo, and the tension between pilots who want to be in control of everything and the reality that increasingly they can’t. For Apollo, the engineers had to ask lots of deep questions about the pilot’s role in the control loop.
Some of the same fundamental issues come up in self-driving cars. A driver who wasn’t actively in the loop has to re-orient before he takes over manual control, and that takes far too long as a useful way to prevent an accident.
Someday there will probably be an book written about the MCAS decisions on the 737 MAX, based on the usual reports and now investigations from the FBI and DoJ.
Well, book or no, there will certainly be at least two more episodes of Air Diasasters on the Smithsonian Channel.
You really should have a blog. I’d read it.
“On the French Airbus flight from Brazil, blaring cockpit alarms saying “stall” were ignored all the way from cruise altitude to the ocean.”
Not true.
One of the reasons they crashed was because the stall warning turned off when the angle-of-attack was too high. So the pilot pulled back on the stick with no stall warning, then pushed forward, and the stall warning came back on. So they pulled back on the stick and the stall warning went away. And they remained stalled until they hit the sea.
Had they ignored the stall warning, the aircraft would still be here, because they would have pushed the stick forward and recovered.
When you give feedback to pilots that’s difficult to understand or completely counter-intuitive, many of them will react badly.
Air Disasters (Mayday in the original Canadian) is a true gem; part mystery, part engineering, with the occasional survival story thrown in. The first episode I ever saw was the 747 and the volcano; I was immediately hooked.
I expect that both of these crashes will go into a single episode within a year of the final reports being released. If I had to guess right now, based on what I’ve seen so far, it’s pilot error/crew resource management first of all, followed by poor information presentation (intermittent alarms are some of the most confusing), questionable training (by the airlines, by Boeing, or both), poor maintenance by the airlines, and, at the bottom of the list, the MCAS system itself as currently designed.
I saw an interesting idea recently–a “sanity check” system within the flight computer that cross-checks GPS, radar, and avionics sensor data against each other and on a serious/sustained mismatch, warns the cockpit to switch to manual immediately and *then* diagnose each system individually as the crew has time.
As for Boeing’s other flaws of late–I can’t find much in the way of excuses for any of those. The tanker issues in particular should be costing Boeing a fortune, and the taxpayer nothing, rather than the other way around.
The “sanity check” idea is pretty much what George T. and I were getting at above. The problem is that “sanity” is hard to check when the only external input you have is from a single broken sensor that, once failed, is, in effect, “gaslighting” the flight control avionics. One needs multiple comparands in order to do any meaningfully useful cross-checks.
I’ve often wondered, for example, why flight avionics systems aren’t equipped with hi-resolution topographical databases. These have been used in long-range cruise missiles for decades. Such a database, combined with GPS data, would provide a “2nd opinion” on actual aircraft position, relative to the ground, in addition to radar-based ground proximity sensors in helping to avoid accidents of the “Controlled Flight Into Terrain” variety.
As for Boeing’s other high-profile derelictions and failures of late, one can only note that corporations, like human beings, tend to exhibit a life trajectory that features growing infirmity, mental decline and even dementia toward its end. Boeing is a centenarian. Most human beings of comparable age are in nursing homes or hospices.
Boeing dodders on because: (1) it is rich and influential, (2) it has lots of people with strong pecuniary interests in covering for its increasingly frequent lapses and, (3) it has no obvious heirs because the “pilot fish” of item (2) have contrived, over the years, to prevent any “heirs” from living much past infancy.
The school of “pilot fish” screwed up, though, anent Boeing’s space business and the rise of SpaceX. What we need are more SpaceX-like entities to succeed the struldbrugian Boeing in civil and military aviation endeavors. A nice bracing dose of Joseph Schumpeter’s “creative destruction” is badly overdue in the American aircraft/defense industry.
Sanity checks could include lots of data that is actually ignored.
For instance, indicated airspeed can be determined pretty accurately just by having the plane sense the stiffness of the flight controls. Every pilot of a plane that uses pure control cables feels the indicated airspeed as a control feedback. If they’re mushy, he’s slow. If they’re stiff, he’s fast. Degrees of deflection per foot-lb of torque (or psi of hydraulic pressure), after going through some aerodynamics tables and engineering tables, should be pretty darned accurate. With two ailerons, rudder, and elevators, there are already four independent sources for the data.
Altitude can use air pressure, ground-sensing radar data referenced to the GPS map’s position, raw GPS altitude data, and back-calculations from air density, which can be calculated from all the lift and drag formulas that determine the flight state of the aircraft. But there’s also no good reason that an airliner can’t have eighteen different solid-state absolute pressure transducers to measure altitude, since they’re cheap, cheap, cheap.
The same applies to inertial sensors, since the passengers are already carrying about a hundred independent INS system in their cell phones.
There’s all kinds of better ways to do a flight control system, and I’m guessing that what we have now is an “evolved” version of the first way that we could do a basic digital flight control system.
George, I’m entirely down with all that, especially massively redundant use of solid-state angle and rate sensors which were pioneered for video game controllers and migrated quickly to cell phones.
It is precisely because the airliner industry doesn’t do obvious stuff like that that I indict said industry as feeble and senile. The real problem is that the airliner industry is now pretty much just Boeing and Airbus at the top end. The industry desperately needs new blood and new thinking – which it isn’t getting so long as there is no stream of entrepreneurial start-ups keeping the incumbents on their toes as is the case with most other high-tech markets.
We need, if not a legion of Elons, at least a decent reinforced rifle platoon of them.
Getting down to some specifics, your notion about using torque measurements at control surface axes as a proxy for airspeed is a good one. It is closely related to something I was noodling around with awhile back anent automated battle damage self-assessment for combat UAV’s.
After receiving any input indicating possible damage, the drone avionics would quickly wiggle each control surface a bit in both directions and use those cheap angle and rate sensors to keep track of the results as well as compare them to known baselines for an intact airframe.
Something like that would quickly allow a damaged drone to figure out how to remain maximally combat-effective in any physical condition short of utter destruction.
If, whether through redundancy or dissimilarity or both, it is possible to design essentially unfoolable sensors for things like AoA and airspeed, the ultimate solution to these sorts of man-machine interface conundrums is to take the man and interface completely out of the equation and let the machine fly the airplane. In another decade, that will almost certainly be normative for surface vehicles. In many respects, automating flying is easier than automating driving. We use technology to compensate for all sorts of other human limitations. Driving and flying as human-mediated activities seem ripe for extinction as well.
Automated flying is at the stage that when something goes wrong, we haul away the debris and fill in the crater. Maybe some day, but not yet.
I think we’re a lot closer than that.
Autopilots have been flying airplanes during the routine parts of flights for decades, so that’s covered. Many planes have been able to do completely automated landings in zero visibility at ILS equipped airports for a long time too, so that’s covered. Takeoffs are generally far simpler than landings and I have no idea why they aren’t also routinely automated, but it wouldn’t take much to do so, especially compared to landings.
So the whole flight is pretty much either already automated or quickly could be made so for nominal circumstances.
After that, one starts working in the abnormal cases based on both the entire historical record of NTSB and equivalent foreign agencies, plus educated guesses at additional scenarios concocted by experts with “best practices” for each worked out via simulation.
This is called building a subject-matter expert system, the subject matter in this case being flying an airliner.
The “Sully solution” for double bird ingestions near a suitable river would be in there in generalized form along with every other Hail Mary pass that any hero pilot ever tried that either worked or partly worked.
Would I step on an airliner flown by a redundant, fault-tolerant exert system AI that knows in milliseconds what the best thing to try is when some wacky thing happens and which doesn’t have the capacity to freeze in terror, hyperventilate or be distracted by its life flashing before its mind’s eye while it’s doing its best to save me and everyone else aboard? In a New York minute Dude! I hate flying and one of the reasons I do is because I don’t trust mere mortals to do it right when the fertilizer hits the ventilator.
That is true, but a whole lot of the crashes are the result of erroneous data in the flight computer, such as when a radar altimeter messes up and as a result, the computer is happily flying hundreds of feet lower than it thinks it is.
On one flight to Australia an Airbus A330 flight computer just went haywire, with 10 simultaneous faults. Instead of just informing the crew, it tried to kill everyone. Sydney Morning Herald article about the incident. They miraculously got it on the ground and rolled to a stop, and the overspeed and stall warnings were still blaring.
Respectfully, almost no crashes are the result of massive failures of flight avionics. Most crashes these days are traceable to: (1) maintenance derelictions or screw-ups, (2) pilot errors, often due to disorientation or fatigue, or (3) air traffic control errors.
Item (3) is why I also think ATC, as well as piloting, needs to be removed from human hands.
Obviously, aircraft designs that do not include multiply- and/or dissimilarly-redundant sensor suites are not ideal candidates for total human-free flight control automation. But that is not a huge problem to fix on new aircraft designs or to retrofit to existing designs.
It’s been more than two decades now since the world chess champion was a human being. In the interim, machines have also vanquished the best human Go players. Even more germane to the current discussion, a couple or three years back an AI managed to consistently beat a veteran USAF Col. with a quarter-century of experience in simulated fighter combat.
Fighter pilots were dubbed “Knights of the Air” almost as soon as they appeared in WW1. But armored knights on horseback were rendered obsolete by the march of technology. Now, the same is about to happen to fighter pilots. I see no reason at all why the same will not prove equally true for civil aviation pilots.
There are a couple of problems with that.
First is that the airline safety record in the US is now virtually perfect. That means that switching to a totally new system run by AI’s cannot improve the safety record, it can only match it or worsen it. Since any radically new system might have serious teething pains as we work out the bugs, my bet would be on “worsen it”.
Another factor is that although pilots mostly have to do the routine, which makes their job seem like we could automate it, they sometimes have to make critical interventions, often due to multiple failures in sensor/data streams, flight controls, or structures and engines. Often the incidents we do still have are where the pilot doesn’t realize that the input data is wrong. An AI system might be even more likely to waltz down the primrose path.
A better system might be a “WTF!” button in the cockpit that beams every single raw piece of data up to a satellite and back down to a central headquarters, where a team of pilots, engineers, and lots of AI software can analyze it in a couple of minutes and advise the pilot on what to do, or perhaps even take over, just as a shortcut on putting the equivalent expert system inside each plane.
I think after 9/11 such a system was discussed as a way to have Boeing completely take over an aircraft’s flight controls in the event that an ethnic minority with a peaceful religion takes over the cockpit. But arguing against it was the possibility that hackers would just use the system to crash airliners without even paying for a ticket.
And we have quite a few incidents where the passengers alert the pilots to a problem. Would we make an AI system where the passengers back-of-the-headrest touch pad includes a menu where a person can report problems?
>Flight Menu>Problem Reports>Aircraft Systems>Airframe>Page 5>
“Engine cowl has fallen off (Left/Right)”
“Left”
Then of course the AI has to know to ask the other left seat passengers to verify.
“Did the LEFT engine cowl really fall off?”
“Yes”
“No joke?”
“Really. It fell right off.”
“I’ll note that in the maintenance log. Have a nice day and please enjoy your flight.”
I wouldn’t want to be the programmer who has to write the AI routines that are supposed to respond to a fire in the AI equipment bay. I think I would just make it transmit humorous yet horrifying little messages, similar to the Mar’s Rover’s Twitter feed, and try to slip it under the noses of the project managers.
Certainly we can continue to make major improvements on the current systems, with more reliable sensor data that’s better integrated into a cohesive, multiply redundant, and robust flight state model and airspace model. We can do a much better job of data presentation to the pilot, and use really fancy HUDs. We can put a lot more effort into eliminating the need for go-arounds due to missed approaches, which currently lead to about 25 percent of accidents.
But really, what’s the point of making safety improvements at this stage of the game? The Democrats want to eliminate air travel in ten years and make everyone ride choo choo trains.
If we did try to switch to an AI system, most of the people on the project team would be fresh hires out of Berkeley and other schools, and a not insignificant number of them would feel that their real job is to write code that crashes enough airliners to make the public abandon air travel, thus saving the planet and all mankind. Sometimes you have to break a few eggs, and that means using a cattle car as a *cough* cattle car.
And bad motivations aside, would you get an an airliner flown by an AI that was written by people hired from Facebook, Google, Apple, and Twitter? I sure wouldn’t. I’m sticking with Pilot Bob.
There are always potential edge cases that will not be survivable unless Jesus H. Christ, himself, steps into the cockpit and takes over when the it-shay hits the an-fay. I can’t prove it, but I think a properly developed pilot AI, fed by a properly redundant sensor suite, would likely save more of the crazy-hard-but-salvagable situations than would human pilots. Perhaps we shall both be so fortunate as to live long enough to find out for sure. Here’s hoping.
You are correct that there is a great deal of institutional inertia favoring keeping the future a modestly upgraded version of the present. But I think the rapid proliferation of low-speed, low-altutude, low-observable drones at the low end and the about-to-arrive era of exponentially expanding ground-to-space and space-to-ground traffic from a steadily expanding list of sites is going to prove impossible to integrate into the existing ATC paradigm. The only practical solution will be automation at all levels.
I like the idea of real-time, high-data-rate links between civil aircraft in flight and ground-based repository/assistance centers. I’ve thought, for quite awhile, that relying on after-the-fact data from FDR’s and CVR’s – if and when found – is almost medieval. As soon as Starlink and OneWeb enter service, in fact, I would like to see such a thing made mandatory as the primary such system with the on-board FDR and CVR demoted to backup status.
As to where the AI’s needed to replace pilots, ATC’s and such would come from, I share your misgivings anent the recent crops of coders at social media platform companies. But modern AI’s aren’t algorithmically programmed down each conceivable branch, they’re designed to be taught and to form their own conclusions based on “experience.”
So, no, I don’t think wet-behind-the-ears types with equally wet ink on their diplomas are who I would trust with such work. Who I would trust are guys (and gals) who’ve got a few years or more of experience with some outfit like – well, SpaceX.
I think that would work, but only once we also replace all the passengers with computers, too. Hardware fails. Some vehicles are good targets for complete automation, but some are not. Ones that immediately stop or roll to a stop, like trams, escalators, elevators, and perhaps some trains can and often are fully automated.
Fully automated cars will never really work because of road hazards. Sometimes a cow or toddler just wanders out. Sometimes a piece of debris rips up the underside of the vehicle. Sometimes tires just fly apart. It’s a hazard rich environment, full of wind, rain, black ice, oil spills, fog, and sometimes people with bad intent, perhaps including that giant injury law firm that advertises on your local TV. Recall what nearly wiped out the US light plane industry. Lawsuits over accidents. When an automated car runs over a toddler, the only pocket to sue is the manufacturer.
As was said about automating airliners many years ago, when a pilot crashes there’s a long chain of problems that occurred. When an automated plane crashes it will probably be by flying straight into a mountain on a clear day, and the public will never accept that.
The other problem with a fully automated airliner is that, as with cars, mechanical things break. Engines fly apart. Aluminum fatigues. When the totally unexpected happens, you need a problem solving brain up front, especially one that really, really doesn’t want to die. If an automated system suffered a double engine failure climbing out of LaGuardia, would it set down in the Hudson with no casualties, or would it just beep a lot as it crashes into apartments short of a runway?
If airlines put ejection seats in the cockpits many people would start taking the bus. We’re trusting the pilot to get himself there safely, and we’re just along for the ride. That’s because fundamentally, hurling through the atmosphere at 550 mph at 33,000 feet, while nibbling on a biscuit, is absolutely insane. Sometimes we don’t appreciate that.
If an automated system suffered a double engine failure climbing out of LaGuardia, would it set down in the Hudson with no casualties, or would it just beep a lot as it crashes into apartments short of a runway?
That’s a brilliant summation.
I’m reminded of the decades-old joke about automated airliners. I don’t recall the whole thing, but the punchline had the robotic pilot’s voice saying “Nothing can go wrong–can go wrong–can go wrong.”
That joke is so old that anyone who has never heard vinyl records won’t get it.
See my comment to MCS above.
Preach it, Brother.
We already have automated flying. The entire UAV/drone industry.
No. We have long-distance pilotage by remote humans. Not the same thing at all.
There’s nothing wrong with MCAS or the 737 MAX. Those crews were going to die whenever they next had a runaway stab trim motor for any reason including actuation by any of the 4 automatic and one manual methods that can run it.
This whole thing has become a disgusting, irrational political witch hunt complete with grandstanding politicians and agencies.
Just brief the crews on the MCAS and if it or the other 5 systems trigger or remain stuck on, hit the stab trim disconnect switches.
The autopilot is one of those systems so a crew can be fat, dumb, happy and complacent with autopilot on at FL350 and suffer a runaway trim because some passengers moved forward or aft and the autopilot drove the stab trim motor and it didn’t turn off. The airplane will get upset pretty quickly, either up or down.
The crews in the crashes just didn’t understand the aircraft systems well enough to take the timely and appropriate action which is meant to be a memory item.
But the crews weren’t going to die without MCAS. All the prior 737’s had all those other systems that adjusted the trim, and all the pilots, like these, were trained how to respond to runaway trim by hitting the TRIM STAB cutout switch. In almost 50 years of flying, 737’s hadn’t been crashing from runaway trim, ever, so far as I can find in the accident lists. Third world pilots flying 737 NG’s instead of MAX’s aren’t crashing due to runaway trim, and those are the same pilots that just transitioned to the MAX, which they are crashing.
And they’re crashing frequently. The MAX is crashing every four months and they’ve built less than 400 of them. If 737 NG’s were crashing at that rate, we’d be losing one a week.
The problem is that MCAS doesn’t manifest as runaway trim until the NTSB finds the jack screw in the debris. It only runs for a handful of seconds and then stops. So when the pilot is trying to figure out if he has a runaway trim problem, the trim wheels aren’t turning, so the answer is no, it can’t be runaway trim. So he goes on checking something else to get the plane back to a normal attitude, probably focusing on why the engines just went to full thrust.
Shortly afterwards, the wheels turn again, and then stop. He keeps right on troubleshooting, and it’s still not manifesting as runaway trim because the trim wheels aren’t turning. And the MCAS system and the pilot might do this little dance all the way into the ground. As one 737 pilot described it, MCAS is insidious because it acts intermittently.
And if the pilots do figure it out and flip the cutout switch, they’re still fighting full down stab trim, and the elevator doesn’t have enough control authority to compensate for that. No other electrical system can correct the problem because they’ve just thrown the cutout switch to disable the stab trim motor, so they have to use the hand wheels, which have cables running all the way back to the jack screw. But it’s possible they can’t do that either if they’re hauling back on the yoke for all they’re worth, and they might be too busy to even take a look at the position of the horizontal trim, the the ground rushing up at them so dramatically.
Have you missed the point that the lionair accident aircraft the day before the accident tried to do the same thing. The crew disconnected it and flew it by hand. The problem is not the airplane itself, it’s the shortcomings of the crew and the maintenance department at lionair. That aircraft should have been repaired before flight, or grounded until it got fixed.
The crew on the prior Lion Air flight had help from a pilot in the jump seat who wasn’t focused on the instruments, so he could notice the trim wheel turning. The wheel is down on the console, not up in plain view.
And the problem would still exist if the sensor went bad on the runway or in flight. Flight instruments go bad, so pilots train on partial panel approaches and such. A bad instrument, one that wasn’t even on most earlier airliners and isn’t on light planes, isn’t supposed to cause 100% fatalities. People were quite shocked when Aeroperu Flight 603, a 757, crashed because someone left duct tape over the static port, but at least that’s a critical instrument.
Certainly these problems will show up first, or more frequently, in airlines that have poorer maintenance procedures or somewhat less pilot training. But, as it happens, that’s also Boeing’s target market. If Boeing’s products keep boring holes in the ground all over the world, and Airbus products don’t, everyone will switch to Airbus.
A lot of aviation companies left the commercial airline market after a string of crashes created a bad reputation with the flying public.