Jump Seats and Flight Tests, Part 2

Jump Seats and Flight Tests, Part 2

This installment for my weekly blog is the second part of a two-part story.?

Not long after I returned to Seattle from my trip to South America, I found out about an opportunity for several engineers to work on-site at the Honeywell facility in Deer Valley just North of Phoenix. I jumped at the opportunity as I thought it would be a great experience and a nice change from the Seattle rain. The 777 program was operating under a concept called “Working Together”. Part of this concept was having customer airline representatives working on-site at the Boeing facilities in the Seattle area, while in turn having Boeing personnel embedded at various suppliers to Boeing. The intent was to improve communications and foster a spirit of teamwork between Boeing and its customers and suppliers.

Another benefit for me personally was that I would be paid per-diem on top of my salary while working on Phoenix. This effectively doubled my take-home pay, which was very enticing!

I was scheduled to move down to the Phoenix area in March of 1991, and Boeing provided a generous amount of time to make the drive down, so I planned out my route to visit several family members that lived in California on the way to Arizona. I first drove directly from Seattle to Riverbank California near Modesto where an aunt and uncle on my mom’s side live. I enjoyed a couple of days visiting them before I headed over to Orinda California, passing through Livermore (where the national laboratory is) on the way. My cousin Joe, who I had traveled to South America with just a few months before, lived there. He was a Civil Engineer and owner of a construction contracting company that did home remodeling, home construction, and other cool projects.?

Joe showed me a barn that he had converted into a home that had wound up on the cover of Architectural Digest magazine. He was proud of that achievement, and I was suitably impressed with the work he was doing. He also took me to the site of Larry Ellison’s estate (Oracle CEO) where he was involved in the construction of multiple buildings and a cool stone bridge for which he had done the civil engineering. It was an amazing compound that was under construction there, and I had the privilege of receiving a personal tour of it from Joe.

Joe also decided to take me on an excursion to Fisherman’s Wharf in San Franciso. We parked in a nearby parking garage and made our way to a nearby stairwell. As we descended the stairwell, I noted that there was a lot of graffiti on the walls. Graffiti on public buildings has always been one of my pet peeves, so I commented to Joe “I wish people wouldn’t put graffiti on buildings like this”. Joe, being liberal minded, responded “well, when the Indians were here they could write on anything they wanted”. Not wanting to argue, I just let it go and didn’t respond to what I felt was a silly statement.

We spent an hour looking around Fisherman’s Wharf, had lunch, and watched seals that hang out on floating docks. We then made our way back to the parking garage and ascended a different stairwell this time. This stairwell reeked of urine. Joe commented “I wish people wouldn’t pee in public places like this”. To which I replied “well, when the Indians were here they could pee wherever they wanted!” The look he gave me was precious as he knew I had hit him with a zinger!

After visiting with Joe, I drove down the coast on highway 1 through Big Sur to LA. I stopped to see Hearst Castle near San Simeon which is a very cool place that was built by William Randolph Hearst on a ridge overlooking the Pacific Ocean. After touring it, I continued down the coast where I stopped to stay and visit with my cousin Mike who had also been on the trip to South America. Mike was studying to become a Podiatrist and was in the midst of a Physics class as one of his science requirements. He gladly accepted my help as a tutor, and I was able to help him learn the concepts and pass the course during my short stay. He and his wife went with me to see the movie Silence of the Lambs which had just come out, and that still sticks in my mind as one of the most intense movie experiences I have had!

I arrived in Phoenix and started off staying in a hotel before finding an apartment in Paradise Valley near the mall. I was assigned a desk in the new building at the Honeywell Deer Valley facility where the 777 project was being engineered, along with another Boeing engineer who was focused on the AIMS cabinet while I was focusing on the AIMS Display System portion. We were supposed to get involved in all the engineering processes at Honeywell, but initially the two of us were treated like aliens from another planet by the Honeywell engineers as they were clearly afraid of us seeing more than we were supposed to. Clearly this was not going to work if we weren’t able to get involved! I went and spoke with Paul Milo and Scott Rausch, two of the engineering managers for the AIMS system, and they assured me that they would deal with the situation so that we could get involved. About a week later at an all-hands meeting, they introduced the two of us, explained that we were here to help them, that we were to be invited to all the engineering team meetings for our functional areas and included in pretty much everything. They also told that that if they had any concerns about anything, to come talk to them about it.

With that problem solved, we were off and running, and I was now invited to the Display Systems engineer meeting, each of the Display ASIC design meetings, the LCD technology meeting, the fluorescent backlight design meeting, and all the other engineering design related activities. The Honeywell engineers quickly realized that I was a great information resource, as I could either answer their questions about the intent of requirements right off the cuff, or quickly contact the proper person back in Seattle to get a quick answer for them. This made me a valuable member of their teams. It also gave me a greater level of involvement and visibility in the AIMS Display hardware, system, and software designs than any of their own engineers who had divided responsibilities.?

I wrote a weekly report I titled “Operation Desert Display” in homage to Desert Storm, which I sent to Len Hardaway and Randy Robertson back in Everett. I made a deal with Paul Milo and Scott Rausch that I would allow them to read my reports before sending them which gave them more visibility into what was happening and what issues I had identified. If they saw something that concerned them, I would take it out of that week’s report and wait until the following week to include it. That gave them ample time to investigate and mitigate problems that I had flagged. When I reported it a week later, they were fully prepared to discuss the issues with Boeing management and look good in the process. It was truly a win-win approach and allowed me to be very effective at fostering open communication, problem solving, and teamwork both within Honeywell and between Boeing and Honeywell. Truly what the Working Together concept was all about.

Because I was in all the Systems engineering and ASIC design teams, I was able to observe things in a different way than anyone else. While reading through the ASIC design documentation, and following the graphics pipeline from document to document, I discovered a problem that would only occur when the redundancy management function had placed the pipeline into a ? update rate mode of operation. There was a serious problem that would occur as defined in the requirements. I have since forgotten the exact details, but I brought it to the attention of the Honeywell chief display technologist, Dick McCartney, who initially disagreed with my assertion. I slowly and carefully walked him through the documents and the scenario when suddenly his face lit up with an ah-ha moment and he saw the problem! He quickly called a meeting, and they were able to fix the issue in the requirements before they had committed the ASICs to silicon.?Dick was awarded a lot of patents for his work at Honeywell including one on LCD Response Time compensation which is now used widely in LCD TVs.

During my year in Phoenix, I put my tax free per diem money to good use and began flight training at the Deer Valley airport. I obtained my single engine land private pilot rating and began to make regular flights around Arizona. Flying to Sedona was a regular favorite. I took one of my friends from Boeing, a DER named Robert Nowak, on one of those flights. Robert had a very stressful job as DERs carry a lot of responsibility. Robert was a very likeable man from Wisconsin with a midwestern charm about him. He was also one of my friends and part of the “AIMS In Crowd”.

Derek Deloe came down regularly for business meetings, and he and I planned a hiking trip in the Grand Canyon in November of 1991. We spent our first night in a hotel on the South Rim and got up the next morning to several inches of fresh snow. We headed down the Kaibab trail with constantly changing spectacular views, and gradually left the snow behind as we descended into warmer air. We went from temperatures in the 30s at the top to temperatures in the 70s at the river. We crossed over the Colorado River on a suspension bridge and made our way downstream to where the Bright Angel creek joins the Colorado River from the North side. We located our reserved campsite in the campground there and made camp. We had also made reservations for dinner at Phantom Ranch, a short way up Bright Angel creek where we enjoyed a steak dinner! After camping under the stars, we packed up the next morning and headed back up to the South Rim along the Bright Angel trail, which gave us more unique and spectacular scenery!

Living in the Phoenix area for a year had been a great time for me, and I enjoyed many things while living there. I returned to Seattle in early 1992 where I began working on tasks related to the building and testing of the 777. I wrote the factory functional test for AIMS Display system which is used by mechanics on the assembly line to test all the interfaces. I did a good job as I only had to sit in on the first test for the first and second shift in the factory on the first 777 on the line, and they told me that it was fine and didn’t need me to come out there anymore.?

I also spent a lot of time in the Boeing 777 Systems Integration Laboratory along the Duwamish River not far from the flight test center at Boeing Field/King County Airport. There we tested the AIMS Display system using several engineering benches that were equipped with full ship sets of AIMS avionics. We followed regular test scripts to ensure that we covered the various requirements. Dave Green and I focused on the redundancy management, brightness control, and other hardware and system related functions such as the Synoptics while other engineers responsible for functions such as the Primary Flight Display format, the Engine Indication and Crew Alerting System, and other instrumentation related functionality. Having the knowledge of the design that I had obtained from my time in Phoenix was invaluable during this period.

I did a couple of ad-hoc tests that weren’t on the test plan, but that I felt were important to conduct. One of these was to test the effect of the loss of cooling air on a Display unit. When a Display Unit lost cooling air, it would begin to overheat, and the electronic components inside were supposed to be protected from damage by a bi-metal thermal switch. I tested this switch and confirmed that both captain’s side DUs would switch off (and back on again) as they heated up, cooled back down, and heated up again when they turned back on. I confirmed that they did indeed work. I restored the cooling air, but within a few days, both DUs suffered hard failures and stopped working. I reported this and was somewhat chastised by my management for causing the damage. The DUs were in limited supply, so replacing them would take up two of the precious spares. Secretly, the Honeywell systems engineers applauded me though since they really wanted to know what would happen in this scenario but weren’t allowed to test it. I had collected the actual temperature data from the DUs that was sent out on the ARINC 429 serial bus, so I had documented how hot they got before they turned off, and that it was too hot to prevent permanent damage and failure. This data was valuable and allowed Honeywell to lower the thermal set point of the bi-metal switch by about 10C to better protect the DUs and avoid hard failures.

Another ad-hoc test I conducted was for the redundancy management system. Each of the two AIMS cabinets had Core Processor Modules (CPMs) with Graphics Generator client cards (GGs) attached to them. Each GG had a coaxial data output cable that went to a 1 to 6 splitter so that each DU could be driven by any of the four CPM/GGs in the two AIMS cabinets. To test the effect of a poor connection and the resulting high bit-error rate, I took a small wrench and loosened the coax connectors on the inputs and outputs of the splitters in various combinations. I observed how the DUs were reassigned to other GGs when they lost contact with the one they were supposed to be connected to. I also power cycled DUs to see how they handled noisy connections. While doing this, I discovered that I could make a DU lock up with a black screen and not recover, a huge no-no for the redundancy management scheme! I wrote up a problem report in the problem reporting system and submitted it.

Several weeks went by, and this problem report was not resolved. Many of the others that had been submitted for the display function had been, but this one lingered. Dave Prescott, the Honeywell system engineer responsible for the redundancy management function, reached out to me, and asked if they could just close it because they couldn’t reproduce it. I told him that I wasn’t willing to close it because it was a real and serious problem. I was headed down to Phoenix for meetings the following week, so we agreed to address it while I was there. Dave pulled me aside during a break in the meetings and had me go with him to one of their engineering benches. They had rigged up this bench with electronic switches that could interrupt any of the coax connections to and from the splitters. He demonstrated how the displays would not go black when they cut a connection this way and power cycled the DUs. I told him to please get me a small wrench and to hold my beer. I then proceeded to loosen a couple of the output connectors by just the right amount and power cycled the DUs, within a couple of attempts, viola! Black displays! After enjoying the resulting oh-shit look on Dave’s face, he ran off and grabbed the engineer responsible for the DU input controller ASIC and we demonstrated the problem to him. He puzzled a few minutes, and then realized he knew what the likely cause of the problem was. The input ASIC was looking for its ID bit pattern to determine which input to receive its display data from, but a noisy input could fool it into locking onto a channel on which the CRC codes for the display format data would always fail, keeping it perpetually blank. Fortunately, they had an FPGA on the front end of the ASIC that allowed them to make the ID source selection function more robust and solve the problem.

Sure enough, not long after this, we began to receive reports of displays going black and staying black after power cycles on the factory and flight test airplanes. I knew the cause, and I worked with Don Kenkman to get a torque specification added to the manufacturing process for connecting the coaxes to the splitters to ensure that they were properly secured. Had I not uncovered this problem by ad-hoc testing, it would have been a mad scramble to determine what the problem was!

As flight testing began, I was able to participate in several of the 777 flight test flights. I recall certain moments from several of them. On one flight I was invited up to the flight deck by the test pilots who knew me from several of the human factors tests they helped us with and allowed me to sit in the jump seat and enjoy the view of fall foliage as we cruised along the Oregon and Washington coastline. On another I brought along a light meter that I used to measure the forward field scene brightness to compare to timestamped data from the remote light sensors to see if they were properly calibrated. I determined that they were not because the calibration wasn’t considering their hoods that constrained their field-of-view to match the pilot field-of-view out the windows and were reading too low. I submitted a problem report and got this resolved so that the Automatic Brightness Control function would work properly.?

I sat in the back on one flight as we descended below the canyon walls of the Grand Coulee valley, watching cars at eye level on the adjacent highway as we dipped to within 50 feet of lakes to test the radar altimeters. I also was sitting watching the engine instruments on an approach into Moses Lake airport when suddenly the RPM readout of the left engine dropped to 0 and the left engine began to spool down. Then just as suddenly the RPM reading returned, albeit to a reading that matched the now slower engine speed, and the engine began to spool back up. Shortly after that both engines spooled up to full power and Buzz Nelson came on the PA and announced that we were aborting our approach and heading back to the flight test center to evaluate the engine problem. It was later determined that one of the extra flight test instrumentation wires to monitor analog engine RPM that are only installed for flight test had been left dangling and had momentarily grounded out. The engine controller had cut off fuel to the left engine as a result, but then restored it after the fault cleared on its own. I think someone probably got chewed out for not properly securing the wires!

On one flight test that was otherwise boring and uneventful, I showed up, and took my customary seat in the back in front of DUs that were repeaters of the ones installed in the flight deck.?While sitting and waiting for longer than normal to depart, someone came out of the flight deck and asked if there was anyone from the AIMS system onboard. I said yes, and he ushered me into the flight deck. I found out that they couldn’t dispatch because too many of the AIMS cabinet I/O modules were offline. I knew that this was the result of a memory leak problem that would eventually lock them up if they were left running for multiple days without resetting them. I began to instruct the mechanic to pull the breakers for both AIMS cabinets, but he replied that he had already tried that. I calmly explained that I needed to walk him through the procedure and asked him to please follow along. He obliged while clearly frustrated and pulled the breakers as asked.?I then asked him to pull the battery bus breakers to the AIMS cabinets as well and wait at least 15 to 30 seconds before pushing them back in, and then to push back in the main breakers. He then confirmed on the Maintenance Access terminal that all the I/O modules were now properly functioning, and we were able to dispatch, which clearly made him and the test pilots happy (the pilots had been watching in amusement as I showed the mechanic how to fix the problem).

I showed up at work the next day at the Everett office where my manager, Len Hardaway, announced that I was a hero for saving fight test! I was somewhat surprised by this, as I didn’t think what I had done was very significant, since most of us working on the AIMS system were aware of the memory leak problem and the solution that we regularly employed on the test benches. I had never been called a hero before!

The last time I sat in a Boeing jump seat was on a flight in 1994 in August as I flew from Seattle to Amsterdam to meet my cousin Pat and his wife to travel around Europe for two weeks. The Martinaire charter flight was on a Boeing 767, so I went to the Boeing gift store and bought a bunch of 767 swag including stickers and other fun stuff. Onboard the flight I handed a bunch of swag to one of the flight attendants along with my business card. Several hours into the flight, the flight attendant returned to tell me that the captain had invited me up to visit the flight deck. When I arrived there was a Dutch first-class passenger who was also apparently a private pilot standing in the doorway chatting with them and marveling at the 767 flight deck. I patiently waited for the conversation to end, and he left to return to the seat. I stepped into the doorway and introduced myself.?The captain then motioned for me to sit in the jump seat and promptly shut the door to the flight deck. It didn’t take long for the grilling to begin! “Do you see how the control column blocks my view of my EICAS display??Same for the First Officer! I have to look at his, and he at mine!” I was aware of this issue and explained how on the 777 the DUs were arranged side by side so there was no control column obstruction. “Do you see these three landing light switches? They go Left/Right/Center. But look at the taxi light switches! They go Center/Left/Right! What is up with that?!” I acknowledged how stupid that was and explained that on the 777 they went Left/Center/Right, Left/Center/Right. “That jump seat you are sitting in swivels, right?” I confirmed that it did indeed swivel. “Turn it to face that chart table to your right!” I did so and whump! The seat hit the side wall preventing me from turning all the way to face the chart table. “What is up with that!” “Yes, that is really stupid” I replied! On the 777 the seat turns without any problem, and instead of a chart table, that is where the Maintenance Access terminal is located so you can inspect the health of all the onboard systems and determine if dispatch requirements are met! I then went on the offensive and began to tell them about all the cool new features of the 777 flight deck. After many minutes of regaling them with the coolness which pervaded the 777 flight deck, they were suitably impressed and abandoned their efforts to beat up on the Boeing guy. After I ran out of features to share with them, the First Officer got a really petulant look on his face and declared “we won’t ever get to fly the 777 because Martinaire is too cheap to buy them!” I resisted the urge to fall out of my seat laughing and maintained my composure like a consummate professional!

After a while the conversation lulled, and I kept my eyes on the displays watching our progress on the moving map. The captain turned back toward me and said, “hey, take a look out the windows!” I looked up and the solid undercast that we had been flying over had disappeared and below us spread a glorious panorama of Greenland with its ice sheets and massive glaciers that wound through rugged mountains to the East coast where numerous ice bergs were floating in the sea. Framed in the 767 fight deck windows it was truly spectacular, and I regretted not having brought my camera from the bag that was in the overhead compartment at my seat! It would have made for a great flight deck poster.

In spring of 1995 I left Boeing to go work for Micron on Field Emission Display technology. I have many fond memories of my Boeing days, and the people that I had worked with. Over time many people come into our lives and then later leave because of circumstances. I worked with many bright and interesting people during my tenure at Boeing. Many of them are no longer living. Our lives are temporary, and none of us get out of here alive, so we need to enjoy the people and places we experience while we can.

In memorium:

Phillip Fenner Sept. 22, 1936 ~ Jan. 13, 2012

Robert "Robin" McKee Merrifield July 15, 1938 to August 11, 2012

Edward Delanty April 12, 1934 to October 6, 2009

Derek Deloe July 16, 1960 to August 21, 2012

Joseph Wilkinson February 26, 1952 to September 2002

Robert Nowak January 20, 1962 to May 28, 2011

Donald Kenkman March 21, 1932 to September 6, 2011