Authors:
Sebastian S. Feger, Felix Ehrentraut, Christopher Katins, Philippe Palanque, Thomas Kosch
Airplanes are among the most common modes of everyday transportation in many countries and rural areas, sometimes preceding cars or trains. This part of aviation, characterized by nonprofessional, noncommercial civilian pilots flying to conduct routine tasks or for pleasure, is referred to as general aviation (GA). In the U.S. alone, around half a million GA pilots fly close to 200,000 GA aircraft, generating more than $150 billion in annual economic activity. At the same time, and in stark contrast to commercial aviation, GA contributes a far larger number of accidents per hours of flight than any other aviation branch.
Public repositories like the U.S. NTSB Aviation Accident Database point to pilot error due to inexperience, misinterpretation, and heavy workload as some of the most common causes of accidents. This is not surprising, given the limited number of hours pilots are required to log for maintaining their flight permissions. In the U.S., for example, GA pilots are only required to take a biannual flight review to maintain their basic flight privileges. In addition, GA planes, often single-piston aircraft with two to six seats and an average age of 38 years, are still being used in the U.S. In many cases, a moving map on a tablet strapped to the pilot's leg or the airplane yoke is the only or best digital flight support that GA pilots get in their single-pilot operations. This is also reflected in the current state of interaction research on the GA cockpit, which is years behind consumer technology interaction research. One prominent example is a recent study of touchscreen interaction in the cockpit [1].
Insights
→ General aviation pilots lack much-needed technology support.
→ HCI researchers and practitioners can offer unique perspectives and methods to improve general aviation safety.
→ Opportunities include AI for decision support and support for crew resource management.
→ Barriers include accounting for a diversity of users, certification challenges, and missing domain knowledge among researchers.
Technological innovations and pilot support are driven mostly by the world's few large commercial aircraft manufacturers, which drive evolutions in interactive technologies internally or through cooperation with their regular suppliers, technology innovators, or external academic partners. These developments, however, largely ignore the practices and needs of GA pilots, as usability, dependability, and safety constraints differ widely. In this article, we identify a set of challenges, barriers, and research opportunities for HCI researchers and practitioners to apply their wide range of tools and experiences to the underrepresented research field of human-centered GA support. We are confident that such a reflection will be effective at providing starting points for valuable research that, through the design of supportive technology, improves the training and routine of GA pilots and helps mitigate single-pilot workload and avoid overload. We are confident that a deeper involvement of the HCI field could drastically contribute to the reduction of incidents and accidents in GA. In the following, we advocate for HCI research and practice in GA through two distinct lenses: 1) opportunities and 2) challenges and barriers, as well as possible solutions.
Opportunities
The opportunities to support GA pilots through modern technology are manifold. For this article, we identified three of the most important advancements in technology that depict the spectrum of possible interventions.
In many cases, a moving map on a tablet strapped to the pilot's leg or the airplane yoke is the only or best digital flight support that GA pilots get in their single-pilot operations.
Augmented-reality support in general aviation. First, we want to present the case for augmented-reality support in general aviation. AR holds the potential to increase and transform pilots' situational awareness in ways similar to the past introduction of GPS and maps. Today, most GA pilots carry a tablet that depicts the airplane's location, rendering navigation through visual references and dead reckoning with a physical map largely obsolete. Such practices are mostly limited to flight training and emergency procedures where technical failure is simulated. Yet few modern GA pilots would navigate into unfamiliar territory without their tablets or preinstalled navigation screens. After all, few of us would want to drive a car in an unfamiliar region without navigation apps or onboard navigation systems to guide us.
While navigation systems in GA significantly improve situational awareness, we argue that common screens are limited in their presentation of relevant information in the three-dimensional space that is relevant in aviation. Think about a common scenario: an air traffic controller asking a GA pilot to report that they have their destination airfield in sight. Visually identifying an airfield can be a challenge, even if the pilot knows the precise direction and distance, both reported by the air traffic controller and their nav system. Further distinguishing the different intersecting runways is an additional challenge that might affect both the pilot's workload in this critical approach situation as well as early planning of the approach pattern. Here, AR applications can support pilots by visually highlighting and annotating the objects in question. Airfields or other airplanes represent examples of objects of interest that might be shown to the pilot. Others include, but are not limited to, relevant traffic in the vicinity, airspace limitations, and approach patterns. In summary, AR holds the potential to effectively augment the 3D space to increase pilots' situational awareness and to lower their workload. Yet current solutions are mostly limited to the display of basic aircraft performance data or simple turn advisories [2].
AI for decision support. Artificial intelligence is transforming our everyday interactions with computer systems, yet AI as a support tool for GA operations has remained mostly unexplored. This is surprising, given the potential for automated decision support. Take engine-off emergency landings as an example. In contrast to common multi-engine commercial flight operations, engine-off landings in single-engine GA flights are rather common. Providing pilots in such emergencies with immediate decision support information about feasible alternative landing spots could save lives and reduce fatal accidents in GA. Such a system would take a number of contextual features into account, including the aircraft's current position and altitude, weather conditions, pilot experience, terrain conditions, obstacles, and bystander threat assessment. Xavion (https://xavion.com/) is a strong example of such a support tool that takes some of these factors into consideration in choosing a suitable runway for emergency landings. Yet such systems also pose new research questions concerning the effective communication and visualization of information to pilots in stressful situations. After all, the pilots must make the final decisions. Along those lines, AI could further detect and classify maneuver errors based on multisensory input processing before most humans would notice their mistakes, and way before those errors manifested in emergency situations. Early stall recognition through neural networks is just one example [3]. Such systems should then propose immediate countermeasures to the pilots. One example is a spin, a critical situation that is the result of a yaw-aggravated stall, requiring a precise response for recovery. But understanding how to communicate information that pilots trust under stress is a key research challenge.
 | A pilot testing interaction with augmented en-route flight information through a mixed-reality headset in a high-fidelity Diamond DA40 simulator. |
Crew resource management (CRM). The previous two sections related to AR and AI decision support represent exemplary strategies of how future assistive systems could support GA pilots, reducing the workload of pilots by supporting them in visual search tasks and maneuver decisions. Generally, resource management is of vital importance in any flight operation. HCI research can affect resource management in the cockpit by systematically exploring how pilots interact with modern digital assistants and technologies that enhance human capabilities and perceptions. We argue that this research thread remains valuable even beyond single-crew flight operations. GA includes operations with complex aircraft and specific conditions that necessitate the joint operation of two crew members. Here, HCI's methods provide an opportunity to not only study and improve traditional crew resource management and interaction with current and future cockpits. The interplay between human crew members and assistive and augmenting technology will also pose research questions related to information sharing and information management in complex and critical situations in GA and beyond.
Derived from CRM, the concept of single-pilot crew resource management (SRM) applies to the operation of GA aircraft with a single pilot. This concept's 5P approach to enhancing safety—plan, plane, pilot, passengers, programming—emphasizes the value of efficiently using the equipment on board to support the pilot in situational awareness, navigation, and decision making, and also stresses the need for intuitive tools.
Challenges and Barriers
Modern technologies and tools provide wide-ranging opportunities to transform GA pilot workload and operation, as well as the overall safety in this key branch of aviation. We emphasize that we see HCI with its methods and tools at the forefront of this transformation. Yet we also note several challenges and barriers that need to be addressed. In this section, we provide an overview of challenges, and suggestions on how to handle them in practice.
Account for diversity in GA. Foremost, we note that GA is characterized by a degree of diversity, in terms of both pilot demographics and flight operations, that is unmatched in other aviation branches. GA maneuvers are conducted by hobbyists and professionals from ages 16 to 80-plus. While pilots are required to meet certain physical conditions, GA restrictions do not impose very restrictive limitations on height, weight, eyesight, fitness, and other physical capabilities. Pilots have different professional backgrounds, differing expertise in the interaction with technology, and fly between zero to several hundred hours annually. Designing for this diversity is a key challenge that is unmet in other aviation branches, including military and commercial aviation. This is also key to understanding why technology transfer from these specific professional branches into GA is challenging. Head-up displays and augmentation have been used in military jets since the late 1990s, but success in the military domain cannot act as a predictor of success in the heterogeneous GA landscape. We can imagine this in the exemplary context of AR glasses. Today, most AR headsets are still heavy and cause fatigue during prolonged usage. Technological advancements are likely to improve their usability for the cockpit, but we still need to map the capability of pilots with strongly differing physical capabilities to use such devices during an entire flight. We argue that this aims at the core of human factors research, demonstrating once again the responsibility of HCI to drive positive change in GA.
Addressing conflicting properties. Embedding new technologies in GA is different from embedding new technologies in mass-market products. These new technologies must not only support the efficiency and user experience of pilots but also demonstrate their dependability and, above all, their positive impact on safety. This is an unusual challenge in the field of HCI that requires multidomain knowledge in design teams to ensure the reliability of the deployed system. Here, user experience might increase valence and arousal while usability will favor the reduction of pilot errors. Dependability and security will favor prevention and removal of errors affecting the system covering both nonmalicious and malicious faults. These two types of properties thus need to be taken into account in an even way, but HCI methods and processes mostly focus on the human-related ones.
AI could detect and classify maneuver errors based on multisensory input processing before most humans would notice their mistakes.
A design process requires certification. Certification authorities such as the Federal Aviation Administration and the European Union Aviation Safety Agency grant approval for any equipment being part of an aircraft for which a failure might induce minor, major, hazardous, or catastrophic consequences. For instance, equipment integrating augmented-reality technologies for which a failure might have catastrophic consequences (e.g., loss of life) must demonstrate that it will not exhibit a failure in less than 1 billion hours of flight. Certification specifications document what has to be performed by the aircraft or equipment manufacturer to be certified. Most of these specifications come with acceptable means of compliance, providing concrete examples on how to reach the certification objectives. There is still limited practice in the field of HCI to invent and design interactive technologies that are compliant with the constraints imposed by certification authorities, even though other domains (e.g., healthcare and automotive) are moving toward the creation of prescriptive certification authorities.
Simulator-focused studies. The actual study environment represents another major challenge. Testing our prototypes in actual flight operations will be dangerous and unethical in most scenarios. Researchers and practitioners faced similar issues related to automotive systems design and evaluation. In response, most labs researching in this field acquired driving simulators or created partnerships with the automotive industry. We note that largely the same possibilities exist for aviation research in general, and GA in particular. In fact, aviation has always been driven by simulated experiences. Pilots in commercial aviation have to regularly enter the simulator to train for standard and emergency situations and to revalidate their licenses. And even GA pilots training for their private pilot license can perform some of the required minimum flight hours on a qualified simulator. Such setups may consist of a gaming computer, a screen, a yoke, a few controls, and software like the qualifiable version of X-Plane. Most labs will have few issues acquiring such basic components. In terms of software, a copy of the current off-the-shelf Microsoft Flight Simulator 2020, with its strong API, will be sufficient to answer most research questions experimentally. The ease of access to hobbyist resources is a strength of GA research that lowers the barriers drastically compared with commercial or military aviation. Further, we want to stress the opportunity to explore emerging mixed-reality and virtual-reality technologies as means to create even more-realistic study environments. The use of such technologies might even further lower lab-setup requirements. A recurring problem in simulated environments is the underrepresentation of environmental aspects, such as turbulence, which deeply affect interaction technologies. Accounting for disturbances due to the environment is thus another challenge for the HCI field.
 | A consumer tablet with a navigation app is usually the most advanced technology support that general aviation pilots have. |
Missing domain knowledge. Related to the above-mentioned automotive simulators, we note that most HCI researchers are likely to have experience driving a car, or share a fundamental understanding of it, lowering the barriers to engage in research in the automotive domain. The same might not be true in the aviation context for most of us. Numerous universities have institutes specializing in aviation, flight systems, and aerodynamics; however, they are often part of a widely different branch in the university's organization than most human factors and interaction design groups. Based on our experience, searching for such groups and engaging in a collaboration profits both sides extensively. Finally, we note that local flying clubs and academic flight schools aiming to make their aviation branch a safer space represent potential accessible partners who are often eager to share their knowledge and resources with researchers.
Conclusion
We hope that this article stimulates a discussion around the opportunities and challenges for HCI to design for a safer general aviation. Further, we hope to spark interest in a series of curated conversations that we plan to conduct in the near future. These discussions will also revolve around questions of creating dedicated scientific outlets for work published within this domain with the goal of increasing the safety of aviation through the design of future supportive systems in general and GA in particular.
References
1. Cockburn, A. et al. Turbulent touch: Touchscreen input for cockpit flight displays. Proc. of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2017, 6742–675; https://doi.org/10.1145/3025453.3025584
2. Haiduk, P.M. A flight guidance display format on smart glasses for private pilots. Aviation Psychology and Applied Human Factors 7, 2 (2017); https://doi.org/10.1027/2192-0923/a000119
3. Saniat, T.S., Goni, T., and Galib, S.M. LSTM recurrent neural network assisted aircraft stall prediction for enhanced situational awareness. arXiv preprint arXiv:2012.04876 (2020); https://doi.org/10.48550/arXiv.2012.04876
Authors
Sebastian S. Feger is a computer scientist and postdoctoral UX researcher at LMU Munich. He holds a U.S. private pilot license and pushes research on interaction design in and outside of the cockpit to contribute to greater safety in general aviation. [email protected]
Felix Ehrentraut is a manager for innovation and flight simulation special projects in the Lufthansa Aviation Training group, where he explores the newest technologies to maintain the highest standards in pilot training. As he is also a pilot, his activities focus on safety and transfer of knowledge in both general aviation as well as commercial aviation. [email protected]
Christopher Katins is a Ph.D. student and HCI researcher at the Humboldt University of Berlin. His research focuses on mixed reality and its applications in the cockpit of civil aviation aircraft. [email protected]
Philippe Palanque is a professor of computer science and head of the Reliability System and Software research department at IRIT. He researches multimodal interaction and pilot support in civil and military aviation. [email protected]
Thomas Kosch is a professor at the Department of Computer Science at the Humboldt University of Berlin. He is researching AI-driven mixed-reality interfaces that extend and augment human cognition. [email protected]
Copyright held by authors. Publication rights licensed to ACM.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2022 ACM, Inc.
Post Comment
No Comments Found