Eye-tracking technology in defense is mainly being used in some form to control aircraft. This includes both manned and unmanned aircraft. In this article, we’ll look at use cases for both scenarios.
Tobii Pro Glasses 2 — Gaze-Driven Drone Flying
Flying a drone (unmanned aerial vehicles) by eye may sound futuristic, but an initiative is underway to make it a reality. When experienced drone pilots are unavailable, less experienced pilots will need to step into the role of drone operation. That is the main use case for gaze-driven drone flying.
Eye controlled drone flying works by tracking the operator’s eye movement using high-definition cameras. The operator wears glasses, which provide a platform for controlling the drone and detecting eye movement. The challenge for controlling a drone by eye is to map 2D eye movements into 3D navigation waypoints — a process that isn’t straight forward, given the lack of depth.
The challenge for controlling a drone by eye is to map 2D eye movements into 3D navigation waypoints — a process that isn’t straight forward, given the lack of depth.
Researchers at the University of Pennsylvania, New York University, and the U.S. Army Research Laboratory (ARL) are involved in the project. The project uses a pair of third party glasses called the Tobii Pro Glasses 2. In addition to eye-tracking cameras, the glasses also have an inertial measurement unit (IMU).
Data from the glasses are fed into an NVIDIA GPU that utilizes deep learning. EyeTracking’s EyeWorks software integrates with the Tobii Pro Glasses 2. The software provides feedback on training and operator cognitive load by monitoring pupils through the glass’s cameras.
“This integration now allows our customers to measure cognitive workload in all facets of their research in any environment,” said James Weatherhead, President, EyeTracking, Inc., in a press release.
BAE Systems’ Fly by Eye
Another eye-tracking wearable in development, by BAE Systems, is the virtual cockpit. The Lockheed Martin F-35 Gen III Helmet Mounted Display System (HMDS) is the most advanced HUD available. It provides the pilot with vast amounts of data in a consumable form. BAE wants to take this a step further by allowing eye engagement of cockpit controls.
It isn’t just eye engaged controls that BAE wants to implement — they want to remove many of the physical cockpit controls and place them in the helmet. Rather than pushing a button by hand, the pilot will gesture in combination with eye tracking, and the system will understand what the pilot wants to do. The goal is to eliminate any unneeded extra activities in the cockpit. BAE calls their system a virtual cockpit.
In this setup, the physical cockpit will be virtually void of displays and various controls. All of them will be moved into the helmet.
“In such a world, we need to think about what controls are critical to the pilot and then make them easier to manage,” Jean Page, a lead technologist with BAE Systems’ Human Factors team, said to The National Interest. “Eye-tracking gives you the option of looking at something to highlight it and then making a gesture to ‘press’ a button, rather than having a series of physical buttons on the aircraft.”
BAE Systems will bring the most important information to the pilot, based on eye movements. Using eye location and gestures, a specific button will be virtually pushed. Pilots don’t need to look around for specific buttons since any button needing attention will be displayed in the pilot’s field of vision.
The system will use AI to predict the pilot’s intentions. By focusing on fewer controls in a less complex visual representation, the pilot’s workload will be reduced.
Optical Sensing Market Size
A Global Market Insights, Inc. report shows that the aerospace and defense optical sensing market is expected to grow at an annual rate of 8.2% from 2016-2024. In fact, a report from Markets and Markets shows the defense and aerospace industries will make up the majority of the overall optical sensing market through 2023. The overall market is expected to grow from $1.2 billion in 2016 to $3.46 billion by 2023.
Eye-tracking technology in defense is removing the complexity associated with certain military systems. In the case of a shortage of drone pilots, the technology is allowing less experienced pilots to step in when needed. Eye-tracking technology is also being used inside the cockpit to display only what’s needed in a pilot’s helmet. The end result reduces the pilot cognitive workload.
For aerospace and defense, the market for optical sensing technology has plenty of growth ahead of it, which should lead to more operator enhancing breakthroughs.
Specifically, for aerospace and defense, the market for optical sensing technology has plenty of growth ahead of it, which should lead to more operator enhancing breakthroughs.
