Imitation Learning with Human Eye Gaze via Multi-Objective Prediction

Ravi Kumar Thakur, MD-Nazmus Samin Sunbeam, Vinicius G. Goecks, Ellen Novoseller, Ritwik Bera, Vernon J. Lawhern, Gregory M. Gremillion, John Valasek, and Nicholas R. Waytowich

Interactive Learning with Implicit Human Feedback Workshop at ICML 2023 · Oral

Imitation learning usually records what a demonstrator does, but not what they look at. This paper introduces Gaze-Regularized Imitation Learning (GRIL), which predicts continuous flight controls and gaze coordinates from the same camera image. Gaze is an auxiliary training objective rather than a policy input, so deployment does not require an eye tracker.

The task is quadrotor navigation in Microsoft AirSim. A drone must find a yellow truck that begins outside its field of view, avoid obstacles, and navigate toward it. Evaluation covers both the demonstrated stationary-target task and a moving-target variant absent from the demonstration data.

For this work, I created the gaze-augmented demonstration dataset, designed the evaluation protocol, and ran the experiments.

Learning from where people look

GRIL receives a 224 × 224 RGB image. A pretrained MobileNet, fine-tuned on the demonstrations, produces a shared representation for two heads: four continuous flight controls and a two-dimensional gaze coordinate. The objective combines mean-squared gaze-prediction and behavioral-cloning losses. In the reported experiments, the gaze and action terms are weighted 0.1 and 0.9, respectively.

\mathcal{L}(\theta) = \lambda_1\,\mathcal{L}_{\mathrm{GP}}(\theta) + \lambda_2\,\mathcal{L}_{\mathrm{BC}}(\theta)

GRIL neural network architecture with a shared image encoder and separate control-command and gaze-coordinate output heads — **Figure 1.** GRIL jointly predicts gaze and control commands through a shared convolutional representation. The gaze head supplies an auxiliary objective during training.Figure from Thakur et al., arXiv:2102.13008v3 (2023).Open full-size figure

Flight experiment

The dataset contains 90 stationary-target trajectories recorded with an Xbox One controller and a screen-mounted eye tracker. Demonstrations cover 24 possible starts with randomized headings. Evaluation used 10 held-out starts and five rollouts from each, for 50 rollouts per model and task. Comparisons were behavior cloning (BC), an adapted Attention-Guided Imitation Learning model (AGIL), and BC with a context-aware gaze loss (BC-CGL).

GRIL had the highest completion rate and lowest collision rate in both reported tasks. Uncertainty below is standard error over the 10 evaluation start locations.

Completion and collision rates
Method	Completion	Collision
Stationary target
GRIL	80 ± 9.9%	20 ± 9.9%
BC-CGL	64 ± 14.8%	36 ± 14.8%
BC	40 ± 13.7%	58 ± 13.5%
AGIL	10 ± 10.0%	60 ± 16.3%
Moving target
GRIL	40 ± 12.3%	40 ± 9.4%
BC-CGL	36 ± 14.8%	42 ± 15.9%
BC	30 ± 12.7%	66 ± 12.3%
AGIL	14 ± 10.3%	86 ± 10.3%

The moving-target policies were trained only on stationary-target demonstrations. The test therefore measures transfer within the same simulator and scene, rather than generalization to a new environment.

Scope

The experiments use one demonstrator and one simulated environment; they do not establish physical-flight performance or variation across people, eye trackers, and settings. The paper does not report learning curves across dataset sizes. Its AGIL and CGL comparisons are task-specific adaptations. GRIL could also lose the moving truck after it left the camera view, although it sometimes recovered when the target reappeared.