Recurrent Neural Network to Predict Saccade Offset Time Points from Electrooculogram Signals for Automatic Measurement of Eye-Fixation-Related Potential

Abstract

Eye-fixation-related potential (EFRP)—an event-related potential that is time-locked to the saccade offset (SO)—can be measured without synchronizing with time when external stimuli occur. Such an advantage in measurement enables the mean amplitude of the EFRP to be used to estimate the cognitive workload, which is known to change the amplitude, under real-world conditions. However, to observe EFRPs reliably, the SO timing must be correctly and consistently determined in milliseconds owing to the high temporal resolution of the electroencephalogram (EEG). As the electrooculogram (EOG) is commonly measured simultaneously with the EEG and the SO timing is reflected as a steep change in the waveforms, attempts have been made to determine the SO timing from EOG signals visually (the VD method). However, the SO timing detected by the VD method may be inconsistent across trials. We propose a gated recurrent unit—a recurrent neural network model—to detect the SO timing from EOGs consistently and automatically. We used EOG data from a task that mimics visual inspections, in which participants periodically traversed their eyes from left to right, for the model training. As a result, the amplitudes of the EFRPs based on the proposed method were significantly larger than those based on the VD method and the previous automatic method. This suggests that the proposed method can prevent the decrease in EFRP amplitudes owing to the inconsistent determination of the SO timing and increase the applicability of cognitive workload estimation using the EFRP in real-world environments.