Spatiotemporal structure and substates in emotional facial expressions

Abstract

AbstractFacial expressions are tools for modulating social interaction, from the display of overt expressions to subtle cues like a raised eyebrow that accompanies speech. How this complex signalling capacity is achieved remains only partially explored. An overlooked factor is the dynamic form of facial expressions signalling, particularly how facial actions combine and recombine over time to produce nuanced expressions and enrich speech. Previous research has focused largely on the static aspects of facial behaviour, reflecting the theoretical and methodological challenges in modelling the production and perception of dynamic social signals. Drawing on theories of motor control we leveraged facial-motion tracking and spatiotemporal dimensionality reduction to investigate the structure and function of facial signalling dynamics. We show that despite the complexity of facial expressions, their emotion-signalling function is achieved through a few fundamental dynamic patterns with subtle yet diagnostic kinematic differences. Classification analysis further show that spatiotemporal components reliably differentiate emotion signals inExpressions onlyandEmotive-speechconditions. The underlying spatiotemporal structure represents an efficient encoding strategy, optimising the transmission and perception of facial social signals. These insights have implications for understanding normative and atypical face-to-face signalling and inform the optimal design of non-verbal expressive capabilities in artificial social agents. This work also contributes new methods for analysing facial movements, applicable to broader aspects human multimodal social communication.Author SummaryIn face-to-face interactions, facial movements play a crucial role in conveying both emotional and speech-related information. Yet, we don’t fully understand how such complex signalling is achieved. Previous research has mainly focused on static signals, such as a single moment during a smile or frown. This study instead explores how facial movements change and evolve over time, particularly in relation to producing communicative emotions. Using advanced techniques to measure and analyse moment-to-moment changes in dozens of facial muscles during facial expression and speech production, we identified fundamental movement patterns that differentiate emotional content. These dynamic patterns enable us to communicate both isolated emotional signals through facial expressions, but also to simultaneously communicate emotions during speech, by adding subtle cues like a raised eyebrow or a brief smile. Our findings have implications to understand typical and atypical face-to-face interactions abilities and could enhance the design of biologically inspired and human-like facial expressions for social robots and virtual agents. Additionally, we contribute new methods for analysing facial movements, applicable to broader aspects of human multimodal social communication.