Facial Stress and Fatigue Recognition via Emotion Weighting: A Deep Learning Approach

Abstract

This research addresses the gap in direct facial expression-based detection of complex emotional states like stress and fatigue. We propose a novel methodology employing a weighted summation of basic emotion probabilities, outputted by deep learning models, to calculate continuous stress and fatigue scores. Crucially, these emotion weights are empirically justified and grounded in established psychological and neuroscientific literature. Evaluating CNN, hybrid (DDAMFN), and Transformer-based (ViT, BEiT) architectures, our results demonstrate the superior performance of Transformer models, particularly ViT, in aligning with human-annotated ground truth data for stress and fatigue. ViT achieved almost perfect Cohen's Kappa (kappa = 0.81) for stress and substantial (kappa = 0.72) for fatigue, validating the human-relevance of our emotion-based formulation. This study highlights the effectiveness of Transformer architectures and literature-informed emotion weights for direct and accurate stress and fatigue detection from facial expressions, paving the way for real-world applications in monitoring and well-being.