Evolutionary mental state transition model helps machine learning algorithms track emotions

Seeking to improve automatic emotion tracking, which detects and monitors emotions over time, a group of researchers in the field of human-computer interaction decided to approach the task by modeling shifts in internal emotions rather than only interpreting external emotional signals.

Using insights from psychology, they developed the evolutionary mental state transition model, a model that incorporates a mental state transition network. They tested its effectiveness on two multimodal emotion datasets, producing noticeably more accurate results than existing alternatives.

Their research was published on April 8, 2024 in Intelligent Computing.

In addition to accuracy, another advantage of the evolutionary mental state transition model for emotion tracking is its reduced computational time and smaller footprint. The model has fewer parameters than other published models, which makes it “suitable for deployment on mobile devices and robots,” according to the authors.

Daily life applications of emotion tracking include public opinion monitoring, marketing communications, mental health monitoring, and online education. Extensions of the authors’ model could be developed to personalize emotion tracking to take into account individual variations in emotional fluctuation. Work in this direction would build on the psychologically realistic nature of the model, which attempts to capture the “natural dynamics of emotions and their impact on mental states.”

The authors’ system for emotion tracking consists of several steps:

1. Multi-modal pattern recognition based on language, vision, and acoustic inputs
2. Feature fusion in a transformer
3. Pooling to calculate “external emotional energy” (apparent emotion)
4. Determination of actual emotion using a unique mental state transition network

In the evolutionary mental state transition model, language, vision, and acoustic features are first extracted from the data and encoded, retaining their chronological order. Next, multihead cross-attention blocks are used to fuse the features at each time step; this is the most computationally intensive step. Third, maximum pooling and average pooling, two varieties of a common deep learning technique, are used for dimensionality reduction, and the features are transformed into external emotional energy at each time step.

Finally, the mental state transition network is used to take into account patterns in changes in the subject’s emotions over time, as well as external emotional energy, to determine the actual emotional state at a particular time step.

The network was built on a set of probabilities resulting from data previously collected from 200 participants about the associations between different emotion pairs. It predicts emotional state in part by weighing the contributions of multiple simultaneous emotions rather than assuming a subject is experiencing only one.

The performance of the evolutionary mental state transition model was compared with that of a number of baseline methods using classification tasks based on two large datasets, the CMU Multimodal Opinion Sentiment and Emotion Intensity dataset and the Ren Chinese Emotion Corpus. The CMU dataset, consisting of recorded monologues in English, identifies happiness, sadness, anger, disgust, surprise, and fear. The Chinese corpus consists of blog texts, and was used to test the mental state transition network component.

Seeking to improve automatic emotion tracking, which detects and monitors emotions over time, a group of researchers in the field of human-computer interaction decided to approach the task by modeling shifts in internal emotions rather than only interpreting external emotional signals.

Using insights from psychology, they developed the evolutionary mental state transition model, a model that incorporates a mental state transition network. They tested its effectiveness on two multimodal emotion datasets, producing noticeably more accurate results than existing alternatives.

Their research was published on April 8, 2024 in Intelligent Computing.

In addition to accuracy, another advantage of the evolutionary mental state transition model for emotion tracking is its reduced computational time and smaller footprint. The model has fewer parameters than other published models, which makes it “suitable for deployment on mobile devices and robots,” according to the authors.

Daily life applications of emotion tracking include public opinion monitoring, marketing communications, mental health monitoring, and online education. Extensions of the authors’ model could be developed to personalize emotion tracking to take into account individual variations in emotional fluctuation. Work in this direction would build on the psychologically realistic nature of the model, which attempts to capture the “natural dynamics of emotions and their impact on mental states.”

The authors’ system for emotion tracking consists of several steps:

1. Multi-modal pattern recognition based on language, vision, and acoustic inputs
2. Feature fusion in a transformer
3. Pooling to calculate “external emotional energy” (apparent emotion)
4. Determination of actual emotion using a unique mental state transition network

In the evolutionary mental state transition model, language, vision, and acoustic features are first extracted from the data and encoded, retaining their chronological order. Next, multihead cross-attention blocks are used to fuse the features at each time step; this is the most computationally intensive step. Third, maximum pooling and average pooling, two varieties of a common deep learning technique, are used for dimensionality reduction, and the features are transformed into external emotional energy at each time step.

Finally, the mental state transition network is used to take into account patterns in changes in the subject’s emotions over time, as well as external emotional energy, to determine the actual emotional state at a particular time step.

The network was built on a set of probabilities resulting from data previously collected from 200 participants about the associations between different emotion pairs. It predicts emotional state in part by weighing the contributions of multiple simultaneous emotions rather than assuming a subject is experiencing only one.

The performance of the evolutionary mental state transition model was compared with that of a number of baseline methods using classification tasks based on two large datasets, the CMU Multimodal Opinion Sentiment and Emotion Intensity dataset and the Ren Chinese Emotion Corpus. The CMU dataset, consisting of recorded monologues in English, identifies happiness, sadness, anger, disgust, surprise, and fear. The Chinese corpus consists of blog texts, and was used to test the mental state transition network component.