Emotions and Recognition Tools

?

?

The recognition of emotions is a crucial factor in many aspects of life and in many scientific fields.

For personal relationships, professional relationships, fields such as psychology, and in areas of medical science such as psychiatry, the recognition of the emotional state of a person has always been a crucial point.

Beyond the above, the ability to accurately identify emotions predicts behavior and assesses a person's reliability within a specific context.

Initially, there is difficulty in defining the concept of emotion, determining its origin and the mechanism of its generation. Another important question is whether the expression of emotions is universal.

Regarding the origin and mechanism of emotional creation, Darwin argued that emotions are the product of human revolution and supported the universality of emotions' expressions on the face (Darwin, 1872).

Furthermore, Matsumoto and Ekman (2009) claimed that emotions are:

'' transient, biopsychological reactions designed to aid individuals in adopting to and copying with events that have implications for survival and well-being.'''

On the other hand, a constructional theory supports that emotions are the product of the interpretation and labeling of our reality in a given situation (Schacter and Singer, 1962).

Another one approach, Component Process Model (C.P.M), suggests that emotions are a dynamic process that evaluates specific goals (Scherer, 2009).

Whatever position someone adopts, the importance of recognizing emotions and, thus, the emotional state of a person under given circumstances remains major.

The decoding of non-verbal communication can contribute to the recognition of emotions. Nonverbal communication represents all range of communication except spoken or written words (Knapp 1972; De Paulo and Friedman 1998).?Decoding body movements, facial expressions, and voice characteristics can help in this direction.

Although facial expressions are one of the most complex signal systems, researchers converge on the view that they are reliable signals of emotional state (Ekman 2003).

Facial expressions and their connection to specific emotions were first mentioned by Darwin, who argued that they are common to humans and animals, are universal, involuntary, and are connected directly with certain emotions (Darwin, 1872).

Darwin's specific research went unused for many years as, at that time, the field of psychology was also trying to be recognized as credible science.

Many years later, interest in the study of emotions returned, and much research began in this area (Tomkins 1962, 1963; Ekman 1972; Izard 1971).

Going one step further, attempts were made to develop methods that would allow the observation and the objective categorization of facial expressions and their direct association with emotions.

In this direction, the Facial Affect Scoring Technique (FAST) was developed by Ekman, Fisher and Tomkins in 1971, as well as Maximally Discriminative Facial Movement Coding System ( MAX) by Izard in 1971.

Both systems were designed to measure facial movements that was considered relevant to emotions in a pre-hypothesized theoretical base. Any other activity on the person’s face was ignored.

That means that both systems were operating according to what was expected to happen on the face during a particular emotional state, based on a specific theory of how each emotion was expected to appear.

The main advantage was time-saving as not all movements had to be coded but only the predefined ones. However, they had narrow scope and objectives.

Additionally, not all facial movements are directly related to emotions, and it was observed that facial muscles can produce over 10,000 facial expressions (Ekman, Scherer, 1972)

For these reasons, FAST and MAX are considered systems unable to produce reliable, acceptable and useful results.

Thus, the need emerged for a scientific, objective, and commonly accepted way of measuring and coding facial muscle movements that would not necessarily be related to emotions. It would simply record and categorize all possible facial movements regardless of their theoretical origin.

The Facial Action Coding System (FACS) was developed by Ekman and Friesen (1979) to serve this purpose. FACS is not a system for coding emotions but provides a taxonomy of the face.

Its main purpose is to identify only visible facial movements and ignore nonvisible or subtle ones. The next step is to categorize them using an anatomically coding approach.

This means that the system observes, records, measures, and categorizes the visible activity of the facial muscles, with the expectation of providing a scientifically accepted basis for discussion in this field based in anatomy.?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????FACS suggests 44 distinct Action Units (AU).

Each visible activity of any muscle or combination of muscles of the face is called an action unit (AU), described accordingly, and numbering.

It is important at this point to mention the difference between Action Units and Muscle units.

There is no direct connection and correspondence between the Muscle unit and AU.

A Muscle Unit (for example, the Frontalis, a pair of muscles) produces more than one Action Unit, AU 1, AU 2, etc.

As the system does not provide a direct association of AUs with specific emotions, any theoretical association is made after observing and ranking the activity of facial muscles. ??????????????????????????????

The system cannot be used for real-time coding. On the contrary, FACS is used for the frame-by-frame analysis of a recorded video or using photographs under the condition that there are other photos to determine a baseline for the persons.

Briefly, the logic and description of the classification are as follows.????????????????????????????????????????????

???Top-side action units

AU 1 is considered as Inner Brow Raiser and refers to the part of the muscle that causes this particular movement.

AU 2 is considered as Outer Brow Raiser and refers to the part of the muscle that causes this particular movement.

Lower face action units

AU 12 is considered as Lip corner puller and refers to the muscle that causes this specific movement.

AU 5 is considered an Upper Lip Raiser and refers to the muscle that causes this movement.

And so on.

In addition to the muscles movement descriptions, FACS suggests several sets of permanent movements and positions of the head and eyes.

Some of these are as follows.

AU 51 and AU 52 describe the situation in which the head is turned to the left or right, respectively.

AU 53 and AU 54 describe the condition in which the head ascends or descends accordingly.

Issues of facial visibility and eye movements are also considered and coded.

AU 70 encodes situations where the forehead is not visible, and AU 71 encodes situations where the eyes are not visible for any reason.

FACS also classifies motion intensity of visible head muscle movement using a 5-point scale as A, B, C, D, and E, where A is considered as trace motion and E as maximum intensity motion.

Finally, the ranking examines to which part of the person's face the action occurred.

The final scoring format is as follows:?

1C+ R2A + 4B scoring can be described as AU1 movement for intense intensity, AU2 on the right side of the face for trace intensity, and AU4 for slight intensity.?

Referring to the application of FACS, the following factors are taken into consideration. Time, duration, onset, peak, intensity, and offset of movement that occurred.

FACS has found use in many fields but also shows promise in many others.

It was originally developed for lie detection and reliability assessment and for use in the security field. Today many technological applications and commercial products for real-time automatic coding are based in FACS. It is also widely used in animation, graphics, and video games.

FACS is also used in the medical sector.

Referring to the diagnosis of neuropsychiatric disorders, as abnormality in facial expressions and inappropriate and flattening facial affect are known characteristic symptoms of schizophrenia (Walker et al., 1993), the identification and classification of facial expressions are considered vital.

In addition, FACS is used in stress analysis models through visual inspection by experts trained in the system (Giannakakis G. et al, 2021).

A real revolution in the coding and analysis of facial expressions occurred with the introduction of the use of technology. Facial expressions are recorded by a camera and analyzed by computer software later or even in real time.

Computer systems have made it possible to analyze huge amounts of data in a short time, more systematically, accurately (Colmenarez et al. 2004), and broadened the potential uses of FACS.

In addition, computers can detect even micro or subtle facial expressions that the human eye might miss.

As expected, computers use predefined mathematical models based on algorithms that correlate AUs to emotion or emotional state (Giannakakis G. et al., 2021).

Although the above process is free of bias and more accurate, it is based on theoretical assumptions about the correlation between emotions and facial expressions.

At this point, it is worth considering the accuracy of the assumptions and of course, the reliability of the algorithm that ''produce'' the results. Given that, the initial assumptions that are used must be the result of scientific, reliable, and acceptable research.

As expected, FACS has both advantages and disadvantages.

The main advantage is that FACS provides an objective, scientific coding system for visible facial muscle activity.

It can also be described as non-theoretical, as it does not assume that AU is produced by emotions or cognitive processes. It does not suggest any connection to the processes mentioned, which means that it is free for any use and any interpretation.

The main disadvantage is that it takes a long time to decode and requires skilled and trained staff, and for sure, the process can not be considered as bias-free. Additionally, the training and certification in the system require a lot of time (more than 100 hours).

Also, given that FACS codes only the visible facial muscle movements, what is the meaning of the term ‘’ visible’’? The term refers to a specific vision ability of someone, and how is this measured? Can different abilities affect the observations? In any case, what about the nonvisible actions? Τheir invisibility means that they don't exist or that they don't mean anything?

Moreover, there is a difficulty in coding the dynamics of movements in a precise way, as AU onsets/offsets could take different trajectories.

Furthermore, the measurement of precise intensity may be a case as it is hard to recognize visually the upper or even the lower limit of an AU.

Another point is the interaction between the lower face and speech and how speech movements that can affect the appearance or recognition of an AU.

Finally, ethical issues may also arise regarding the way such systems are used, which should be used according to the law and, in any case, with respect to the personal data of the individual.

There are many future challenges for FACS and the possibility of accurately linking facial expressions to specific emotional states.

Initially, with technology, it is no longer necessary for FACS to be limited to facial expressions visible to the human eye. It is useful to investigate and encode facial micro expressions as well. Then examine if should be different or additional coding for them. In this process, the EMG can help to confirm the observations.

If using reliable and scientifically accepted procedures, and then match even basic emotions such as happiness, sadness, anger, and fear with specific facial expressions, in different contexts, many valuable applications would appear.

For example, in the security field, the emotional state of anger could be linked with potential offending or deviant behavior. The same could apply in the business context for predicting the potential behavior of employees under certain circumstances.

Another interesting use would be to examine, using FACS, the relationship between adults and children in different contexts. For example, how a child or group of children feel about their teacher or for their educators in the institution where they reside. In the same context, it would be interesting for teachers and educators to be trained in FACS in order to come closer to the theory of facial muscle activity and what that may mean concerning children’s emotional state.

Also, training in FACS can definitely be used for anyone interested in increasing his/her emotional intelligence.

Finally, if we could code in detail all -or a part- of the six communication channels (body language channel, for example) suggested by EIA, Lansley, C., (2017) , as FACS did for facial movements, we could have a more reliable tool concerning decoding behavior and emotional state. ??????????????????????????????????????For example, supposing that BM 1 (BM stands for body movement coding) means ‘’arms crossing’’ ?and BM 2 means ‘’lean body back’’, a combination of BM 1 + BM 2 + AU 4 + AU 24 theoretically, would provide a more accurate approach of the intentions and emotions of a person.

?

References

Ekman, P. & Rosenberg, E. L. (2005).?What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS). Oxford, Oxford University Press.

Mandal, M.K. and Awasthi, A. eds., (2014).?Understanding facial expressions in communication: Cross-cultural and multidisciplinary perspectives. Springer.

Lansley, C., (2017). Getting to the Truth: A Practical, Scientific Approach to Behaviour Analysis for Professionals. Sevenoaks: Emotional Intelligence Academy Ltd.

Ekman, P. (1992). Facial expressions of emotions: New findings, new questions.

Giannakakis, G. et al. (2021)

‘Automatic stress analysis from?facial videos based on?deep facial action units recognition’

Institute of?Computer Science, Foundation for?Research and?Technology Hellas (FORTH), Heraklion, Greece