Professor Kim McDonough – Professor Pavel Trofimovich | New Insights into Non-verbal Aspects of Second Language Learning

Although most people study a second language in school, this education alone is seldom sufficient to achieve the level of proficiency needed to live in a foreign country. But in a world where global migration is increasingly common, learning a second language has become essential for millions of people. Professors Kim McDonough and Pavel Trofimovich at Concordia University, Canada, study language learning, and in doing so, are helping those trying to become competent in the use of a second language.

The Need to Communicate

Most of us will have seen commercials in which a tourist in a foreign country talks into a phone and an app translates the sentence for the locals who respond the same way. Everybody is understood and there are smiles all around. Unfortunately, the reality is not quite so simple. While technology is getting better at supporting simple conversations about the location of a bus stop or tourist attraction, there are significant limitations to its use. Sentences translated by artificial intelligence can be difficult, or even impossible to understand. Furthermore, for those living in a foreign country using a device for every interaction is simply impractical. In our increasingly mobile world, learning a second language, often to a very high level of proficiency, is a fundamental necessity for millions of people working or studying in a foreign country.

The Challenges of Learning Languages

Anyone who has spent time around someone trying to use another language will have stories about the often amusing mistakes they make. As Professors Kim McDonough and Pavel Trofimovich of Concordia University explain, common errors fall into three main categories: the use of incorrect/inappropriate words, so-called lexical errors; the incorrect use of grammar; and issues with the pronunciation of certain words, a problem linguists call phonological errors.

When such mistakes happen, teachers correct them using a process scientists refer to as recasting. For example, if an adult was to say ‘yesterday I go to a movie’, the teacher would typically correct this to: ‘you went to a movie’. Sometimes, learners will repeat the correct expression, and eventually, they will learn to use verbs correctly.

However, depending on the person’s age and circumstances, the mechanisms for corrections and the learners’ responses to recasting may be different. Extensive research indicates that social behaviours, interpersonal interactions, cultural factors, and attitudes can profoundly affect language learning by adults. The research group led by Professors McDonough and Trofimovich is particularly interested in how subtle non-verbal cues, such as eye contact, facial expressions and hand gestures, aid communication between adult language learners and those interacting with them.   

If in Doubt: Gesticulate!

One of the questions the research group at Concordia University wanted to answer was whether seeing the face and/or the upper body of a person speaking English as a second language influenced how ‘understandable’ they would be perceived to be.

In these experiments, the researchers recorded speech samples of students – half of whom were Spanish and the other half of Chinese background – and asked fellow students (‘assessors’) to rate how easily the speakers in the videos could be understood, how fluent they were, and how strongly detectable their accent was.

To evaluate the effect of facial expressions and hand gestures, the scientists manipulated the recorded speech samples. A third of the assessors listened to the speakers while only seeing a stationary image of their faces. Another group listened to the same speech samples accompanied by a video showing visual clues restricted to the faces of the speakers – such as a smile, raised eyebrow or a frown. However, in the video shown to this group, the shoulders and hands of the speakers were ‘frozen’, thus eliminating the gestures conveyed by the movements of the torso or hands. Finally, a third group of assessors were shown the full video, closely replicating a real-world interaction with the speaker.

The findings were conclusive. The same speaker was perceived as more fluent, understandable, and less heavily accented when their face was moving, and this rating improved even further when upper body gestures and hand movements were also visible. Analysing the recorded images and the assessors’ ratings in more detail resulted in two unexpected findings. Speakers moving their eyebrows more often were perceived to have less pronounced foreign accents, while those who looked away from the listeners more often were judged to be easier to comprehend.

Anxiety and Non-verbal Cues

Predictably, previous research has shown that when speaking English as a second language, students affected by anxiety and other negative emotions performed poorly and became harder to understand. However, this research was carried out using a questionnaire that simply asked students to rate how nervous they were while speaking. To evaluate the role of anxiety more accurately, the scientists attached sensors to the fingers of the students using English as a second language to measure electrical conductance. Since skin conductance changes very rapidly when someone is stressed or experiences other negative emotions, so this allowed the scientists to monitor objectively how nervous the students were while discussing various topics.

Professors McDonough and Trofimovich and the team also explored whether certain facial expressions or hand gestures were specifically associated with periods of anxiety. To identify stress-associated behaviours, they analysed the skin conductance traces while simultaneously watching a video recording of the speakers to determine specifically what actions corresponded with higher levels of anxiety. 

While the finding that anxiety makes speaking a second language harder is not new, what is important here is that the scientists were able to clearly identify gestures and behaviours associated with the students being nervous. For example, periods of anxiety were associated with more frequent blinking, and anxious students averted their gaze and tended to touch their faces and hair more often. In contrast, during more relaxed conversations, speakers were more likely to nod their heads. Interestingly, most students found talking about themselves more stress inducing than discussing more general topics.

Such findings have clear practical value in language education. Recognising stress-related behaviours in students would allow language teachers to helpfully intervene – by changing conversation topics, or by providing reassurance, for example – to reduce the impact of negative emotions on language learning.  

Where Interlocutors Look Matters

In another recent project, Professors McDonough and Trofimovich and their team analysed how eye contact influenced the effectiveness of corrections provided by language teachers. Here, students learning English as a second language participated in discussions about complex topics. However, during these experiments, the ‘teachers’ listening to the student were also research assistants involved in the study, and specialist equipment was used to monitor the eye movements of both the students and the interacting research assistants. Furthermore, the conversations were also video-recorded from multiple angles.

Analysing these recordings, the scientists identified over 1,000 instances when corrections (recastings) were made by the research assistants. Exploring the type of errors and analysing eye contact (gaze) between the language learners and the teaching assistants allowed the scientists to draw conclusions about the importance of the observed exchanged glances.

The results indicated that mutual gaze immediately after a teacher made a correction was more likely to result in the learner repeating the correct word/expression. Further in-depth analysis also identified that learners were more likely to correct themselves when the mistake involved the use of the wrong word or expression (lexical error). In contrast, grammatical errors were less likely to be corrected, although mutual gaze as seen on the video indicated that learners registered these corrections too. The other interesting detail emerging was that such corrections were associated with brief eye contact, while longer exchanged glances suggested that the learner either did not understand what the error was or was unable to make the correction.

Non-verbal Cues of Non-understanding

Professors McDonough and Trofimovich also wanted to explore whether listeners gave visual cues indicating that they failed to understand a statement made by a speaker. In these experiments, 60 observers, acting as impartial assessors, listened to conversations and were asked to rate the likelihood that the listeners understood the information the speaker was trying to convey. Interestingly, when the observers could see the faces of the listeners, they were able to identify instances of non-understanding much more accurately than when the listener’s face was obscured. During episodes of non-understanding, the listeners appeared to be ‘frozen’ with changes to their posture, facial expression or the direction of their gaze being suspended – a phenomenon scientists call ‘hold’ – while waiting for clarification. The scientists first noticed that these holds were associated with periods of non-understanding in a small study in 2019 and were able to confirm this in experiments conducted with larger groups of participants in 2021 and 2022.

The work of Professors McDonough and Trofimovich with their colleagues has clear practical relevance to the effective teaching of second languages. For example, recognising the signs of stress can enable teachers to change the direction of conversations or reassure students, thus helping to eliminate the negative impact of anxiety on language learning. Similarly, the use of appropriate facial expressions by teachers could improve the learning experience. However, they believe that the most important consequence of their work may be a better understanding of non-verbal communication between individuals. In this context, they hope that the findings of their research will help not only students but couples and work colleagues to communicate more effectively.

The Way Forward

Professors McDonough and Trofimovich and their team are committed to extending this important work. They are particularly keen to expand their work from the laboratory to more natural settings. They are also conscious that some students may have an inherently more nervous disposition that makes differentiating between the stress caused by a challenging speaking task and the consequences of general anxiety, rather difficult. Addressing these issues will represent the next steps for the team in developing effective strategies to support those learning a second language.