What’s the first thing that comes to mind in response to “Italian”? Lots of things probably, from Lamborghini to cannelloni and Bertolucci. Perhaps you also thought of how Italians talk. Even if we don’t speak Italian, we probably know that Italians do a lot of talking with their hands: in case you have any doubt, the video below provides a first lesson.
Of course, it’s not just Italians who use their hands while talking—we all do, to a greater or lesser extent. Gestures are an integral part of human speech. We can recognize how important gestures are when someone is not using their hands to talk, but has to rely on someone else’s, as shown in the next video:
It has been estimated that gestures can carry 50% or more of the information that a listener is encoding. So gestures during speech are important, but how exactly do they function? What do gestures accomplish in a conversation?
A recent article in the Psychonomic Bulletin &Review tackled those questions to provide a first glimpse at the role of gestures in human communication.
Researchers Judith Holler, Kobin Kendrick, and Stephen Levinson pursued their research questions within the context of conversations. Conversations are among the most natural human communicative activities, and we routinely engage in conversations, sometimes many times a day. Yet, the structure of conversations presents a cognitive puzzle: The time in between burst of vocalizations by different speakers—that is, the time required for “turn-taking”—is in the order of 200 ms (1/5th of a second). This is considerably less time than is required for a person to produce a simple one-word utterance when asked to do so (around 600 ms in the laboratory).
It follows that a conversation must involve considerable predictive processing: While listening to our spouse, we not only pay careful attention to what she (or he) is saying, but we apparently also try to predict throughout when she (or he) might finish and it is our turn to respond—which we can then do in a brief instant.
Perhaps gestures play an important role in signaling turn-taking in a conversation?
To find out, Holler and colleagues video-recorded several casual, non-directed conversations between three acquaintances in the laboratory. The figure below shows the apparatus and explains the methodology.
Each participant was wearing eye-tracking glasses to obtain a view from a first-person perspective. Three additional cameras provided further triangulation of how the participants interacted, and each participant wore a microphone to capture their vocal utterances. The apparatus ensured that all relevant aspects of the conversations—both visual and auditory—were captured for analysis.
Analysis focused on question-response sequences, because questions present a particularly clear invitation to turn-taking. For each question, gestures (defined as movement of the hand or head) were timed and their temporal position relative to the speech signal was recorded. Additional measures involved the gap between the offset of the spoken question and the onset of the response (i.e., the turn-taking time), as well as measures of the “prosody” of the question (i.e., it’s auditory frequency and volume envelope).
Holler and colleagues found that out of 281 questions analyzed, more than 60% were accompanied by a gesture. Analysis of the turn-taking time revealed that the gaps following questions without gestures were most frequently in the order of 200 ms, as is commonly observed in conversations. For questions accompanied by gestures, this gap was reduced to near zero.
The figure below shows the distributions of turn-taking times for the two types of question.
Negative times can be observed when a response commences while the question is still being asked (not necessarily recommended during inter-spousal communication). It is clear that the presence of a gesture triggered earlier responding, and in particular that responses often commenced before the end of the question when a gesture was present.
When the gestures were split into those involving the hand, and those involving the head, the effect persisted for each subset. Whether using hands or head to gesture, it tends to elicit a faster response..
In a further analysis, Holler and colleagues examined the role of so-called “retractions” of gestures. Retractions are movements at the end of a gesture that involve a reversal of the movement—for example, retracting a finger after using it to point at a person. Around 40% of all observed gestures involved a retraction, and of those, around 60% began before the end of the verbal utterance.
The figure below shows that turn-taking was particularly quick when a gesture retraction occurred before the end of a question.
This retraction effect was observed in addition to the overall gesture effect: so not only do gestures facilitate turn-taking, but if the gesture involves an early retraction, then responding is accelerated even more.
What do these data tell us?
As Holler and colleagues note, “in conversation, time is of the essence.” Gestures clearly serve to meet this essential need. Intriguingly, previous research has linked the duration of gaps in turn-taking to the amount of processing that is required to comprehend an utterance. The longer the gap, the more difficult the utterance was to understand. It follows that the reduced gap duration observed with gestures may reflect the greater ease of comprehension afforded by accompanying speech with gestures. However, this possibility remains to be pinned down by further research as the present study did not include any comprehension measures.
My sense based on the Italian video above is that gestures do facilitate comprehension, nessun dubbio a riguardo. And this intuition is supported by other recent research that we blogged about here.
Article focused on in this blogpost:
Holler, J. Kendrick, K. H., Levinson, S. C. (2017). Processing language in face-to-face conversation: Questions with gestures get faster responses Psychonomic Bulletin &Review. DOI: 10.3758/s13423-017-1363-z.