Variability in tone and gesture within language
Marisa Cruz1, Marc Swerts2 & Sónia Frota1
University of Lisbon1, Tilburg University2
[email protected], [email protected] & [email protected]
Intonational systems may vary between languages [1, 2, 3], and between language
varieties [4, 5, 6]. Previous studies on facial gestures as a complement of spoken language
revealed that visual signals are organized into a system sharing several features with the
prosody of spoken language [7, 8, 9]. Thus, it is reasonable to assume that visual cues also
vary across languages and language varieties. Moreover, variation in gesture could mirror
intonational variation, or compensate lack of tonal contrast. However, the variability of
gestures (if any) and its relation with intonational variation remains largely understudied.
Visual prosody in European Portuguese (EP) has not received much scholarly attention.
This work taps into the tone/gesture relation in EP by addressing three main questions: (i) Do
EP varieties use different facial gestures to convey the same sentence type/pragmatic
meaning? (ii) Is variation in the visual domain related to intonational variation? (iii) What are
the best predictors for facial gestures (sentence type, pragmatic meaning, intonation, or
region)? Facial gestures from the InAPoP audiovisual database [10], obtained by means of a
Discourse Completion Test, were analyzed in four EP varieties previously described for
intonation [11, 12, 13, 14, 15]: the Standard variety (SEP), two central-southern varieties
(Ale, Alg), and one insular variety (Azores-PtD). A total of 197 utterances of two sentence
types (statements/yes-no questions) and pragmatic meanings (broad/narrow focus statements),
produced by three speakers (20-45 years old) per variety, were annotated for facial gestures
following FACS [16]. Potential visual parallels for intonational features (pitch accent,
boundary tone, and nuclear contour configuration) were inspected.
Results are given in Table1. Broad focus statements were produced with the same facial
gesture across varieties, co-occurring with the same contour type. Narrow focused statements
also show the same nuclear contour across varieties and basic facial gesture, but display more
visual complexity (additional varying cues). By contrast, yes-no questions show different
intonational patterns co-occurring with different head movements. However, the concomitant
eyebrow raising characterizes interrogativity across varieties. Overall, yes-no questions
present greater variability of both tonal and visual cues than statements (Figure1), and narrow
focused statements exhibit more variability than broad focus statements, especially in the
visual domain, but less variability than yes-no questions. Moreover, not all varieties display
similar patterns of variability (Figure2). These results suggest that sentence type, pragmatic
meaning, and region have an impact on facial gestures. To determine which factors contribute
the most to the variability found in gestures, two multinomial logistic regressions (facial
gestures timed with nuclear pitch accent (NPA); facial gestures timed with boundary tone
(BT)) were run. Region*sentence type have a significant effect on facial gestures timed with
NPA type and with BT type. Sentence type*pragmatic meaning were also good predictors of
gestures. However, region alone is not a good predictor, unlike sentence type or pragmatic
meaning (Table2).
In conclusion, visual cues, like intonational cues, may vary across varieties, sentence
type and pragmatic meaning. Since in production facial gestures are affected by linguistic and
nonlinguistic factors, we hypothesize that speakers across varieties will be sensitive to visual
information, especially in the absence of tonal contrast (e.g. SEP perceiving PtD questions,
see also Figure2), and in the presence of audio-visual mismatches. These predictions will be
addressed in future research.
Statements
EP
SEP
Broad focus
Tonal
Visual
Tonal
H+L*
head up-down
L%
neutral position
(H+)L*
head up-down
L%
neutral position
L%
neutral position
H%
head up-down
H*+L
head up-down
+eyes closed
L*+H
L%
neutral position
L%
neutral position
H%
(H+)L*
head up-down
H*+L
L%
neutral position
L%
Ale
H+L*
Alg
PtD
Questions
Narrow focus
Tonal
Visual
head up-down
H*+L
+eyebrow raising
L%
neutral position
head up-down
H*+L
+eyebrow lowering
head up-down
+eyebrow raising
neutral position
H+L*
LH%
(H+)L*
(H+)L*
L%
Visual
head up-down
+eyebrow raising
neutral position
head up-down
+eyebrow raising
head back-forward
+eyebrow raising
eyebrow raising
head back-forward
(+eyebrow raising)
head up-down
+eyebrow raising
neutral position
Table 1 – Visual cues aligned with pitch accent/boundary tone types per sentence type (statements versus
questions) and per pragmatic meaning (broad focused versus narrow focused statements) across EP varieties
(SEP, Ale, Alg, and PtD). The dominant tone and gesture patterns across and within speakers are represented.
Figure 1 – Variability of tonal and visual cues in broad focus statements (left top), narrow focused statements
(right top) and yes-no questions (bottom) across EP varieties. Within and across speaker variation is
represented.
Multinomial Logistic Regression
Predictors of facial gestures timed with NPA
Predictors of facial gestures timed with BT
Sentence type
χ² (5)=28.24, p=.000
Sentence type
χ² (3)=10.51, p=.015
Pragmatic meaning
χ² (5)=13.66, p=.018
Pragmatic Meaning
χ² (3)=12.30, p=.006
Region
χ² (15)=18.68, p=.229 Region
χ² (9)=10.05, p=.346
Region*Sentence Type
χ² (30)=49.16, p=.015 Region*Sentence Type
χ² (18)=36.27, p=.007
Sentence type*Pragmatic meaning
χ² (5)=43.07, p=.000
Sentence type*Pragmatic meaning
χ² (3)=11.79, p=.008
Table 2 –Results for the two multinomial logistic regressions: facial gestures timed with NPA (left) and facial
gestures timed with BT (right).
Figure 2 – Variability of tonal and visual cues across sentence types in SEP and PtD (zooming-in main tonal
patterns). More visual variability relates to tonal contrast between sentence types (SEP); less variability to
absence of tonal contrast between sentence types (PtD).
Submission for talk/poster
References:
[1]
Gussenhoven, C. 2004. The Phonology of Tone and Intonation. Cambridge: Cambridge University Press.
[2]
Ladd, D. R. 2008. Intonational Phonology. 2ª ed., Cambridge: Cambridge University Press.
[3]
Jun, S.-A. (Ed.). 2014. Prosodic Typology II: The Phonology of Intonation and Phrasing. Oxford: Oxford
University Press.
[4]
Bruce, G. 2005. Intonational prominence in varieties of Swedish revisited. In S.-A. Jun (Ed.), Prosodic
Typology I. The Phonology of Intonation and Phrasing (pp. 441-429). Oxford: Oxford University Press.
[5]
Prieto, P. & P. Roseano (Eds.). 2010. Transcription of Intonation of the Spanish Language. Lincom
Europa: München.
[6]
Frota, S. & P. Prieto (Eds.). In press. Intonation in Romance. Oxford: Oxford University Press.
[7]
Krahmer, E., & M. Swerts (Eds.). 2009. Audiovisual prosody (special issue). Language and Speech,
52(2-3), 129-386.
[8]
Mol, L., E. Krahmer, A. Maes & M. Swerts. 2011. Seeing and Being Seen: The effects on gesture
production. Journal of Computer-Mediated Communication, 17(1), 77-100.
[9]
Loehr, D. P. 2012. Temporal, structural, and pragmatic synchrony between intonation and gesture.
Laboratory Phonology. Special Issue Gesture as Language, Gesture and Language 2, 3(1), 71-89.
[10] Frota, S. (Coord.). 2012-2015. InAPoP – Interactive Atlas of the Prosody of Portuguese Project.
[Webplatform available at http://www.fl.ul.pt/LaboratorioFonetica/InAPoP/].
[11] Frota, S. 2000. Prosody and focus in European Portuguese. Phonological phrasing and intonation. New
York: Garland Publishing.
[12] Frota, S. 2014. The intonational phonology of European Portuguese. In S.-A. Jun (Ed.), Prosodic
Typology II: The Phonology of Intonation and Phrasing (pp. 6-42). Oxford: Oxford University Press.
[13] Cruz, M. 2013. Prosodic variation in European Portuguese: phrasing, intonation and rhythm in centralsouthern varieties. Unpublished PhD Dissertation, University of Lisbon.
[14] Crespo-Sendra, V., M. Cruz, J. Silva & S. Frota. 2014. Asking questions across Portuguese varieties. Talk
given at the 6th International Conference on Tone and Intonation in Europe (TIE), September 10-12,
University of Utrecht, Netherlands.
[15] Frota, S., M. Cruz, F. Fernades-Svartman, M. Vigário, G. Collischonn, A. Fonseca, C. Serra & P.
Oliveira. In press. Intonational variation in Portuguese: European and Brazilian Varieties. In S. Frota & P.
Prieto (Eds.), Intonation in Romance. Oxford: Oxford University Press.
[16] Ekman, P., W. V. Friesen & J. C. Hager. 2002. Facial Action Coding System. Salt Lake City, UT: A
Human Face.