World futures through RT's eyes: multimodal dataset and interdisciplinary methodology (2025)

Multimodality across Communicative Settings, Discourse Domains and Genres

2016

Gestures, ideologies and meaning in TV debates: A multimodal approach

2019

In this article we will analyze how political actors involved in the presidential TV debates use certain semiotic strategies to manage the relationship between gestures and their ideological orientation. We developed a multimodal analysis for some relevant sequences during the presidential debates from November 2014. For this purpose, we used a Multimodal Professional Analysis Tool, ELAN, which allowed us the annotation and dynamic analysis of the semiotic behavior of the political actors involved in the analyzed sequences.

Analysing political discourse in film language: A multimodal approach

"Multimodality across Communicative Settings, Discourse Domains and Genres", V. Bonsignori and B. Crawford Camiciottoli (eds) , 2016

How to prepare the video component of the Diachronic Corpus of Political Speeches for multimodal analysis

Research in Corpus Linguistics, 2021

The Diachronic Corpus of Political Speeches (DCPS) is a collection of 1,500 fulllength political speeches in English. It includes speeches delivered in countries where English is an official language (the US, Britain, Canada, Ireland) by English-speaking politicians in various settings from 1800 up to the present time. Enriched with semi-automatic morphosyntactic annotations and with discourse-pragmatic manual annotations, the DCPS is designed to achieve maximum representativeness and balance for political English speeches from major national English varieties in time, preserve detailed metadata, and enable corpus-based studies of syntactic, semantic and discourse-pragmatic variation and change on political corpora. For speeches given from 1950 onwards, video-recordings of the original delivery are often retrievable online. This opens up avenues of research in multimodal linguistics, in which studies on the integration of speech and gesture in the construction of meaning can include analyses of recurrent gestures and of multimodal constructions. This article discusses the issues at stake in preparing the videorecorded component of the DCPS for linguistic multimodal analysis, namely the exploitability of recordings, the segmentation and alignment of transcriptions, the annotation of gesture forms and functions in the software ELAN and the quantity of available gesture data.

Language technology tools and resources for the analysis of multimodal communication

2016

In this paper we describe how the complexity of human communication can be analysed with the help of language technology. We present the HuComTech corpus, a multimodal corpus containing 50 hours of videotaped interviews containing a rich annotation of about 2 million items annotated on 33 levels. The corpus serves as a general resource for a wide range of re-search addressing natural conversation between humans in their full complexity. It can benefit particularly digital humanities researchers working in the field of pragmatics, conversational analysis and discourse analysis. We will present a number of tools and automated methods that can help such enquiries. In particular, we will highlight the tool Theme, which is designed to uncover hidden temporal patterns (called T-patterns) in human interaction, and will show how it can applied to the study of multimodal communication.

Multimodality I: Speech prosody and gesture

The Routledge Handbook of English Language and Digital Humanities, 2020

In the age of the Internet, trillions of bytes of media data are generated every day through telecommunications and social media. This surge of born-digital media data, for example, instant voice/video messages, conference calls, podcasts, video blogs and so on, offers researchers unprecedented opportunities to deepen their understanding of how human beings communicate and go about their social activities. However, such a large amount of data also brings a new problem: how may we plough through so much media data and extract meaningful information efficiently? This chapter explores opportunities and challenges at the interface between digital humanities and multimodality research which focuses on the use of prosody and gesture in spoken communication. Following an overview of key methods and frameworks in prosody and gestures research, it highlights selected projects which have showcased the ways in which today’s computer technology has revolutionised multimodality as an area of research. In recent years, many new computer tools have become available to aid media data acquisition, processing and analysis. These tools have (semi-)automatised many processes which were labour-intensive, expensive and tedious. Therefore, researchers can now afford to compile and process substantially larger multimodal datasets much faster and at a much lower cost. The chapter also introduces tools which open up new avenues for researchers to acquire new types of multimodal data (e.g. YouTube videos) and data streams (e.g. GPS, heartbeats). In the sample analysis, we demonstrate the typical workflow for using a range of these latest computer tools to generate a corpus of YouTube videos, automatically annotate prosodic patterns, align multiple data streams and perform a multimodal analysis on the use of the epistemic stance marker ‘I think’ in video blogs.

Toward an infrastructure for data-driven multimodal communication research

Linguistics Vanguard

Research into the multimodal dimensions of human communication faces a set of distinctive methodological challenges. Collecting the datasets is resource-intensive, analysis often lacks peer validation, and the absence of shared datasets makes it difficult to develop standards. External validity is hampered by small datasets, yet large datasets are intractable. Red Hen Lab spearheads an international infrastructure for data-driven multimodal communication research, facilitating an integrated cross-disciplinary workflow. Linguists, communication scholars, statisticians, and computer scientists work together to develop research questions, annotate training sets, and develop pattern discovery and machine learning tools that handle vast collections of multimodal data, beyond the dreams of previous researchers. This infrastructure makes it possible for researchers at multiple sites to work in real-time in transdisciplinary teams. We review the vision, progress, and prospects of this resea...

Multimodal human discourse

ACM Transactions on Computer-Human Interaction, 2002

Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform two independent sets of analyses on the video and audio data: video and audio analysis to extract segmentation cues, and expert transcription of the speech and gesture data by microanalyzing the videotape using a frame-acc...

Multimodal Communication in Political Speech Shaping Minds and Social Action

Vissta: A tool for analyzing multimodal discourse data

2002

Human communication, seen in the broader sense, is multimodal involving the words spoken, prosody, hand gestures, head and eye gestures, body posture variation and facial expression. We present the multimedia Visualization for Situated Temporal Analysis (Vis-STA) system for the analysis of multimodal human communication video, audio, speech transcriptions, and gesture and head orientation data. VisSTA is based on the Multiple Linked Representation, MLR strategy and keeps the user temporally situated by ensuring tight linkage among all interface components. Each component serves both as a system controller and display keeping every data element being visualized synchronized with the current time focus. VisSTA maintains multiple representations that include a hierarchical video-shot organization, a variety of animated graphs, animated time synchronized multi-tier text transcriptions, and an avatar representation. All data is synchronized with the underlying video.