Multimodality has most recently been described no longer as a research field or discipline on its own, but rather as a “stage of development within a field” (Bateman 2022a, 49). The realization that (1) many different fields and disciplines now enter their own multimodal phase with new interest in multimodal phenomena and that (2) these disciplines all commit to the development of multimodality research with their own theoretical principles and methodological tools, brings with it not only an immense breadth of potential analytical objects, but also many new meta-methodological issues. “We need to find ways of ‘combining’ insights from the variously imported theoretical and methodological backgrounds brought along by previous non-multimodal stages of any contributing disciplines” (Bateman 2022a, 49). At the same time, the search for a meta-methodology for multimodal analyses is pushed further by the recent trend towards more empirical approaches to multimodal phenomena and the development and use of larger multimodal corpora that just as well require theoretical and methodological refinements. “We need to develop ways of strengthening claims with robustly applicable methods which nevertheless remain firmly anchored theoretically” (Bateman 2022b, 64). For a productive handling of these issues, disciplinary triangulation and finding a ‘common language’ or metalanguage (Maton & Chen 2016) for an ‘integrationist interdisciplinarity’ (van Leeuwen 2005) are the greatest challenges in contemporary multimodality research (Bateman 2022a). Also, there is a need for reconceptualizing the practice of analysis by making available large-scale corpora and broader and more complex empirical setups to fully process the ‘move from theory to data,’ and to substantiate long-lasting theoretical and methodological hypotheses (Pflaeging et al. 2021). For this project, we see these challenges productively as “a multimodal task from the ground up,” as John Bateman (2022b, 64) has phrased it in one of his most recent papers. This Research Topic will address this task by convening the most recent theoretical, methodological, practical, and empirical developments within contemporary multimodality research. The aim is to gain new insights in • the metalanguages or external languages that are currently being developed for multimodal analysis in many different research fields and disciplines, e.g., in pedagogy, literary theory, cultural studies, design, argumentation theory, computer science, and (experimental) psychology; • newest results from data collection methods and multimodal corpus analyses that expand the current quantitative work by, e.g., applying existing theories and methods to larger datasets, or exploring the newest communication technologies. We are particularly interested in seeing how works addressing these aspects contribute to finding ways of productive triangulation and integration for and within a meta-methodology for multimodality research. This Research Topic aims to bring together scholars from a variety of disciplines interested in multimodality research to review, explore, and advance the contributions that John Bateman, as one of the key figures in multimodality research, has made to both theory- and method-building as well as to the driving forward of multimodal empirical and corpus analyses. We welcome contributions that, for example, • critically address the theoretical and methodological advancements that John Bateman has made with regard to the notions of semiotic mode, discourse semantics, genre, textuality, etc.; • apply one of the many approaches that John Bateman has developed for the empirical analysis of multimodal artefacts (e.g., the GeM model for page-based documents, his work on multimodal film and audio-visual analysis, and the discourse semantics and/or annotation approach to visual narratives) to larger corpora or currently newly developing communicative situations; • expand on one of the abovementioned aspects with new ideas and insights from disciplines that have not yet been included in multimodality research.

tionship indicates how multimodal medical image processing can be unified to a large extent, e. g. multi-channel segmentation and image registration, and extend information theoretic registration to other features than image intensities. The framework is not at all restricted to medical images though and this is illustrated by applying it to multimedia sequences as well. In Chapter 4, the main results from the developments in plastic UIs and mul- modal UIs are brought together using a theoretic and conceptual perspective as a unifying approach. It is aimed at defining models useful to support UI plasticity by relying on multimodality, at introducing and discussing basic principles that can drive the development of such UIs, and at describing some techniques as proof-of-concept of the aforementioned models and principles. In Chapter 4, the authors introduce running examples that serve as illustration throughout the d- cussion of the use of multimodality to support plasticity.

This book brings together cutting-edge research on multimodal texts and the "discourses" generated through the interaction of two or more modes of communication, for example pictures of language, typography and layout, body movement and camera movement. The contributors collected within this volume use systemic functional linguistics to analyze how meaning is generated within a series of case studies. The result is a comprehensive survey of the ways in which enhanced meaning emerges through the interaction of more than one mode of communication. Multimodal Discourse Analysis will be useful to researchers interested in the application of systemic functional linguistics to media studies, discourse analysis and cognitive linguistics.

Multimodal Pedagogies in Diverse Classrooms examines how the classroom can become a democratic space founded on the integration of different histories, modes of representation, feelings, languages and discourses, and is essential reading for anyone interested in the connection between multimodality, pedagogy, democracy and social justice in diverse classrooms. Pippa Stein combines theory with material taken from post-apartheid classrooms in South Africa where students from different language and cultural backgrounds negotiate the ongoing tensions between tradition and modernity, Western and African intellectual thought, as well as the apartheid-past of their parents, and their own aspirations for the future. This insightful book argues that classrooms can become ‘transformative’ sites in which students can develop curricula and pedagogies which speak to the diversity of global societies, and looks at: How multimodality can be used to promote social justice and democracy in diverse classrooms; The forms of representation through which students make meaning in classrooms; How those forms contribute to the building of democratic cultures; The cultural resources available to students, and how they are used for learning; Difference as a productive energy for learning. Dealing with issues such as democracy, politics of difference, diversity, multicultural and multilingual classrooms, this book is as pertinent to readers across the globe as it is to those in South Africa, and will be invaluable and fascinating reading for anyone working or interested in this field.

The phenomenon of multimodality has, as Jewitt observes, generated interest "across many disciplines...against the backdrop of considerable social change." Contemporary societies are grappling with the social implications of the rapid increase in sophistication and range of multimodal practices, particularly within interactive digital media, so that the study of multimodality also becomes essential within an increasing range of practical domains. As a result of this increasing interest in multimodality, scholars, teachers and practitioners are on the one hand uncovering many different issues arising from its study, such as those of theory and methodology, while also exploring multimodality within an increasing range of domains. Such an increase and range of interest in multimodality heralds the emergence of a distinct multimodal studies field: as both the mapping of a domain of enquiry, and as the site of the development of theories, descriptions and methodologies specific to and adapted for the study of multimodality. The present volume presents a range of works by an impressive international roster of contributors who both explore issues arising from the study of multimodality and explore the scope of this emerging field within specific domains of multimodal phenomena. Contributors aim to show that each individual work and works in general within multimodal studies represent a dialectic or complementarity between the exploration of issues of general significance to multimodal studies and the exploration of specific domains of multimodality; while characterizing specific works as tending to some degree towards one or other of these main areas of focus. Such a characterization is seen as part of a move towards the identification and thus development of a distinct field of multimodal studies.

This collection brings together a range of perspectives on multimodal communication in intercultural interaction, bridging cognitive, social, and functional approaches towards promoting cross-disciplinary dialogues and taking research at the intersections of these fields into new directions. The volume assembles conversationalist, socially oriented, cognitive, and sensory approaches in considering culture as a dynamic construct, co-constituted and (re)negotiated among participants in interaction and filtering it through a multimodal lens, drawing on a range of examples, such as educational settings or online video platforms. Each chapter offers a unique perspective on "culture" and "intercultural," while also situating their own definitions of these labels against those of the other chapters. Taken together, the chapters form a fluid conversation on the nature of intercultural encounters in today’s globalizsd world, as digital environments intertwine with the physical mobility of people, encouraging researchers across these fields to adopt a more holistic multimodal perspective to approach intercultural interaction. This book will be of interest to students and scholars in intercultural communication, multimodality, sociolinguistics, cognitive and interactional linguistics, and semiotics.

Most dialogues are multimodal. When people talk, they use not only their voices, but also facial expressions and other gestures, and perhaps even touch. When computers communicate with people, they use pictures and perhaps sounds, together with textual language, and when people communicate with computers, they are likely to use mouse “gestures” almost as much as words. How are such multimodal dialogues constructed? This is the main question addressed in this selection of papers of the second “Venaco Workshop”, sponsored by the NATO Research Study Group RSG-10 on Automatic Speech Processing, and by the European Speech Communication Association (ESCA).

This volume presents a comprehensive overview of multimodal approaches to curriculum and programmatic implementation across a diverse range of teaching environments and across geographic and cultural boundaries. Featuring contributions from scholars within and across both disciplines, the book examines the ways in which new technologies link to expanding definitions of literacy and, building on this, how multimodal approaches might most effectively address the unique opportunities and challenges instructors face in contemporary classrooms and professional development programs. Chapters draw on case studies from both existing scholarship and findings from the authors’ own experiences in practice, including examples from writing, rhetoric, and composition courses, open online learning courses, and interdisciplinary faculty training programs. The final section of the book showcases how the conversation might be further extended to address increasingly multilingual classrooms by exploring how multimodality has been implemented in transnational settings. Engaging with key questions at the intersection of programmatic and curricular development and multimodal studies, this book is a fundamental resource for graduate students and scholars in multimodality, rhetoric studies, language education, applied linguistics, and communication studies.

Multimodal Signs of Learning proposes a methodology to uncover evidence of learning in students’ multimodal compositions. Informed by social semiotic theory, the book tracks representation of subject content from physical and embodied teaching resources to students’ handmade artefacts and physical presentations. Using materials from secondary school history and science classrooms, multimodal realizations of specific representational processes are tracked from the input of resources through to the students’ multimodal compositions – their posters, models and physical presentations. Through tracking semiosis, the book exposes the epistemologies inherent in the representational choices articulated in the students’ multimodal designs. These, it is argued, are to be valued as signs of learning. Learning is thus characterized as ‘design’ and the transformation of subject content through representation in different modes shown not only to promote learning, but also to contain evidence for its recognition. The book raises important questions about what constitutes multimodal learning and how it can be applied. It contributes to the growing body of research into the changing dynamics of classrooms and assessment practices and will be of great interest to researchers, and academics in the fields of education research, multimodality, semiotics and communication.

A guide that offers a step-by-step process to data-driven qualitative multimodal discourse analysis Systematically Working with Multimodal Data is a hands-on guide that is theoretically grounded and offers a step-by-step process to clearly show how to do a data-driven qualitative Multimodal Discourse Analysis (MDA). This full-color introductory textbook is filled with helpful definitions, notes, discussion points and tasks. With illustrative research examples from YouTube, an Experimental and a Video Ethnographic Study, the text offers many examples of how to deal with small to large amounts of data, including information on how to transcribe video data multimodally, including online videos, and how to analyze the data. This textbook contains ample theory, directions for literature, and a teaching guide to help with a clear understanding of how to work with multimodal data. Contains new research data, exceptional illustrations and diagrams Offers step-by-step processes of working through examples, transcriptions and online videos Goes into great depth so that students can use the book as hands-on material to engage with their own data analysis Designed to be easy-to-use with color-coded definitions, tasks, discussion points and notes Written for advanced undergraduate, graduate and PhD level students, as well as participants in research workshops, Systematically Working with Multimodal Data is an authoritative guide to understanding data-driven qualitative Multimodal Discourse Analysis.