From 2nd to 3rd May 2013 in Lisbon was held the international conference Multimodal Communication: Language, Performance and Digital Media”, organized by the FCSH and the FCT of Universidade Nova de Lisboa, in the framework of the TKB project conclusion.

Coordinated by Carla Fernandes, the Transmedia Knowledge Base project seeks to provide a research space for rigorous and critical exploration of the relationship between linguistics, contemporary dance and new digital media.

Therefore, research questions lay in a fairly hybrid territory. How is a choreographer’s imagetic type of thought translated into an embodied-type of thought in motion via the dancers? Can choreography be interpreted as thoughts in motion? 
What is the impact of verbal language on the dancers’ movements? How to capture dance’s ephemeral nature through digital representations that overcome plain motion capture or video recording?

Following the transdisciplinarity of this 3-year investigation, at the crossroad between cognitive linguistics, video annotation, performing arts documentation and new media technologies, the event brought together contemporary artists and researchers from a broad range of academic areas. Scott DeLaHunta, Sarah Whatley, Sue Hawksley, Rui Horta, Berta Bermúdez and Sally Jane Norman were just few of the outstanding personalities that took part in the meeting. Someone you may want to know once in your life if you are interested in the field of dance and digital media.


A Transmedia Knowledge-Base for Performing Arts

At the core of the conference was the presentation of some software tools developed during the TKB project, aimed at designing and constructing an open-ended multimodal knowledge-base to document, annotate and facilitate the creation of contemporary dance pieces.

The Creation Tool, presented by Nuno Correia and Diogo Cabral, is an original video annotator that supports multimodal annotation to assist the compositional processes of choreographers, functioning as a digital notebook. Choreographer and researcher Stephan Jürgens offered practical examples of the adoption of this interface, which proved particularly effective especially to share insights with the dancers and the interaction designer in the creation of interactive performances. Indeed, thanks to TKB video annotator, it is possible to synchronize video projection streams and simultaneously take notes of ideas, comments or sketches on a tablet. In this way correlations can be visualized between the dancers’ movement and the visual output of the interactive system employed.

The second Tool, Knowledge Base, is a collaborative digital relational platform for creators, performers and researchers aimed at mapping and connecting multimodal documental content. This digital archive of selected Portuguese contemporary dance creations will be soon online and it will help artists share their compositional processes, working methods or finished pieces.


Carla Fernandez reported, the aim is to gradually include the newest creations of international choreographers showing interest in the initiative, as well as the existing press and television documentation and a spectator’s gallery for feed-back and audience intervention. Techniques to record embodied motion date back thousand of years, as scholar Sally Jane Norman brilliantly demonstrated. All these different systems devised to preserve and convey human movement also mirror diverse understandings of it.

Following this contextualization, Scott DeLaHunta, project leader of Motion Bank (The Forsythe Company), discussed some recent case studies in the field of dance documentation research. Indeed, artists and the organizations around them have begun to think or rethink how to create, manage and disseminate their choreographic resources in novel ways.

As DeLaHunta reminded, a well-debated tension exists between the live dance performance and its documentation or recording. Even if neither videotapes and motion capture, nor written inscriptions can grasp the ephemerality of danced choreography, one could argue that while the dance may disappear, a valuable creative resource still remains. More than a mere ‘trace’ this resource is useable in several ways: it can be transmitted and disseminated, it is transferable and embodies information that can feed back into the choreographic process.

In Amsterdam, for instance, the dance company Emio Greco|PC is researching ways to create an information resource derived from their creative work from which they, their performers, other artists and designers, researchers and thinkers might draw. How to create a ‘living archive’ based on principles of movement and choreography that are constantly evolving? What notation system can capture inner intention as well as the outer shape of gestures and phrases? How to analyze and represent open processes in relation to artworks? In this framework, dancer and researcher Berta Bermúdez reported about “Pre-choreographic elements”, a research project developed inside this world-renowned dance company. Based on a selection of movement concepts used within the body of work of Greco|PC, a multimodal interface for dance archiving and dissemination has been created, where words, sound and images are combined.


Following this example, scholar Sarah Whatley introduced the recent projects of British choreographer Siobhan Davies to examine the way she has been engaging with contemporary forms of dance inscription to influence her creative process. Indeed, new forms of digital documentation both challenge and reconfigure the notion of archive. For Davies’s online digital archive new visualization strategies have been prototyped, allowing the user to browse digital resources in alternative ways. These presentations bring together objects organized in layers, according to their role in the making of a work.

Digital Technologies as “reproductive media” for Performing Arts

Following the TKB tools, other platforms for intangible cultural heritage preservation were also introduced.  For instance, A Dança Portuguesa A Gostar Dela Própria (DPAGDP) is an online video archive realized by the association PédeXumbo and film director Tiago Pereira with the aim of collecting current Portuguese choreographic practices.
With a more traditional structure, the platform Terpsicore-critica – curated by researcher Daniel Tercio – is an online archive of documents about Dance and Performing Arts, especially press material.

Alongside these documentary collections, other projects were presented that seek to go beyond video recordings and hypertexts to envisage novel ways to disseminate the embodied knowledge lying in performative practices.

The most ambitious is perhaps Choreo-haptic project discussed by Sophia Lycouris, which deals with the complex issue of representing visual perception of dance pieces through haptic devices. Can dance audiences access the viewing experience of dance without using their sense of vision? Can we “see” dance in other ways? Addressing these issues, the Choreo-haptic project explores how blind dance audience members can use their hands to experience the dynamic qualities of live performances through the sense of touch. A vibrating pad has therefore been conceived which seeks to map dancers’ movement into pulses on the surface. It is not conceived as a translation device, rather it aims to create an experience for blind persons, which is similar to that received by the sighted audience. To achieve this, the haptic device is conceptualized as a performance space where dynamic events take place.

Digital Technologies as “productive media” for Performing Arts

Other contributions focused more on the employment of motion capture, sensing technologies or robots in the creation of live performances. Researching in the field of HCI, Ubiquitous Computing and AR, Fernando Nabais offered exciting examples of interaction between dancers and artificial creatures in live performances. In “I CARE US” flying “social” robots, commonly known as drones, move along humans, enchanting the audience with their in air stability and fluidity.

Working with media artist Simon Biggs and composer Garth Paine, dancer and choreographer Sue Hawksley offered insights about the use of somatic data in interactive dance performances. “Bodytext” is an example of dialogue between speech, movement, sound, body and visuals where a dancer is sited within an environment augmented by technologies, such as sensing instrumentation, speech recognition systems and digital audio-visual displays. The dancer’s movements and speech are captured and rewritten into a projection of text animated by the choreography of the performance, which is also tracked to generate the soundtrack for the performance in real-time.

Adopting a much more theoretical approach, researcher Paula Varanda offered a reflection about the nature of new dance forms based on digital media.

Now that the dance can enter the machines, it seems relevant to review current practice and discuss how features that are fundamental to define dance as a preforming art can be identified also in new media forms, such as cyberdance. Here the focus was on the web as a stage to reach new audiences and, above all, to create new art works. As Varanda underlines, artistic dance practice, intentionally conceived for presentation on the web, is rare on respect to other art forms. This could be explained also considering the gaps existing in theory, which has not yet extended conceptualization about elements and attributes that constitute dance performance to the principles that operate in new media. In her analysis, Varanda contributed to demonstrate that constituent elements that define theatre based choreographic works remain references to identify creative dance practice developed for the cyberspace.


Gathering leading experts in their respective fields, the conference “Multimodal Communication: Language, Performance and Digital Media” has been the perfect conclusion to TKB international project. Meetings like this set the standard for high-level debates about performing arts digital preservation and choreographic thinking dissemination, multimodality in human-computer interaction and embodied cognitions.