Nuno Correia is teacher, researcher and artist (http://www.nunocorreia.com/). His Doctor of Arts thesis, with the title “Interactive Audiovisual Objects” (2013), published by Aalto University, is grounded on dialogues between ephemeral and fixity, between artist and users, between technologies and possibilities for interactions (https://www.taik.fi/kirjakauppa/product_info.php?cPath=26&products_id=269&language=en).
The following text results from an exchange online and aims at highlighting several aspects from Nuno Correia’s thesis. The research is based on four audiovisual projects: Heat Seeker (2006), AVOL (2007), Master and Margarita (2009) and AV Clash (2010), developed in collaboration with André Carrilho, under the collective name Video Jack.
These projects are not only performative but also exist as net art, single channel and installation. Central themes of the research constitute a singular triangulation between media, performance and collaboration, by developing an Interactive AudioVisual Objects (IAVO) approach, taking a crossmedia perspective, combining audiovisual content with interaction design and user experience.
Ana Carvalho: In the book, one of the most relevant subjects concerns the relationships between audio and image (audiovisual) and the way these co-exist as a unit within an interactive work. IAVO – Audio Visual Objects, is the term you coined, departing from Michel Chion’s “Audio-Vision. Sound on Screen”, to include the object. Chion mentions this relationship as “audiovisual contract”, which defines as a shift away from the concept of synesthesia (the two sensory stimulus are combined naturally in the brain) and seeks to establish a set of connections towards the construction of a similar and unified experience. This approach understands the difference between human mechanics of earing and seeing and assumes synesthesia as a condition of very few people. How does an “audiovisual contract” translate into your practice?
Nuno Correia: My notion of IAVO (Interactive Audio Visual Objects) aims to add interactivity to Michel Chion’s concept of synchresis: “the forging of an immediate and necessary relationship between something one sees and something one hears at the same time” (Chion 1994, p.224). I define IAVOs as modules that combine GUI (Graphical User Interface) and sound visualisation, forming a cohesive whole, where the GUI is embedded and aesthetically integrated in the visualisations.
In my thesis I also use Chion’s definition of audiovisual contract: “a kind of symbolic contract that the audio-viewer enters into, agreeing to think of sound and image as forming a single entity” (Chion 1994, p.216). I propose the concept of “interactive audiovisual contract” associated to an IAVO: a symbolic contract that the users enter into, agreeing to think that sound, visualisation and GUI form a single entity – the IAVO. I define this single identity as an “object”, following also the concept of Audio-Visual Objects by Kubovy and Schutz, which they define as “cross-modal experiences” that share a strong auditory and visual binding – an “audio-visual linkage” (Kubovy & Schutz 2010). Kubovy and Schutz analyse in-depth what it means to perceive something as an object. Again, I aim to include interactivity in that binding. As I summarize in my thesis (p. 25), “My own interpretation is that IAVOs are players and dancers of a virtual band, each playing their own sound/instrument with an accompanying ‘movement’ to that sound, in a virtual stage that is the computer screen.”
You mention synesthesia, which I approach in my thesis – I dedicate a section to it. However, I find Chion’s theories more interesting for my artistic practice than synesthesia. I agree with Scott Lipscomb, who argues that synesthesia is not a solid foundation upon which to build an appropriate theory of multimedia perception, since the number of synesthetes is small, and because its effect upon those synesthetes varies (Lipscomb 2002, pp.230–231). Authors such as Chion and Michael Kubovy present concepts that are on a higher level, that deal with perception on a more abstract way, and are in that sense more open. For example, as Chion states, synchresis can occur with a high number of combination of disparate audio and visual items: “play a stream of random audio and visual events, and you will find that certain ones will come together through synchresis and other combinations will not” (Chion 1994, p.63).
Ana Carvalho: Is the object within the term IAVO only virtual (existent as software or in an online platform) or is it also physical, to include such devices as the Reactable?
Nuno Correia: IAVO can be a software object or a combination of software and hardware. Of course, I was very influenced by Reactable, as one of my thesis supervisors was Sergi Jordà, one of the creators of Reactable (http://www.reactable.com/). Reactable follows a similar modular logic to the notion of IAVO. But in Reactable the sound visualisation exists in-between the modules (in the links/lines/connections between modules): “the lines that draw the connections between the objects, convey the real waveforms of the sound flow” (Jordà et al. 2007, p.142). The Reactable modules are mainly tangible interaction elements, augmented with GUI, not sound visualisation units. The approach in Reactable is both concerned with the modules and with the connections between modules, following a PureData/Max-MSP/data flow programming paradigm. So, Reactable follows a slightly different logic than IAVO. Reactable’s logic is somehow less object-oriented and more of a object/system hybrid – at least in my interpretation.
Ana Carvalho: The contract, as described by Chion, can be extended to encompass, in a broader sense, the user and the work presented, being it a performance or a net art piece. In this sense, closer to Peggy Phelan’s definition of contract, the user enters an agreement of sharing time as he or she enters the experience. On the other hand, in my perspective, each of the projects your research is focused on is situated between the two types of models of creative practice you address: between creator-centric, which includes the development of content and interface, between you and André Carrilho, to be used in performance situations, and experiencer-centric, which adds the appropriate interactive capacities and transforms a performance into a net art experience. The projects of you research seem to extend from one model into another. Is the model of your work possible to define? What are its specificities/particularities?
Nuno Correia: Indeed, Shamma & Shaw argue that new media is questioning the boundaries between creator-centric and experiencer-centric models. As examples, they point out “the production of media intended for active remix and reuse by others” (Shamma & Shaw 2007, p.277), combining the role of the creator and experiencer. They propose to seek new models in which the user of a creative work “takes on a generative role, not just an interpretive or interactive one”. I believe that my projects belong to these new models. Particularly since AVOL (AudioVisual OnLine, 2007), which was created from scratch to be used and “remixed” by others. I tried to define a model for my work, using the concept of Interactive AudioVisual Objects (IAVO) that I have presented earlier. It is an object-oriented model, following an object-oriented aesthetics, using the words of Gabriel Shalom (Shalom, n.d.). The IAVO approach favours manipulation and reinterpretation. The tendency in my projects has been to give the user more possibilities for intervention and appropriation. This becomes clear, I believe, if you compare AVOL, for example, with AV Clash (2010). The number of audiovisual modules and manipulation options (effects, loop trimming, sequencing, etc) is considerably larger in AV Clash.
Ana Carvalho: How does the contract and the two approaches to collaboration may possibly inform the future development of a community as you express on the final chapter of your dissertation?
Nuno Correia: After AV Clash, the last project developed in the scope of my thesis, I have been working on AVVX (AudioVisual Vector eXchange, www.avvx.org). AVVX aims to allow users not only to integrate their own sounds and visuals, but to contribute those visuals to the AVVX community, and also reuse other visuals from that community. Besides, the project is open-source, therefore the “users” can even change the code entirely. Here the word “user” may not make sense anymore, if she/he becomes more of a “contributor”. In www.avvx.org, a basic repository of visuals can be found. Hopefully in the near future, AVVX.org can be expanded to be a proper database of visuals, which could be loaded in real-time over the Internet to the AVVX app, with an API. AVVX users/contributors can share their outcomes on the website as well, as videos. For now, this can be done using a Vimeo AVVX group and an AVVX Facebook group (links available from www.avvx.org). I’ve conducted three AVVX workshops so far (in Ljubljana, Tallinn and Helsinki), with a fourth one coming up soon (in Riga). I’m happy to see that AVVX community members (at this point, mostly former workshop participants) are creating music videos with AVVX and using the software in their performances, as can be seen in the AVVX Vimeo and Facebook groups. With these developments, the two models identified by Shamma & Shaw become even more blurred, as “users” contribute to a common pool and reuse content created by other contributors. It becomes more of a peer-to-peer model, with some loose supervision (that role is played, at the moment, by me), closer to the “hacker ethic” identified by Pekka Himanen (Himanen 2001).
Ana Carvalho: In your projects, and now in the platform, ephemerality and fixity are combined. In the ephemeral, each performance is documented and in the fixity, the documents, that is, the database of videos and sound files, can be played / performed by the artists as well as by the users. This is possible within the project structure, which connects several media. The ephemeral, the live performance, is it at this point an environment for experimentation (a laboratory?) as it was early in the research?
Nuno Correia: Indeed, I feel that it still is. I try to bring the ephemeral and the fixity as close together as possible, and I use different approaches towards this. I think this is more of a cross-media issue than an interaction design issue. I see performance as composed of three parts: the preparation (preparing the audiovisual content and software/hardware); the performance itself; and the documentation of the performance, which allows for sharing and re-experiencing the performance in some way. In my work, I try to open up the preparation part and the documentation. And I also open up the performance, so that anyone can become a performer with my software and materials. In many cases, a performance can be an “audience of one” as identified by Axel Stockburger, where “composer, performer and audience converge in the playing subject” (Stockburger 2009, p.122). So I’d like to discuss these three parts in more detail.
Regarding performance preparation, I believe that I should open up the materials I create for others to use it in their performances (“audiences of one” or other), not just me. And I try to facilitate ways for others to do the same. The web is a wonderful resource for that. In AV Clash, for example, I use Freesound.org as the sound source. My own sound loops have been uploaded to Freesound.org and tagged as “avclash”, which is a way to bring any sound from Freesound to the project. Regarding AVVX, I’ve created an online repository of visuals, avvx.org, where people can upload their own visuals, and download visuals from the AVVX community. This repository is still in a very early stage. Concerning software, with AVVX I began releasing my software as open-source (available from avvx.org).
In performances, I would like to dilute the border between a “traditional” audiovisual performance and other means of presentation. For example, I release my performance projects also as net art. I’ve been presenting Master and Margarita and AV Clash in performances from the browser. So there is no major difference between the tools I’m using in my performances and what any person could access from the web. Except knowledge of the tool and performance style. I see installations, for example, as performances. They are somehow exploratory performances – the users are mainly experiencing the tools for the first time, discovering the possibilities, and often also “entertaining” the other persons attending the installation space. I quite enjoy it when different presentation means are mixed together. For example, at the Pluto Festival in Belgium, in the end of 2012 (http://pluto-festival.be/#home), AV Clash was presented as installation and performance. People who tried out the installation could then watch me performing with it (or vice-versa). I would often drop by the installation space and do short demos. I distributed stickers with the AV Clash URL (www.avclash.com), so people could also continue using it at home.
Finally, concerning documentation, I try to document my performances as much as possible, and to distribute that documentation on the web. I also aim to facilitate ways for others to document and distribute. For example, AVVX groups have been set up on Facebook and Vimeo. Regrettably, none of my projects so far has a “record” button. But both with AV Clash and AVVX, users have done their own screen recordings, and these have been shared on Facebook and/or Vimeo. AVVX has also been used by members of the community in their own performances (outside of the scope of AVVX workshops), and some of the documentation materials have been shared.
In conclusion, the ephemeral is extremely important. The state of flow one enters when performing live, and the social component of sharing a performance experience with others, are the reasons I started doing performance work to begin with. But the ephemeral experience can be extended into the fixity: upstream – to the building blocks of the performance – and downstream – to the performance recordings and sharing of those documentation materials. All of these elements can be revisited and recreated by the audience.
– Chion, M., 1994. Audio-Vision: Sound on Screen, New York: Columbia University Press.
– Himanen, P., 2001. The Hacker Ethic: A Radical Approach to the Philosophy of Business, New York : Random House.
– Jordà, S. et al., 2007. The reacTable: Exploring the Synergy Between Live Music Performance and Tabletop Tangible Interfaces. In Proceedings of the 1st international conference on Tangible and Embedded Interaction. Baton Rouge, Louisiana: ACM, pp. 139–146.
– Kubovy, M. & Schutz, M., 2010. Audio-Visual Objects. Review of Philosophy and Psychology, 1(1), pp.41–61.
– Lipscomb, S.D., 2002. Modeling Multimedia Cognition: A Review of Nicholas Cook’s Analysing Musical Multimedia. Intégral, (16/17), p.225/236.
– PHELAN, Peggy (1996) – Unmarked: The Politics of Performance. London and New York: Routledge. ISBN 0-415-06822-3.
– Shalom, G., n.d., An Introduction to Hypercubism. Available at: http://www.gabrielshalom.com/theory/
– Shamma, D.A. & Shaw, R., 2007. Supporting creative acts beyond dissemination. In Proceedings of the 6th ACM SIGCHI Conference on Creativity & Cognition. C&C ’07. New York, NY, USA: ACM, pp. 276–277.
– Stockburger, A., 2009. An Audience of One – Sound Games as a Specific Form of Visual Music. In C. Lund & H. Lund, eds. Audio.Visual – On Visual Music and Related Media. Stuttgart: Arnoldsche Art Publishers, pp. 116–124.