Completing the Circle: Incorporating Evaluation Methods in Creative Work

The event took place on January 19^th 02009 in Covent Garden, London, at the BCS headquarters. This one-day symposium was organised by Stephen Boyd-Davis from the Landsdown Centre for Electronic Arts and supported by the Design Research Society DRS, the British Computer Society BCS and CAS, the Computer Art Society. Presenters were Piotr Adamczyk from the Metropolitan Museum of Art, NYC, Mark Springett from Middlesex University, Robin Hawes from the University College Falmouth, Michael Hohl (the author) from the University of Hertfordshire, Tony Renshaw and Richard Stevens from Leeds Met, Jarmo Laaksolahti from Stockholm University and finally Ernest Edmonds from the University of Technology, Sydney.

The call for papers had been refreshingly controversial and provocative by suggesting that quantitative methods such as skin conductivity level measurements or eye-tracking were useful for a better understanding of the recepiences' perception ... possibly insinuating that the work could be 'improved.'

The event clearly was a great success and attracted a diverse audience from computing, the arts and from design.

Presentation 1. Piotr Adamczyk explores ubiquitous and pervasive computing. I find his visual aids very interesting. They include semantic-differentials (which people use for tagging experiences such if an item is fashionable or unfashionable on a - to + scale) or Cartesian Planes.
He showed the work of a colleague, of an audio-tour in which the walking speed controlled the tempo of the audio, speeding it up or down. Only when the tour-ist walked the "right" speed the voice would be in the proper speed too, such giving a guidance.
V2 Capturing Unstable Media: Essential in these ephemeral pieces is that the documentation becomes very important. It is what remains when the project or the performance is over. The documentation almost became the piece.
In Blast-Theory drop-outs of the GPS were perceived as stimulating, interesting and not primarily detrimental.
Piotr mentions that HCI methods in the arts were turning into another 'design tool'. Through engagement with HCI artists values were changing, becoming almost dominated by the clear HCI frameworks. Are they becoming 'reflective design'?
Value Centred Design: System-centred 70s, User Centred 80s, Context-Centred 90s. Mapping the values from another discipline such as Ethnography, Participatory Design or Situationism had its own set of problems.
Mapping Science, Places & Spaces, or How Scientific Paradigms Relate: http://didi.com/brad/mapOfScience/, a topic-map: "The Listening Post"

Presentation 2. Mark Springett investigated cause and effect in user evaluation and negative user experience.
His interest lies in understanding 'trust' aspects of web-design. I thought this had been more or less well explored ... but he presented results from is own research analysing what users of bank websites perceived as "untrustworthy" or even sinister. He distinguished between a 'hedonistic' and a 'instrumental' User Experience. The former being about novelty or beauty while the later was about achieving goals. The evaluation allowed getting into the users minds. "Trust" was often perceived as boring (for the user, or the designer?) but is from a system's design perspective a key quality.
He introduced a Phenotype/Genotype analysis, distinguishing between the 'surface' of the website and how users perceive its causality/make use of it. Values and beliefs of the organisation. (I believe the designers play a less crucial role here then the management: How do they want to be perceived? This usually requires input from the top-management side.)

Presentation 3: Robin Hawes (has a PhD in art psychology? and) introduced us to his "Vision and Reality in Art." Among his main interests is perception, cognition and how our brain makes sense of the world. "The brain as a medium." He explained that our eyes move in approximately five saccades a second and that we had no vision while our eyes were moving. "We are blind for about 90 Minutes every day." We "see" with the brain.
He showed a number of his provocative works of art that very much reminded of 'critical design', artefacts that appear to provide a solution yet actually raise interesting question about our relationship to artefacts, consciousness and the world around us. "Psychology + Physiology + Philosophy = Art"
His research compares eye-tracking data used on "real" photographs and on abstract artworks. How does the human eye study these? The study reveals that while we study human faces (or better photographs of human faces) in a very similar pattern - abstract images appear to be scanned very individualistic.
Quote Damasio: "We are so biologically similar among ourselves, that we construct similar neural patterns of the same thing. It should not be surprising therefore, that similar images arise of those similar neural patterns. That is why we can accept, without protest, the conventional idea that each of us forms in our minds the 'reflected' picture of some particular thing. When in reality, we don't" Damasio, A, (2004), Looking for Spinoza: Joy, Sorrow and the Feeling Brain
In his practical work he uses foveal vision determined by eye-tracking equipment to distort or alter a projected still-life painting. (To me it wasn't clear if this is taking place in realtime or is rendered later. It also reminds a little of "Zerseher" (“Disviewer”) by Joachim Sauter and Dirk Lüsebrink from 1992. Here the viewer's gaze factually 'destroys' the painting by blurring the area his/she is looking at.)

Presentation 4: I gave a presentation of how i developed a Grounded Theory from Interview data with participants of my immersive, telematic environment and explained the relevance of my findings. There would have been much more to say, such as the hierarchy of the senses, spatiality and transportation ... but i had to be brief. Still there was an unequalled amount of time: 35 Minutes. The longest presentation I believe i ever gave!

Presentation 5: After the break Tony Renshaw and Richard Stevens gave an overview over their extensive series of eye-tracking studies with 175 viewers watching Hitchcocks' film "Vertigo." Very original and tongue in cheek was the didactic approach of their ppt. It was an actual demonstration of how film directors (or magicians) use our perceptive apparatus to manipulate our gaze. Ten minutes into their presentation - that made abundant use of the many colourful graphics and motion effects of the MS repertoire - they repeated the entire presentation, revealing to the audience which techniques they had used in each slide to direct our gaze. This was IMO a rather brilliant didactical method by not only explaining those rules of perception but actually being demonstrated their effectiveness in person. This was well worth the ten minutes introduction into film-techniques such as the "180° rule", foreground - background, 'shallow focus' etc., especially regarding the topic of eye-tracking and film rhetorics. From the eye-tracking data of all 175 people they had created a cummulative heatmap and underlying it with b/w footage of the original footage. Their findings are fascinating: There was no difference between male and female viewers. There is an interesting difference between the eye-tracking data and interviews: People's reasoning is very different to what tracking data shows (Maybe it would be interesting to record the sessions of people watching themselves watching and commenting?). People study static images and moving images very differently.

From what i perceive people firstly look, in that hierarchy, at: things that move, eyes, mouth once people talk. Their research appears to be a major research project involving a whole team of researchers and i believe we have only seen a brief overview of their rich results.

Personally it is a very intense experience of deconstructing film and understanding its rhetorics. On one hand it is historic, in that it reveals the director's method (conscious/unconscious skill?) of guiding our gaze, thus enabling us to understand how the film 'works.' Secondly the experience also points to own practice and the future: One cannot not apply what one learned during this session in own making.

Presentation 6: Jarmo Laaksolahti presented their extensive and fascinating research into their 'biomorphic objects." An evaluation method for non-verbal communication of emotions during gaming. Very interesting approach involving eight "semantic objects", such as a "spikey" or a "bubbly" object - that are used by gamers to express their emotions during a particular phase of the game. (What stupefies me at the moment is: How can people hold play the game AND hold the "expressive object" at the same time?) The teams motivation is to find empowerment for the audience beyond language, to let evaluation become an enjoyable and engaging experience.
My impression is that the method has some drawbacks. While claiming that no language is involved the method is qualitative not quantitative. Observers take notes and describe all the activities of participants in language or a notation system in so called 'transcripts'. This again brings all the drawbacks of language. How important are those transcripts? Is there a notation system that allows different observers to code in identical, repeatable terms? Isn't this transcript prone to reactivity such as mood and personality of the observer/note taker? I believe that letting people draw, scribble and doodle on a moving tape of paper (timeline) would provide more conclusive although much more individual results of emotional states. It would provide the markers of an emotional conversation, a feel for the ambience etc. I think we can learn to interact with abstract objects while we explain something. Hiroshi Ishii from MIT's "Tanglible Media Group" gave some examples of how people that had been exposed to 'tangible media' began using objects too when they tried to explain something. However these were everyday objects that began to symbolise something in a particular context. Here it is a different situation ... it appears more passive and more of a reaction too something. A very exciting approach, complex and ambiguous.

Presentation 7: Ernest Edmonds presented research into interactive art in public places and three different perspectives upon the work (curator, evaluator, artist). Thoughts on art and creative media.

Resources: A 1973 Leonardo paper on interaction + computers, Dewey's comment that the "visitor creates the art." The studio becomes a laboratory. At their setup artists show interactive work that is in progress. They witness the audience interacting with it. As a method (protocol analysis + video cued recall) visitors are filmed during their visit and later watch this film and comment on their actions and experience on the screen. This input gives the artists valuable feedback to immediately change their work. Visitors reflect on their art experience. A shift has taken place from 'usablity' to 'experience architecture.'
The artist view: "the functioning of the artwork"
The curator view: "the audience-artwork encounter"
The evaluator view: "human behaviour and cognition"

Another level includes 'engagement':
attractors: What attracts visitors initially? Do they walk past? What makes things attractive?
sustainers: Once people got it - do they still stay?
relaters: Do people come back? Do they send friends?

attractors: change attracts visitors
sustainers: order and complexity makes them stay (Once they understand, the aha! moment, they build a mental model; Now it isn't the contradictory 'change' that has attracted them but the predictability, the pattern)

Learned:
- Artists learn from seeing work in action with real people
- the tension between clarity and complexity
- not usability or smoothing or difficulty
- audience as material
- reducing audience - artwork gap
- audience experience gap

Leonardo transactions http://www.leonardo-transactions.com/
The project website: http://www.creativityandcognition.com/

A great inspiration as been their particular approach consisting of the 'three perspectives" together with their methods of providing feedback to the artists is a huge step in closing the gap between the work and the audience. IMO it is a beautiful, practical and down-to-earth solution. Especially as it appears to be very natural "bottom up" approach in which the work changes over time in small iterations. Such an approach is probably suited much better for individual artworks that make use of novel interactions. We all are learning in this process.

Links:

Abstracts: http://www.cea.mdx.ac.uk/?location_id=116
Landsdown Centre for Electronic Arts: http://www.cea.mdx.ac.uk/
Design Research Society, DRS, UK: http://www.designresearchsociety.org
Computer Art Society, CAS, part of the BCS, the British Computer Society: http://www.computer-arts-society.org/