Augmented Reality Art: the Emotional Compass featured on a new book

The Emotional Compass featured in a chapter of the new “Augmented Reality Art” book from Springer, edited by Vladimir Geroimenko together with Mark Skwarek, Tamiko Thiel, Gregory L. Ulmer, John Craig Freeman, Conor McGarrigle, Patrick Lichty, Geoffrey Alan Rhodes, Jacob Garbe, Todd Margolis, Kim Vincs, Alison Bennett, John McCormick, Jordan Beth Vincent, Stephanie Hutchison, Ian Gwilt, Judson Wright, Nathan Shafer, Salvatore Iaconesi, Oriana Persico, Dragoş Gheorghiu and Livia Ştefan, Simona Lodi, Margaret Dolinsky, Damon Loren Baker.

view Augmented Reality Art on Springer

In the book our contribution is titled: “An Emotional Compass: Emotions on Social Networks and a new Experience of Cities

cite as:

Iaconesi, S. and Persico, O. (2014). “An Emotional Compass: Emotions on Social Networks and a new Experience of Cities” in Augmented Reality Art: From an Emerging Technology to a Novel Creative Medium, part of the Springer Series on Cultural Computing, Geroimenko, Vladimir (Ed.). New York: Springer. ISBN 978-3-319-06202-0. http://www.springer.com/computer/hci/book/978-3-319-06202-0

Here is a short sample of the introduction of the chapter:

“The map is not the territory.” (Korzybski, 1933)

 

“The map is not the thing mapped.” (Bell, 1933)

 

“The tale is the map that is the territory.” (Gaiman, 2006)

 

“We say the map is different from the territory. But what is the territory? The territory never gets in at all. […] Always, the process of representation will filter it out so that the mental world is only maps of maps, ad infinitum.” (Bateson, 1972)

 

When we experience territories, we create stories. We model these stories using mental maps, referring to one person’s point of view perception of their own world, influenced by that person’s culture, background, mood and emotional state, instantaneous goals and objectives.

 

If we move along the streets of my city in a rush, trying to find a certain type of shop or building, our experience will be different than the one we would have had if we were searching for something else.

 

Focus will change. We will see certain things and not notice other ones which we would have noticed otherwise. Some things we will notice because they are familiar, common, or because associate them to our cultures, to memories and narratives. All this process continuously goes on as our feelings, emotions, objectives and daily activities change, creating the tactics according to which we traverse places and spaces, to do the things we do.

 

In the density of cities, this process happens for potentially millions of people at the same time. In his “the Image of the City” (Lynch, 1960), Lynch described cities as complex time-based media, symphonies produced by millions of people at the same time in their polyphonic way of acting, moving, interpreting, perceiving and transforming the ambient around themselves: a massive, emergent, real-time, dissonant and randomly harmonic, work of time-based art with millions of authors that change all the time.

 

In this, our mental maps – the personal representations of the city which we build in our minds to navigate them to fulfill our needs and desires – live a complex life as our perception joins into the great performance of the city.

 

Dissonance is the essence of the city itself, and represents its complexity, density and opportunities for interaction.

 

Augmented Reality Art

Augmented Reality Art

an Emotional Compass: new ideas for wayfinding in cities

Why would we need an Emotional Compass?

And, first of all, what is an Emotional Compass?

“The map is not the territory.” – A. Korzybski

 

“The map is not the thing mapped.” – E.T. Bell

 

“The tale is the map that is the territory.” – N. Gaiman

 

“We say the map is different from the territory. But what is the territory? The territory never gets in at all. […] Always, the process of representation will filter it out so that the mental world is only maps of maps, ad infinitum.” – G. Bateson

When we experience territories, we create stories.

We model these stories using mental maps. These maps have seldom anything to do with what actually lies within the territories themselves. A mental map refers to one person’s point of view perception of their own world, and is influenced by that person’s culture, background, mood and emotional state, instantaneous goals and objectives.

If we move along the streets of my city in a rush, trying to find a certain type of shop or building, our experience will be different than the one we would have had if we were searching for something else.

Focus will change.

We will see certain things and not notice other ones which we would have noticed otherwise.

Some things we will notice because they are familiar, common, or because associate them to memories and narratives. Some will stand out because they react with some element of our culture or background.

All this process continuously goes on as our feelings, emotions, objectives and daily activities change, creating the tactics according to which we traverse places and spaces, to do the things we do.

In the density of cities, this process happens for potentially millions of people at the same time.

In his “the Image of the City”, Kevin Lynch described cities as complex time-based media, symphonies produced by millions of people at the same time in their polyphonic way of acting, moving, interpreting, perceiving and transforming the ambient around themselves: a massive, emergent, real-time, dissonant and randomly harmonic, work of time-based art with millions of authors that change all the time.

In this, our mental maps – the personal representations of the city which we build in our minds to navigate them to fulfil our needs and desires – live a complex life as our perception joins into the great performance of the city.

Dissonance is the essence of the city itself, and represents its complexity, density and opportunities for interaction.

Harmony represents affordances, the things which are recognised and shared by different cultures.

Those elements of the perceptive landscape onto which we can agree upon, which we recognise and attribute compatible meanings, allowing us to collaborate, meet, do things together.

For example, Haken and Portugali have suggested a broad definition of landmarks to refer to any distinguished city elements that shape our mental map. Or as Appleyard, Golledge and Spector who have conducted studies about the imageability of urban elements not because of their visual stimulus but because they possess some personal, historical, or cultural meaning.

These features found within our mental maps enable the possibility to design the affordances of places and spaces. We can use the understanding of what is consistently recognized and understood to design the elements of space/time which will be able to describe to people what is allowed or prohibited, suggested or advised against, possible or imaginable.

Lynch’s concepts of legibility and imageability are closely related to James J. Gibson’s notion of affordances developed in his direct perception theory, according to which the objects of the environment can afford different activities to various individuals and contexts. And, again, in Haken and Portugali, all elements of a city afford remembering, as they shape in the mental maps in human minds.

In a further step in the direction of citizen activation, we can also imagine to make this type of understanding widely known and usable, to enable people to express themselves (and their mental maps of how they perceive the world) more effectively and powerfully.

These possibilistic scenarios have become radically viable with the widespread of ubiquitous technologies. Nomadic devices (such as smartphones) and their applications we are able to merge our physical understanding of the world with the digital one: the two have, in fact, become so interweaved and interconnected as to form a new physicality, visuality and tactility which shape our everyday experiences of the world.

According to Mitchell’s “City of Bits”, McCullough’s Digital Ground, Zook’s and Graham’s DigiPlace we are constantly immersed in emergent networks of interconnected data, information and knowledge which is produced by millions of different sources and subjects in the course of their daily lives.

This data and information radically shapes the ways in which we have learned to work, learn, collaborate, relate, consume and perceive our environment.

If we are strolling in a park and we receive a notification of some sort on our smartphone, the natural environment could instantly transform into an ubiquitous, temporary office.

If we want to make a decision about a certain thing we would like to purchase while in a shop, a quick look online will help define our opinion in ways that can be very powerful.

If we receive a message on our smartphone, our mood could change for the rest of the day.

Situated and ubiquitous information is able to powerfully transform, in real-time, the ways in which we experience places, objects and services, by providing the wide accessibility of other people’s stories, emotions, expectations and visions.

This scenario is the one we have tried to address in our research: the conceptualisation, design and implementation of a tool for urban navigation, in which the emotional, narratives expressed by people while inhabiting and using urban places, spaces and objects become instantly and radically available, accessible and usable.

We used this approach to define a novel vision on the opportunity to design new types of affordances for our cities.

We have decided to start from the idea of a Compass.

You can find a first result of our research here at the following link:

An Emotional Compass Harvesting Geo-located Emotional States from User Generated Content on Social Networks and Using them to Create a Novel Experience of Cities

An Emotional Compass harvesting emotions from social networks

An Emotional Compass harvesting emotions from social networks

Augmented Reality: the Augmented City, communication and citizenship

On May 17th 2012, we took part  to prof. Marco Stancati’s course of Media Planning at La Sapienza University of Rome with a lecture on the scenarios offered by Augmented Reality to the creation of novel opportunities for communication and business.

HERE you can find some information about our lecture and the MediaPlanning course.

HERE you can download the slides we used for the lecture

(The slides are a lighter version of the ones we used in class, which were full of videos and hi-res images: please feel free to contact us should you want the original ones)

In the lecture, we started from a series of definitions about what, in our times, can be considered as “Augmented Reality”

Augmented Reality in the city

Augmented Reality in the city

In our definitions we chose to describe a wider form of the term, not limiting it to the set of applications to which we’ve all been accustomed to , and abandoning for a moment the vision of people happily strolling through cities with their smartphones raised in front of their faces.

Nonetheless we used classical examples of AR to introduce a possible evolution of what is/will be possible in our cities using ubiquitous technologies.

We focused on the idea of the Augmented City.

augmented city and its many voices

augmented city and its many voices

In this vision of the city, many subjects (individuals, organizations and, using sensors, also the city itself) add layers of digital information, in real-time. We can access and experience these  sets of information in multiple ways, and we can also use them to compose, dynamically, our personal vision of the city, by remixing, re-arranging, re-combining and mashing-up all the information layers which are available.

This is a very interesting situation for cities and their citizens, as it enables for the creation of entire new scenarios for communication, business and personal expression.

It also opens up possibilities which will probably have a high impact on the ways in which, for example, enterprises design their own products, and the ways in which they create the strategies according to which products and services are communicated, marketed, monitored.

We discussed this scenario below, among the many possible:

augmenting the voices on products

augmenting the voices on products

The physical packaging of products usually hosts information and messages which are created by a very limited number of voices (e.g.: the manufacturer, marketing team..).

In the drawing we see depicted a scenario which is becoming progressively more frequent: a multiplicity of subjects are now able to join the Brand in adding digital information to products and services, using Augmented Reality, QRCodes, computer vision, tagging (e.g.: RFID) and location based technologies. We call this Ubiquitous Publishing.

For example, in the Squatting Supermarkets project we used products’ packaging as visual reference for critical Augmented Reality experiences. In the performance, people could use their smartphone to “look at” products on a supermarket shell. When they did, a series of information became available:

  • a map, created using MIT’s Open Source Map, showing where the product and its components came from, where its materials came from, where it had been processed and assembled, and in which places it stopped during transportation; the product becomes a map of the planet highlighting all the places which it touched during its manufacturing and distribution processes;
  • a series of visualizations showing the product’s composition, the percentages of organics, chemicals, fat, aromas… all shown through interactive information visualizations.
visualizations in Squatting Supermarkets

visualizations in Squatting Supermarkets

One of the visualization was analyzed in deeper detail. On the top right of the previous image, is a timeline of the real-time conversations about the product: the timeline scrolls left and right and each colored block represents a conversation and its general sentiment (meaning: the sentiment which is most represented in the conversation); green, yellow and red code positive, mixed and negative sentiments.

So: while strolling through the aisles of a supermarket, you take a take a picture of your favorite product and you are able to see what people on social networks are saying about it, in real-time.

We observed this possibility (to publish real-time, user generated information using ubiquitous publishing techniques and accessible information visualizations) to describe an interesting loop which we are able to make.

We can imagine (and do) to harvest user generated information in real-time about our topics of interest (from blogs, websites, social networks and social media sites) and to publish them when/where they are most useful.

VersuS, the real-time lives of cities

VersuS, the real-time lives of cities

The image above shows the experiment we performed with the VersuS project during the city-wide riots taking place in Rome on October 15th 2011.

The 3D surface covering the map of the city of Rome shows the intensities of the social network conversations taking place during the protests and riots. The image is part of a real-time visualization through which we have been able to observe how social media conversations closely followed the evolution of the protest.

By using harvested conversations (from Facebook, Twitter, Flickr and Foursquare) we have been able to analyze what was being said and where, and we actually were able to demonstrate how a massive amount of useful information was being published by users: about violence, injuries, possible escape routes, missing people. All this information could have been actually collected and organized, and accessed by protesters through specifically designed interfaces to achieve important, pragmatic results such as avoiding being hurt, finding safe escape routes from the riots or find our friends who were lost in them.

We were able to design, for example, the simple Augmented Reality application for smartphones pictured below:

Augmented Reality for Riots

Augmented Reality for Riots

Augmented Reality for Riots

Augmented Reality for Riots

This experimental interface shows how a rioter could have visualized on the screen of the smartphone the degree of safety in the direction he/she was facing, as it could be inferred by the social media conversations with a geographic reference.

An immediate, easy to use tool to achieve important goals.

During the lesson we focused on how it would be possible to use these technological opportunities to conceive and enact innovative communication practices.

We described a couple of scenarios, which we can imagine being applied in different forms, ranging from the creation of scenarios for the public lives of cities and their citizens, to the needs of communicators for their work with enterprises and administrations, to the needs of marketing and advertising.

In synthesis, we imagined a novel, more extensive, definition for Augmented Reality, according to which a loop is formed among the digital and physical world.

In this definition of AR it is possible

  • to harvest user-generated (as well as database and sensor generated) real-time information about relevant places/topics/products/services,
  • to process it using techniques such as Natural Language Processing and Sentiment Analysis,
  • to publish it ubiquitously, where/when it is more useful, using interfaces and interaction schemes which ensure accessibility and usability (including smartphone apps, urban screens, wearable technologies, digital networked devices, information displays…) and
  • to provide ways according to which users are able to both contribute to the flow of information and to re-assemble and re-interpret it, creating additional points of view