How the Illusion of Touch establishes engagement between human and embodied agent

This research has been published at CHI '16 Extended Abstracts: ACM Conference on Human Factors in Computing Systems [ ACM | Paper | Poster ].

Abstract

This project explores the quality of engagement between human and agent in large displays. We designed a musical application and introduced the illusion of touch as a novel interaction concept where the user and the agent interact through mediated virtual touch. We conducted a lab study to better understand users’ behaviors with our system. Our method relies on Poggi’s definition of engagement, the Temple Presence Inventory (TPI) questionnaire, observations, and video analysis. Our main findings from the design exploration are: 1) users are engaged in the interaction with an interactive agent even though the agent is not perceived as engaged by the user; 2) users are more involved in the experience when the agent is interactive; and 3) the agent’s gaze is effective in grabbing the user’s attention. These findings provide initial insights on designing for engagement in displays that make use of agents and should be validated with a field study.

Participant attempted to dance with the agent

Participant performed wide arm gestures

Participant had to bend to hold the agent's hand

Design Implications

1. In designing displays in spacious settings, freedom of movement is necessary to allow maximum discoverability of the system’s features. Hand and feet interaction whether in the form of dance or other full-body movements are also seen as playful but could be mapped to specific meanings that can trigger actions for an intuitive and richer experience.
2. Establishing awareness that an agent gazes at humans is an essential first step to building and eventually sustaining engagement. Going further, the ability of simulating an illusion of touch through the user’s silhouette presents a novel way of interaction between human and agent in large displays. It is a potentially powerful concept that captures the need for a mediated virtual touch in interaction.
3. The initial insights can be applied to other domains. When agents are used as artificial companions to the elderly, for example, the illusion of touch can add another layer in the communication pipeline. The use of this concept can also make for a fuller and enjoyable experience in immersive gaming.

Demo

Limitations

There is a need to conduct field study to validate the initial findings. This research, however, opens up new directions to understand how the illusion as opposed to the sensation of touch can engage human and agent. The comparison between the two could pave way for research that explores whether perception outweighs sensation in human-agent interaction in large displays.

Interaction has been defined in a very limited way in this study. It is necessary to investigate other ways where interaction also involves the agent’s conversational aspect and the nuances in communication on top of its ability to respond to touch.

Read the paper for more details.

Tools

Unity, C#, Microsoft Kinect, Greta (Embodied Conversational Agent)