XVI Congreso Internacional de Interacción

Persona-Ordenador
7 al 9 de SepƟembre de 2015
Vilanova i la Geltrú. Barcelona. España

Actas del XVI Congreso Internacional de  
Interacción Persona‐Ordenador. 
Interacción 2015

EDITORES
Pere Ponsa, José Antonio Román, Daniel Guasch
 XVI Congreso Internacional de Interacción Persona-Ordenador

Del 7 al 9 de Septiembre de 2015, Vilanova i la Geltrú, Barcelona, España

EDITORES

Pere Ponsa
José Antonio Román
Daniel Guasch
Escola Politècnica Superior d’Enginyeria Vilanova i la Geltrú

Universitat Politècnica de Catalunya

Maquetación: Yolanda Guasch

Copyright: ACM
ENTIDADES ORGANIZADORAS

Asociación Interacción Persona Ordenador

ICE PhD Erasmus Mundus Joint Doctorate

in Interactive and Cognitive Environments
 PATROCINADORES

KUKA Robots Ibérica

Hewlett Packard

Universitat Pompeu Fabra.

Barcelona School of Management

Tobii

ASAC group, Universitat Autònoma de Barcelona

 COMMITTEE

HONORARY CHAIR

Enric Fossas
President of the Universitat Politécnica de Catalunya BarcelonaTech, Spain

Conference Chair

Pere Ponsa
Technical School of Vilanova, Spain.

Conference Co-Chair

José Antonio Román

Technical School of Vilanova, Spain

Program Chair

Daniel Guasch
Technical School of Vilanova, Spain.

Special Track Chairs

Cristina Manresa-Yee
University of Balearic Islands

Ann Morrison
Aalborg University

Workshop Chairs
Ana M. González Ramos
GENTIC, Internet Interdisciplinary Institute, IN3, Open University of Catalonia

Carina González
Department of Computer Engineering, University of La Laguna

Concurso AIPO TFG/TFM Chair

Asociación Interacción Persona Ordenador, AIPO
Student Design Challenge Chairs
Núria Ferran
Universitat Oberta de Catalunya, Open University of Catalonia

Muriel Garreta
Universitat Oberta de Catalunya, Open University of Catalonia

XVI Congreso Internacional de Interacción Persona-Ordenador

 PROGRAM COMMITTEE

Abascal, Julio Bravo, Crescencio

Universidad País Vasc, Spain Universidad de Castilla-La Mancha,
Spain
Abdulrahman H. Altalhi.
King Abdulaziz University,Arabia Saudí Carro, Rosa Maria
Universidad Autónoma de Madrid,
Aciar, Silvana Vanesa Spain
Universidad Nacional de San Juan,
Argentina Carron, Thibault
University of Savoie, France
Acuña, Silvia
Universidad Autónoma de Madrid, Ceres, Ramón
Spain Instituto de Automática Industrial
(CSIC), Spain
Alamán, Xavier
Universidad Autónoma de Cerezo, Eva
Madrid,Spain Universidad de Zaragoza, Spain

Albedawi, Ibrahim Castro, Silvia

King Abdulaziz University, Arabia Universidad Nacional del Sur,
Saudí Argentina

Arellano, Diana Cetina, Carlos

Filmakademie Baden-Wuerttemberg, Universidad de San Jorge, Spain
Germany
Clunie , Clifton
Azorín, José M. Universidad Tecnológica de Panamá,
Universidad Miguel Hernández, Spain Panama

Baldassarri, Sandra Clunie, Gisela

Universidad de Zaragoza, Spain Universidad Tecnológica de Panamá,
Panama
Bez, Marta
Universidad de Rio Grande do Sul, Cole, Ian
Brazil University of York, United Kingdom

Borges, Marcos Collazos,César

Universidad Federal de Rio de Janeiro, Universidad del Cauca, Colombia
Brazil
Coto, Mayela
Botella, Federico Universidad Nacional de Costa Rica,
Universidad Miguel Hernández, Spain Costa Rica

XVI Congreso Internacional de Interacción Persona-Ordenador

 PROGRAM COMMITTEE

De Castro, Carlos Garrido, José Luis

Universidad de Córdoba, Spain Universidad de Granada, Spain

Deco, Claudia Gil, Rosa M.

Universidad Nacional del Rosario, Universitat de Lleida, Spain
Argentina
Giraldo, William
Díaz, Antonio Universidad del Quindío, Colombia
Universidad de Málaga,Spain
Giulianelli, Daniel
Fardoun, Habib Universidad Nacional de La Matanza,
King Abdulaziz University, Arabia Argentina
Saudí
González, Carina
Fabregat, Ramón Universidad de la Laguna, Spain
Universidad de Girona, Spain
González, José Mariano
Fernández, Eduardo Universidad de Sevilla, Spain
Universidad de la República, Uruguay
González, Juan Manuel
Ferré, Xavier Universidad Autonoma de Puebla,
Universidad Politécnica de Madrid, México
Spain
González, Mª Paula
Frizera Neto, Anselmo CONICET, Argentina
Federal University of Espírito Santo
(UFES), Brazil
González, Pascual
Universidad de Castilla-La Mancha,
Gallud, José A. Spain
Universidad de Castilla-La Mancha,
Spain
González, Víctor
Instituto Tecnológico Autónomo de
Garay, Néstor México ITAM, Mexico
Universidad País Vasco, Spain

González-Sánchez, José L.
García, Roberto Universidad de Granada,Spain
Universitat de Lleida, Spain

Granollers, Toni
García-Pañeda, Xabiel
Universitat de Lleida,Spain
Universidad de Oviedo, Spain

Guerrero, Josefina
García-Peñalvo, Francisco Universidad Autonoma de Puebla,
Universidad de Salamanca, Spain
México

XVI Congreso Internacional de Interacción Persona-Ordenador

 PROGRAM COMMITTEE

Guerrero, Luís Macías, José A.

Universidad de Costa Rica, Costa Rica Universidad Autónoma de Madrid,
Spain
Gutiérrez, Francisco
Universidad de Granada, Spain Manresa, Cristina
Universidad de las Islas Baleares,
Haya, Pablo Spain
Instituto de Ingeniería de
Conocimiento, Spain Marco, Javier
Universidad de Zaragoza, Spain
Hernán, Isidoro
Universidad Rey Juan Carlos, Spain Marcos, Mari-Carmen
Universidad Pompeu Fabra, Spain
Hersh, Marion A.
University of Glasgow, Scotland Martín, Estefanía
Universidad Rey Juan Carlos, Spain
Hoppe, Ulrich
University Duisburg-Essen, Germany Marty, Jean-Charles
University of Savoie, France
Jaén, Fco. Javier
Universidad Politécnica de Valencia, Mashat, Abdulfattah
Spain King Abdulaziz University, Saudi
Arabia
Kafure, Ivette
Universidade de Brasilia, Brazil Mendes, Antonio José
Universidad de Coimbra, Portugal
Latorre, Pedro
Universidad de Zaragoza,Spain Molina-Massó, J. Pascual
Universidad de Castilla-La Mancha,
Llamas Nistal, Martín Spain
Universidad de Vigo,Spain
Montero, Francisco
Universidad de Castilla-La Mancha,
López, Juan Miguel
Spain
Universidad País Vasco, Spain

López, Víctor Mor-Pera, Enric

Universidad Oberta de Catalunya,
Universidad de Castilla-La Mancha,
Spain
Spain

Lozano, María Morán, Alberto

Universidad Autónoma de Baja
Universidad de Castilla-La Mancha,
California (UABC), Mexico
Spain

XVI Congreso Internacional de Interacción Persona-Ordenador

 PROGRAM COMMITTEE

Moreno, Lourdes Pastor, Óscar

Universidad Carlos III de Madrid, Universidad Politécnica de Valencia,
Spain Spain

Moreno, Mario Penichet, Víctor

Universidad Tecnológica de la Mixteca, Universidad de Castilla-La Mancha,
México Spain

Moreno-Ger, Pablo Peñalver, Antonio

Universidad Complutense de Madrid, Universidad Miguel Hernández, Spain
Spain
Plaza, José Antonio
Moriyón, Roberto Instituto de Microelectrónica de
Universidad Autónoma de Madrid, Barcelona IMB-CNM (CSIC), Spain
Spain
Pons, Claudia
Muñoz-Arteaga, Jaime Universidad Nacional de La Plata,
Universidad Autónoma de Argentina
AguasCalientes, México
Pow, José A.
Oliva, Marta Pontificia Universidad Católica del
Universitat de Lleida, Spain Perú, Perú

Ortega, Manuel Puerta, Angel

Universidad de Castilla-La Mancha, RedWhale Software,USA
Spain
Redondo, Miguel A.
Padawreski, Patricia Universidad de Castilla-La Mancha,
Universidad de Granada, Spain Spain

Padilla Zea, Natalia Reyes, Arcadio

Universidad de Granada, Spain Universidad de Málaga, Spain

Palanque, Philippe Ribera, Mireia

Universidad Paul Sabatier, France Universitat de Barcelona, Spain

Panach, José I. Rodríguez, Inmaculada

Universidad Politécnica de Valencia, Universitat de Barcelona, Spain
Spain
Rodríguez, Juan José
Paredes, Maximiliano Bankinter, Spain
Universidad Rey Juan Carlos, Spain

XVI Congreso Internacional de Interacción Persona-Ordenador

 PROGRAM COMMITTEE

Rojas, Luis A. Velázquez, Angel

Universidad Autónoma de Madrid, Universidad Rey Juan Carlos, Spain
Spain
Van der Veer, Gerrit
Rossi, Gustavo Universidad de Vrije, The Netherlands
Universidad Nacional de la Plata,
Argentina Vera, Pablo
Universidad Nacional de La Matanza,
Rusu, Cristian Argentina
Pontificia Universidad Católica de
Valparaiso, Chile Vicari, Rosa
Universidad de Rio Grande do Sul,
Sánchez, J. Alfredo Brazil
Universidad de las Américas Puebla,
Mexico Vilanova, Ramón
Universitat Autònoma de Barcelona,
Sanz, Cecilia Spain
Universidad Nacional de la Plata,
Argentina Winckler, Marco
Sendín, Montserrat Université Paul Sabatier, France
Universitat de Lleida, Spain
Xu, Diana
Serrano, Marcos University of Central Lancashire,
Institut de Recherche en Informatique England
de Toulouse (IRIT), France
Zepeda, Joaquín Sergio
Sevilla Moróder, Joaquin Universidad Autónoma Metropolitana,
Universidad de Navarra, Spain Mexico

Sigut, José F.
Universidad de La Laguna, Spain

XVI Congreso Internacional de Interacción Persona-Ordenador

 ORGANIZING COMMITTEE

Local Committee

Víctor Agramunt
Maria Hortènsia Álvarez
Bernardino Casas
Andreu Català
Yolanda Guasch
José María Merenciano
Montserrat Mestres
Luis Miguel Muñoz
Helena Querol
David Raya
Montserrat Solsona
Frederic Vilà

Technical School of Vilanova, Spain

XVI Congreso Internacional de Interacción Persona-Ordenador

PRESENTATION
 Presentación

En el siglo XXI, el ordenador de sobremesa está siendo desplazado por tecnología que

se mueve con nosotros. El cambio en la forma de interactuar supone revisar los

paradigmas clásicos existentes y avanzar en una interacción natural entre la persona, la

interfaz y el entorno. Interacción 2015 es un congreso internacional fomentado por la

Asociación de Interacción Persona-Ordenador (AIPO) que tiene como objetivo principal

promover y difundir los avances recientes en el área de la Interacción Persona-

Ordenador (IPO), tanto a nivel académico como empresarial. Este congreso es punto de

encuentro de profesionales y académicos en accesibilidad, usabilidad, ergonomía,

robótica, experiencia de usuario, realidad aumentada, etc. La IPO abarca un espectro

multidisciplinar como por ejemplo el diseño de productos-servicios, la sociología,

comunicación audiovisual, ciencias empresariales, bellas artes, psicología, ingeniería

industrial, etc.

La organización de Interacción 2015, XVI edición del congreso, está a cargo de la

Escuela Politécnica Superior de Ingeniería de Vilanova i la Geltrú y cuenta con la

colaboración de la Cátedra de Accesibilidad de la Universidad Politécnica de Catalunya

y el Centro de Estudios Tecnológicos para la Dependencia y la Vida Autónoma

(CETpD) y el apoyo de diversas entidades.

La presente edición del congreso ha recibido un total de 105 trabajos. En concreto, 94

contribuciones científicas procedentes de 22 países, de las cuales se han seleccionado 66

en forma de 41 artículos largos, 21 artículos cortos y 4 trabajos para el coloquio

doctoral. A estas contribuciones se suman los envíos de 9 propuestas para el Concurso

AIPO TFG/TFM 2015 (los finalistas (2) y vencedores (2)) y las propuestas recibidas

(2) del Student Design Challenge. A partir de los tópicos iniciales del congreso, los

trabajos seleccionados se han agrupado en las sesiones: natural user interfaces,

methodologies and models, enGendering Technology, child computer interaction,

XVI Congreso Internacional de Interacción Persona-Ordenador

 Presentación

interface design, interaction devices, interacton for people with disability, games,

usability and user experience, software-architecture and interaction, accessibility and

semantic web.

El congreso cuenta con la presencia de conferencias plenarias de Panos Markopoulos

(Interaction Design for Rehabilitation), Els Rommes (Including Gender in the

Interaction) y José Antonio Plaza (Towards Intracellular Computer-Human Interaction:

a microelectronic perspective), la conferencia de Manel Garrido (Robots Sensitivos) y el

tutorial de Antonio Miguel Baena (Desarrollo de aplicaciones de realidad aumentada

para tabletas y smartphones). Finalmente el panel de discusión sobre el presente y

futuro de IPO a cargo de José Antonio Macías (AIPO) y Marina Talavera (Hewlett

Packard).

En esta edición del congreso se presenta dos concursos en su primera edición: Concurso

AIPO al mejor Trabajo Fin de Grado y Trabajo Fin de Màster TFG/TFM en el ámbito

de la interacción y el Concurso de Diseño Student Design Challenge para estudiantes de

diseño y con el reto de interacción entre ciudadanos y ciudad mediante productos

digitales interactivos.

En Interacción 2015 se ha querido establecer un equilibrio entre el programa científico y

el social, en este sentido la colaboración del Ayuntamiento de Vilanova i la Geltrú ha

estado clave para mostrar aspectos de la vida de la ciudad, el patrimonio cultural

presente en el Museo del Ferrocarril de Catalunya y el Museo Víctor Balaguer. Otra

muestra es la de torres humanas castelleras a cargo de LLunàtics UPC Vilanova.

El programa científico y el programa social se complementan con exposición de

producto en el hall de la Escuela. Un agradecimiento a todas aquellas entidades que han

patrocinado y dado soporte al congreso: KUKA Robots Ibérica, UPF Barcelona School

XVI Congreso Internacional de Interacción Persona-Ordenador

 Presentación

of Management, Tobii, grupo investigación Automatización y Sistemas Avanzados de

Control ASAC 1 UAB, Hewlett Packard.

El congreso agradece a todas las personas que han colaborado en las diversas

actividades programadas. Gracias a José Antonio Macías, presidente de AIPO el soporte

para la confección del Congreso, así como a Víctor M. Penichet, CHISPA, y a todo el

equipo del capítulo español de ACM SIGCHI, por la suma de esfuerzos entre el

congreso y Association for Computing Machinery ACM. Finalmente, in Memoriam

Jesús Lorés por sus esfuerzos en la fundación de AIPO y el impulso de la disciplina IPO

desde el grupo de investigación GRIHO de la Universitat de Lleida.

Pere Ponsa Conference Chair

José Antonio Román, Conference Co-Chair
Daniel Guasch Program Chair

1
Grupo con el que se colabora en programa CiCYT DPI2013-47825-C3-1-R.

XVI Congreso Internacional de Interacción Persona-Ordenador

 Presentation

Interacción 2015 is the International Conference promoted by the Spanish Human

Computer Association (In Spanish: Asociación para la Interacción Persona-Ordenador,

AIPO), whose main objective is to promote and disseminate the recent advances in the

field of Human-Computer Interaction. This conference provides a forum for discussion

and exchange of ideas on design and application of techniques and methodologies with

a multidisciplinary approach (from engineering to human factors, human-robot

interaction, accessibility, interface design, usability, natural interaction, etc.),

The 16th International Conference Interacción 2015 is organized by the Technical

School of Vilanova I la Geltrú (Universitat Politècnica Catalunya Barcelona Tech) and

has the support of the Accessibility Chair, the Interactive Systems Design Lab and the

Technical Research Center of Dependency Care and Autonomous Living.

This edition has received 105 contributions: 94 scientific contributions from 22

countries, 9 papers into the AIPO Challenge best final project (degree, master) and 2

papers into the Student Design Challenge. In the 94 scientific contributions, the 66

accepted papers in these categories: 41 full papers, 21 short papers and 4 papers for the

Doctoral Colloquium. The sessions in this conference are classified in: natural user

interfaces, methodologies and models, EnGendering Technology, child computer

interaction, interface design, interaction devices, interacton for people with disability,

games, usability and user experience, software-architecture and interaction, accessibility

and semantic web.

The conference structure is a set of sessions and a set of social and scientific activities.

In example, the plenary conferences of Panos Markopoulos (Interaction Design for

Rehabilitation), Els Rommes (Including Gender in the Interaction) and José Antonio

Plaza (Towards Intracellular Computer-Human Interaction: a microelectronic

perspective), the Spanish Conference of Manel Garrido (Robots Sensitivos and the

XVI Congreso Internacional de Interacción Persona-Ordenador

 Presentation

Spanish tutorial de Antonio Miguel Baena (Desarrollo de aplicaciones de realidad

aumentada para tabletas y smartphones). Finally, a discussion about the present and

future of HCI with José Antonio Macias and Marina Talavera (Hewlett Packard).

Thank you for the support of all the people, local committee, chairs, entities (Víctor

Penichet CHISPA, Spanish local Chapter of ACM SIGCHI), (José Antonio Macías,

AIPO) and sponsors: KUKA Robots Ibérica, UPF Barcelona School of Management,

Tobii, research group ASAC UAB, Hewlett Packard and Llunàtics UPC Vilanova

(Human Towers).

In memoriam Jesús Lorés for the effort and support of HCI activities through the

GRIHO Research Group from the Universitat de Lleida.

Pere Ponsa Conference Chair

José Antonio Román, Conference Co-Chair
Daniel Guasch Program Chair

XVI Congreso Internacional de Interacción Persona-Ordenador

TABLE OF CONTENTS
CONFERENCES, TUTORIALS AND PANEL DISCUSSIONS, 23
Plenary Conference: Interaction Design for Rehabilitation, 24
Panos Markopoulos
Tutorial: Desarrollo de Aplicaciones de Realidad Aumentada para Tabletas y Smartphones, 25
Antonio Miguel Baena
Conferencia: Kuka – Robots sensitivos, 26
Manel Garrido
Plenary Conference: Including Gender in the Interaction, 27
Els Rommes
Plenary Conference: Towards Intracellular Computer-Human Interaction: a micro-electronic
perspective, 28
José Antonio Plaza
Panel de discusión: Reflexiones sobre la Interacción Persona-Ordenador, 29
José Antonio Macías, Marina Talavera

NATURAL USER INTERFACES (I), 30

Diseño de actividades de mejora de capacidades cognitivas para tabletops tangibles, 31
Clara Bonillo, Javier Marco, Eva Cerezo, Sandra Baldassarri
Analyzing Learnability of Common Mobile Gestures used by Down Syndrome Users, 40
Alfredo Mendoza G., Francisco J. Álvarez R., Ricardo Mendoza G., Francisco Acosta E., Jaime Muñoz A.
Multi-touch Technology in Early Childhood: Current Trends and Future Challenges, 49
Vicente Nacher, Javier Jaen
Vibrotactile Vest and The Humming Wall: “I like the hand down my spine”, 54
Ann Morrison, Cristina Manresa-Yee, Hendrik Knoche

NATURAL USER INTERFACES (II), 62

Exploración del busto humano en tiempo real mediante interacción natural con fines educativos, 63
Roi Méndez, Julián Flores, Rubén Arenas
Performance evaluation of gesture-based interaction between different age groups using Fitts’ Law, 70
Diana Carvalho , Luís Magalhães, Maximino Bessa, Eurico Carrapatoso
Including multi-stroke gesture-based interaction in user interfaces using a model-driven method, 77
Otto Parra González, Sergio España, Oscar Pastor
Interfaces de Usuario Tangibles como Mecanismo de Interacción en Entornos Multi-dispositivos, 86
Elena de la Guía, María D. Lozano, Víctor Penichet, Luis Orozco, Vicente López

METHODOLOGIES AND MODELS, 93

Limitaciones del Modelo de Tareas del W3C para aplicaciones Post-WIMP, 94
Miguel A. Teruel, Arturo C. Rodríguez, Francisco Montero, Elena Navarro, Víctor López-Jaquero,
Pascual González
An Ontology-Driven Approach to Model & Support Mobility and GeoLocation Based Campus
Interactions, 103
Maha Khemaja , Félix Buendía
SEGA-ARM: A Metamodel for the Design of Serious Games to Support Auditory Rehabilitation, 112
David Céspedes-Hernández, Jorge Luis Pérez-Medina, Juan Manuel González-Calleros, Francisco J.
Álvarez Rodríguez, Jaime Muñoz-Arteaga
Towards the Definition of a Framework for the Management of Interactive Collaborative Learning
Applications for Preschoolers, 121

XVI Congreso Internacional de Interacción Persona-Ordenador

 Liliana Rodríguez-Vizzuett, Jorge Luis Pérez-Medina, Jaime Muñoz-Arteaga, Josefina Guerrero-García,
Francisco J. Álvarez-Rodríguez

DOCTORAL COLLOQUIUM, 130

Intelligent Playful Environments for Animals, 131
Patricia Pons, Javier Jaen, Alejandro Catala
Multi-Display Environments to Foster Emotional Intelligence in Hospitalized Children, 133
Fernando Garcia-Sanjuan, Javier Jaen, Alejandro Catala
KINDERTIVITY: Using Interactive Surfaces to Foster Creativity in Pre-kindergarten Children, 135
Vicente Nacher, Javier Jaen
Facial Emotion Analysis in Down’s syndrome children in classroom, 137
Pablo Torres-Carrión, Carina González-González

CHILD COMPUTER INTERACTION, 140

Métodos y técnicas para la evaluación de la experiencia emocional de niños y niñas con videojuegos
activos, 141
Carina S. González-González, Vicente Navarro-Adelantado
The SARA Project: An Interactive Sandbox for Research on Autism, 152
Diana Arellano, Ulrich Max Schaller, Volker Helzle, Reinhold Rauh, Marc Spicker, Oliver Deussen
Model for Analysis of Serious Games for Literacy in Deaf Children from a User Experience Approach,
157
Sandra Cano, Jaime Muñoz Arteaga, César A. Collazos, Viviana Bustos Amador
Uso de Aplicaciones Interactivas para Apoyo a la Escritura en Niños con Problemas de Aprendizaje, 166
Jaime Muñoz Arteaga, Dulce Morales Hernández, Ricardo Mendoza, Carina S. Gonzalez
Enseñando Emociones a Niños Mediante Videojuegos, 171
Noemí Marta Fuentes García, Rafael López Arcos, Francisco Luis Gutiérrez Vela, Patricia Paderewski
Rodríguez, Natalia Padilla Zea

INTERFACE DESIGN, 180

CamScan, an application to identify everyday objects for users with vision impairments, 181
Rodrigo Capa-Arnao, Cristina Manresa-Yee, Ramon Mas-Sansó
Interacción de los Usuarios con Aplicaciones Web Offline: un Caso de Estudio, 189
Félix Albertos Marco, Víctor M.R. Penichet, José A. Gallud
Elaborating a Web Interface Personalization Process, 198
J. Eduardo Pérez, Xabier Valencia, Myriam Arrue, Julio Abascal
Goal Driven Interaction (GDI) vs. Direct Manipulation (MD), an empirical comparison, 202
A. L. Carrillo, J. Falgueras
Generating User Interface from Conceptual, Presentation and User models with JMermaid in a
learning approach, 207
Jenny Ruiz, Gayane Sedrakyan, Monique Snoeck

INTERACTION DEVICES (I), 216

Blind Source Separation Performance based on Microphone Sensitivity and Orientation within
Interaction Devices, 217
Navya Amin, Thomas Gross, Susanne Rosenthal, Markus Borschbach
DIY computer mouse for special needs people, 225
Lluís Ribas-Xirgo, Francisco López-Varquiel

XVI Congreso Internacional de Interacción Persona-Ordenador

 Explorando la Viabilidad de un Sistema de Tracking Inercial para la Mano Basado en un solo Sensor
229
Ernesto de la Rubia, Antonio Diaz-Estrella
An investigation into the comprehension of map information presented in audio, 234
Feng Feng, Tony Stockman, Nick Bryan-Kinns, Dena AI-Thani

INTERACTION DEVICES (II), 242

InclineType – An Accelerometer-based Typing Approach for Smartwatches, 243
Timo Götzelmann, Pere-Pau Vázquez
Basketball Activity Recognition using Wearable Inertial Measurement Units, 248
Le Nguyen Ngu Nguyen, Daniel Rodríguez-Martín, Andreu Català, Carlos Pérez-López, Albert Samà,
Andrea Cavallaro
EMG-based biofeedback tool for augmenting manual fabrication and improved exchange of empirical
knowledge, 255
Guillermo Bernal, Dishaan Ahuja, Federico Casalegno

INTERACTION FOR PEOPLE WITH DISABILITIES, 263

Performing universal tasks on the Web: interaction with digital content by people with intellectual
disabilities, 264
Tânia Rocha, Maximino Bessa, Luís Magalhães, Luciana Cabral
Towards Intelligent and Implicit Assistance for People with Dementia: Support for Orientation and
Navigation, 271
Nam Tung Ly, Jörn Hurtienne, Robert Tscharn, Samir Aknine, Audrey Serna
Terapia Ocupacional para Personas con Discapacidad Física utilizando Ambientes Interactivos, 276
Héctor Cardona Reyes, Jaime Muñoz Arteaga, Francisco Acosta Escalante, Francisco J. Álvarez
Rodríguez, Ángel Eduardo Muñoz Zavala
Experiencias de evaluación de herramientas tecnológicas para la asistencia de personas con
discapacidad cognitiva, 281
Juan C. Torrado, Javier Gómez, Germán Montoro

ENGENDERING TECHNOLOGY (I), 289

An ICT experience in Computer Women Role promotion: WikinformáticA! in Aragón. Promoting the use
of Wiki tools and visualizing the role of women in ICT, 290
María Teresa Lozano, Raquel Trillo-Lado, María Villarroya-Gaudó, Ana Allueva, Eva Cerezo
Inclusion and promotion of women in technologies, 294
Beatriz Revelles-Benavente, Lidia Arroyo Prieto, Nùria Vergés Bosch
Moving towards Accommodating Women with ICT: Paying Attention to Self-inclusion Mechanisms,
298
Núria Vergés Bosch

ENGENDERING TECHNOLOGY (II), 306

Influencia del Género en el Pensamiento Computacional, 307
Elisenda Eva Espino Espino
Dos sentidos de lo tecnológico en relatos de vida de mujeres tecnólogas, 312
Adriana Gil-Juárez, Ester Conesa, Joel Feliu, Montse Vall-llovera
Acercando las mujeres a la ingeniería: iniciativas y estrategias que favorecen su inclusión, 319
Patricia Paderewski, Carina González González, Maribel García Arenas, Eva M. Ortigosa, Rosa Gil
Iranzo, Natalia Padilla-Zea

XVI Congreso Internacional de Interacción Persona-Ordenador

GAMES, 327
Desarrollo de experiencias lúdicas interactivas geolocalizadas, 328
A. J. Soriano Marín, J. L. González Sánchez, F. L. Gutiérrez Vela
La importancia de las emociones en el diseño de historias interactivas, 337
José Rafael López-Arcos, Patricia Paderewski Rodríguez, F. L. Gutiérrez Vela, Natalia Padilla-Zea,
Noemí Marta Fuentes García
Agente virtual emocional para dispositivos móviles, 341
Sandra Baldassarri, Eva Cerezo
Second Mind: A System for Authoring Behaviors in Virtual Worlds, 345
Manish Mehta, Andrea Corradini
Juegos Serios Tangibles con Objetos Reales como Herramienta de Apoyo para Trabajar con Niños que
Requieren Necesidades Especiales, 351
Iván Durango, Alicia Carrascosa, José A. Gallud, Víctor M. R. Penichet

USABILITY AND USER EXPERIENCE (I), 361

ECUSI: una herramienta que apoya la Evaluación Colaborativa de la Usabilidad de Sistemas
Interactivos, 362
Andrés Solano , Juan Camilo Cerón, César A. Collazos, Habib M. Fardoun, José Luis Arciniegas
Towards an Integration of Usability and Security for User Authentication, 368
Paulo C. Realpe, Cesar A. Collazos, Julio Hurtado, Antoni Granollers
Evaluando la usabilidad de aplicaciones groupware mediante un método dirigido por modelos para el
análisis de la interacción del usuario, 374
Rafael Duque Medina, Alicia Nieto-Reyes
Supporting Users Experience in a 3D eCommerce Environment, 383
D.Contreras, M. Salamó, I. Rodríguez, A. Puig, A. Yañez
La delgada línea roja entre la usabilidad y la experiencia de usuario, 387
Yuliana Puerta Cruz, Cesar A. Collazos, Toni Granollers

USABILITY AND USER EXPERIENCE (II), 391

Effect of Snippets on User Experience in Web Search, 392
Mari-Carmen Marcos, Ferran Gavin, Ioannis Arapakis
Evaluación de la eficiencia de uso de las versiones de escritorio y tableta de una aplicación, 401
Juan P. Moreno, Antonio Peñalver, Federico Botella
Lenguaje Visual de Consulta sobre Grafos de Datos: Un enfoque desde el Diseño Centrado en el
Usuario, 410
María Constanza Pabón, César A. Collazos

SOFTWARE, ARCHITECTURE AND INTERACTION, 419

An Agile Information-Architecture-Driven Approach for the Development of User-Centered Interactive
Software, 420
Luis A. Rojas, José A. Macías
Hacia la Caracterización de la Calidad de Interacción, 429
Cristina Roda, Víctor López-Jaquero, Francisco Montero
Extending MOOC ecosystems using web services and software architectures, 438
Juan Cruz-Benito, Oriol Borrás-Gené, Francisco J. García-Peñalvo, Ángel Fidalgo Blanco, Roberto
Therón
Arquitectura para la interacción en un videojuego para el entrenamiento de la voz de personas con
discapacidad intelectual, 445

XVI Congreso Internacional de Interacción Persona-Ordenador

 Mario Corrales, David Escudero, Valle Flores, Cesar González, Yurena Gutiérrez
Caracterización de las Empresas Desarrolladoras de Software en Panamá en Materia de Usabilidad y
Accesibilidad, 449
Giankaris Moreno, Vanessa Castillo, Kaiser Williams, Nyder Menéndez

ACCESSIBILITY AND SEMANTIC WEB, 453

The Effects of Cross-modal Collaboration on the Stages of Information Seeking, 454
Dena Al-Thani, Tony Stockman, Anastasios Tombros
Modelado de perfiles de usuario accesibles para servicios académicos basados en MOOCs, 463
Francisco Iniesto, Covadonga Rodrigo
Creación de documentos EPUB accesibles por usuarios no técnicos: un largo camino por recorrer, 469
Jordi Roig, Mireia Ribera
Exploring language technologies to provide support to WCAG 2.0 and E2R guidelines, 478
Lourdes Moreno, Paloma Martínez, Isabel Segura-Bedmar, Ricardo Revert
Building a unified repository of interaction patterns, 487
Alfons Palacios, Roberto García, Toni Granollers, Marta Oliva

1º CONCURSO 2015 AIPO DE TFG/TFM, 492

Desarrollo de una herramienta para el diseño y ejecución de actividades enfocadas a ancianos con el
tabletop NIKVision, 493
Clara Bonillo Fernández
Análisis de requerimientos y prototipado de una aplicación accesible para personas ciegas basada en
la API de Google Maps, 496
Rubén Alcaraz Martínez
Factores importantes para un sistema de recomendacion de una red social educativa, 499
Virginia del Castillo Carrero, Isidoro Hernán-Losada, Estefanía Martín-Barroso
Case study on mobile Applications UX: Effect of the usage of a cross-platform development framework,
502
Esteban Angulo

INDEX OF AUTHORS, 503

XVI Congreso Internacional de Interacción Persona-Ordenador

CONFERENCES, TUTORIALS AND PANEL
DISCUSSIONS
 Plenary Conference: Interaction Design for Rehabilitation
P. Markopoulos

Plenary Conference: Interaction Design for

Rehabilitation

Panos Markopoulos

Department of Industrial Design, TU/e Eindhoven University of Technology, Eindhoven. The

Netherlands

[email protected]

ABSTRACT

There is a growing demand for rehabilitation and this necessitates the development of
rehabilitation technology. Further, such technologies can not only improve the quantity of
rehabilitation that can be offered but also its quality. For a long time, this challenge has been
considered as a purely engineering and technical challenge. The prolific growth of such
technologies and the increasing relevance of tele-rehabilitation scenarios, mean have drawn the
attention of industry and academia to the challenge of designing such technologies to motivate
patients, to ensure compliance, good ergonomics and product design to support the correct
execution of training exercises, and of course the potential of generating immense amounts of
data that can help monitor and provide feedback regarding patient progress. This talk presents a
few cases of such technologies and discusses some general challenges such as designing
appropriate feedback, including patients in participatory design processes, acceptance of
technologies, evaluation in the field, and interaction design issues relating to the implementation
of innovations in healthcare.

XVI Congreso Internacional de Interacción Persona-Ordenador | Conferences, Tutorials and

Panel Discussions
| 24 |
 Tutorial: Desarrollo de Aplicaciones de Realidad Aumentada para Tabletas y Smartphones
A. M. Baena

Tutorial: Desarrollo de Aplicaciones de Realidad

Aumentada para Tabletas y Smartphones

Antonio Miguel Baena

AMB Piensa
[email protected]

RESUMEN

Este tutorial se centra en la realidad aumentada y en cómo puede ser utilizada para coeditar y
desarrollar aplicaciones para tabletas y smartphones. El tutorial contempla estudio de casos
como los llevados a cabo recientemente en el Museo Thyssen Bortnemisza y la Central Nuclear
José Cabrera.

XVI Congreso Internacional de Interacción Persona-Ordenador | Conferences, Tutorials and

Panel Discussions
| 25 |
 Conferencia: Kuka – Robots sensitives
M. Garrido

Conferencia: Kuka – Robots sensitivos

Manel Garrido

KUKA Robots Ibérica

RESUMEN

La nueva era en la robótica pasa por transferir a los robots capacidades sensitivas. KUKA ha
conseguido que su robot LBR iiwa sienta y gestione con gran precisión los esfuerzos que se
generan en su entorno de acción. Por lo tanto nos permite automatizar procesos tanto
industriales como fuera de ese entorno, donde la sensibilidad de las personas es imprescindible
para conseguir el objetivo de la acción. Como consecuencia de esa virtud sensitiva, además, este
robot permite trabajar de forma segura en entornos colaborativos.

XVI Congreso Internacional de Interacción Persona-Ordenador | Conferences, Tutorials and

Panel Discussions
| 26 |
 Plenary Conference: Including Gender in the Interaction
E. Rommes

Plenary Conference: Including Gender in the

Interaction

Els Rommes

Institute for Gender Studies, Radboud University Nijmegen

The Netherlands

ABSTRACT

What does it mean to pay attention to gender in the design of computer systems? Should
everything be pink, or all content be about fashion? In this lecture, Rommes will discuss various
notions of what ‘inclusion of gender’ can mean, and which advantages and disadvantages the
various way of ‘including gender’ may have. In addition, she will discuss why it could be
relevant to pay attention gender and how this can be done through the use of various design
methodologies.

XVI Congreso Internacional de Interacción Persona-Ordenador | Conferences, Tutorials and

Panel Discussions
| 27 |
 Plenary Conference: Towards Intracellular Computer-Human Interaction: a micro-electronic perspective
J. A. Plaza

Plenary Conference: Towards Intracellular

Computer-Human Interaction: a micro-electronic
perspective

José Antonio Plaza

Micro and Nano-tools research group

Institute of Microelectronics of Barcelona
IMB-CNM (CSIC)
[email protected]

ABSTRACT

Success of the semiconductor industry has been driven in part by the miniaturization process, as
approximately every three years we see a new generation of memory chips and microprocessors,
in which the size of their fundamental elements, the transistors, is reduced 33%. The sizes of
these transistors are several orders of magnitude smaller in relation typical human body cell
(tens of microns of diameter). The microfabrication techniques of the electronics industry are
being also routinely adapted to fabricate systems which are able to integrate mechanical,
thermal, optical, magnetic, chemical, or even fluidics components in the same silicon chip, in
addition to combining them with electronic components. These silicon chips inside human
living cells could provide endless possibilities, beyond the scope of our imagination. In this talk,
we present our research in the bases of this incipient future field. We believe that the study of
this field will open a new line of research based on Human-Computer Interactions to investigate
the relationship between chips and human living cells, human organs or human actors.

XVI Congreso Internacional de Interacción Persona-Ordenador | Conferences, Tutorials and

Panel Discussions
| 28 |
 Panel de discusión: Reflexiones sobre la Interacción Persona-Ordenador
J. A. Macías, M. Talavera

Panel de discusión: Reflexiones sobre la

Interacción Persona-Ordenador

José Antonio Macías Marina Talavera

Presidente Asociación Interacción Persona- Experience Design Strategist

Ordenador Hewlett Packard Española
Universidad Autónoma de Madrid [email protected]
[email protected]

OBJETIVO

En el siglo XXI, el ordenador de sobremesa está siendo desplazado por tecnología que se mueve
con nosotros. El cambio en la forma de interactuar supone revisar los paradigmas clásicos
existentes y avanzar en una interacción natural entre la persona, la interfaz y el entorno.
Interacción 2015 es un congreso internacional fomentado por la Asociación de Interacción
Persona-Ordenador (AIPO) que tiene como objetivo principal promover y difundir los avances
recientes en el área de la Interacción Persona-Ordenador (IPO), tanto a nivel académico como
empresarial. Este panel de discusión es un punto de encuentro de profesionales y académicos
para reflexionar conjuntamente sobre el presente de la interacción persona-ordenador y
tendencias de futuro. El panel lanza las siguientes preguntas, ¿Hacía donde avanza el diseño de
la interfaz de usuario?, ¿Cómo mejorar la experiencia de usuario?, ¿Cuáles serán los siguientes
paradigmas clásicos?, ¿Qué retos deben afrontarse en la sociedad ante los cambios
tecnológicos? , ¿Cómo potenciar la sinergia entre profesionales y académicos?

XVI Congreso Internacional de Interacción Persona-Ordenador | Conferences, Tutorials and

Panel Discussions
| 29 |
NATURAL USER INTERFACES (I)
 Diseño de actividades de mejora de capacidades cognitivas para tabletops tangibles
C. Bonillo | J. Marco | E. Cerezo | S. Baldassarri

Diseño de actividades de mejora de capacidades

cognitivas para tabletops tangibles
Clara Bonillo Javier Marco Eva Cerezo,
GIGA Affective Lab GIGA Affective Lab Sandra Baldassarri
Dep. Informática e Ingeniería de Dep. Informática e Ingeniería de GIGA Affective Lab
Sistemas, Sistemas, Dep. Informática e Ingeniería de
Universidad de Zaragoza, España. Universidad de Zaragoza, España. Sistemas, Instituto de Investigación en
[email protected] [email protected] Ingeniería de Aragón (I3A)
Universidad de Zaragoza, España.
{ecerezo, sandra}@unizar.es

ABSTRACT Además de las actividades tradicionales de mejora o

El objetivo de este trabajo es explorar la potencialidad del uso de mantenimiento, durante los últimos años se ha empezado a
tabletops tangibles en el desarrollo de actividades terapéuticas de explorar el uso de dispositivos tabletop para trabajar la
mejora o mantenimiento de capacidades cognitivas deterioradas estimulación cognitiva, centrándose principalmente en ancianos
por la edad, enfermedades o tratamientos. Para ello se ha llevado a [2] [7]. Un tabletop es un dispositivo, con un aspecto más o menos
cabo un análisis de las áreas cognitivas y de las actividades más cercano al de una mesa convencional, cuya superficie está
habituales utilizadas por los terapeutas en el trabajo de dichas aumentada mediante la proyección de imagen y sonido procedente
áreas. Ello ha llevado a la definición de un lenguaje de marcas de una aplicación informática y en el que la interacción con dicha
basado en XML destinado a la definición de las actividades, y de aplicación se lleva a cabo mediante movimientos de los dedos en
un reproductor que permite la ejecución de dichas actividades contacto con la superficie de la mesa (multitáctil). Esta forma de
sobre el tabletop tangible NIKVision del grupo GIGA Affective interacción presenta muchas ventajas, ya que la superficie del
Lab de la Universidad de Zaragoza. Nueve actividades han sido tabletop supone un espacio amplio para las capacidades visuales y
desarrolladas con dichas herramientas y evaluadas por una motrices de los usuarios. La estimulación audiovisual les resulta
terapeuta y su grupo de pacientes, pertenecientes a una asociación motivante y se les ofrece un mayor rango de actividades que
local de adultos con enfermedad mental. Los resultados de la pueden abarcar uno o más aspectos de la estimulación cognitiva.
evaluación son muy prometedores y animan a continuar Sin embargo, este tipo de interacción presenta una importante
trabajando en la línea. desventaja y es que la complejidad y precisión de muchos gestos
táctiles puede hacerla dificultosa y frustrante para usuarios con
Categorías and Términos Descriptores problemas de motricidad, como los ancianos o personas con
H.5.2. Interfaces de usuario: estilos de interacción, prototipado. trastornos mentales graves. Por ello, otros modelos de Interacción
D.2.2 Herramientas de diseño y técnicas. Interfaces de Usuario. Natural, como la Interacción Tangible, pueden ofrecer una
alternativa más adaptada a las características de dichos usuarios
Términos Generales [4]. La Interacción Tangible plantea que la interacción entre el
Diseño, Lenguajes. usuario y la aplicación informática sea a través de objetos físicos
de uso cotidiano. Técnicamente, un dispositivo tabletop tangible
Palabras clave es capaz de identificar distintos objetos colocados en su
Interacción tangible, interacción natural, estimulación cognitiva, superficie, así como seguir las distintas manipulaciones que los
mesa interactiva, tabletop, lenguaje de marcas. usuarios realizan con ellos, mostrando información proyectada
sobre la misma superficie donde se manipulan los objetos.
1. INTRODUCCIÓN
El envejecimiento de la población en los últimos años ha supuesto El grupo GIGA Affective Lab cuenta con NIKVision [8], un
una revolución demográfica: se estima que actualmente hay unos tabletop Tangible en el que la interacción se lleva a cabo mediante
600 millones de personas mayores de 60 años y parece que esta objetos colocados sobre la superficie de una mesa. El objetivo de
cifra va a ir aumentando. Este hecho supone un reto para el resto este trabajo ha sido explorar las potencialidades de la Interacción
de la sociedad, puesto que hay que considerar el declive tanto Tangible en un dispositivo tabletop como NIKVision en la
físico como cognitivo que se produce a medida que una persona estimulación cognitiva.
envejece. Por otra parte, se estima que el 14% de la carga global El artículo está organizado de la forma siguiente: en la sección 2
de las enfermedades a nivel mundial puede ser atribuida a se realiza un estado del arte en el que se estudian tabletops
trastornos neuropsiquiátricos [10], los cuáles también producen un destinados a la estimulación cognitiva; en la sección 3 se realiza
declive en las capacidades cognitivas ya sea por la propia un estudio de las áreas cognitivas que más se suelen trabajar y del
enfermedad o por efectos secundarios de la medicación que dichas tipo de actividades más frecuentes en cada una de ellas; así
enfermedades conllevan. Una forma de ayudar a estas personas a mismo, se presentan las herramientas generadas para la definición
mejorar dichas capacidades, o por lo menos evitar su deterioro, es y ejecución de actividades: el lenguaje de marcas y el reproductor.
a través de actividades que potencien el uso de dichas capacidades En la sección 4 se presentan las actividades desarrolladas y en la
[9].

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 31 |
 Diseño de actividades de mejora de capacidades cognitivas para tabletops tangibles
C. Bonillo | J. Marco | E. Cerezo | S. Baldassarri
sección 5 su evaluación con usuarios. Finalmente en la sección 6 3. ESTUDIO DE ACTIVIDADES
se presentan conclusiones y trabajo futuro.
COGNITIVAS
2. ESTADO DEL ARTE Nuestro objetivo es llevar a cabo una selección de actividades que
A continuación se va a hacer un repaso del estado del arte en cubran el máximo número de áreas cognitivas y que además
cuanto a experiencias de trabajo de capacidades cognitivas aprovechen al máximo el uso del tabletop y de la Interacción
haciendo énfasis en aquellas que hacen uso de tabletops. Tangible.

En Kown et al. 2013 [5] se presenta el sistema E-CoRe para la 3.1 Áreas cognitivas
mejora de la capacidad cognitiva a través de un dispositivo Antes de realizar la selección de actividades se explicará en qué
tabletop. La una aplicación llamada “Making Cookies” utiliza consiste cada una de las áreas cognitivas que suelen trabajarse [9],
distintos objetos para simular que se hacen galletas y su objetivo junto con ejemplos de actividades que se suelen desarrollar para
es mejorar tres áreas cognitivas: la memoria, el razonamiento y la potenciar dichas áreas.
atención.
• Memoria: es una de las funciones cerebrales cuyo
En Gamberini et al. 2006 [2] se ha desarrollado un tabletop con deterioro es más patente con la edad. Un ejemplo típico
una aplicación similar al juego de encontrar parejas. La de juego de memoria es el juego de encontrar las
interacción se lleva a cabo con unos lápices especiales en lugar de parejas.
con el dedo a través de diversos minijuegos, en los que se busca
mejorar funciones cognitivas como la memoria, el razonamiento, • Atención: dentro de la atención podemos distinguir a su
la atención selectiva y dividida, y la clasificación. vez tres subtipos [1]: selectiva (aquella que se basa en
reaccionar ante estímulos concretos de entre todos los
En Leitnet et al. 2007 [6] se ha desarrollado un tabletop Tangible que se presentan), dividida (aquella en la que hay que
para la rehabilitación física y cognitiva. En él se plantea trabajar centrarse en varios estímulos a la vez, perteneciendo
con distintas actividades tipo puzle manipulando cubos con generalmente estos estímulos a sentidos diferentes como
distintos patrones dibujados sobre ellos, siendo el objetivo de la por ejemplo la vista y el oído), y sostenida (aquella que
tarea alinearlos para que coincidan con el patrón que se muestra requiere concentración continua). Un ejemplo de
en la mesa. actividad de atención es el juego de encontrar las
Por último, Sociable [11] es un proyecto europeo para el diferencias o los puzles, que se utilizan para trabajar la
entrenamiento cognitivo de personas mayores. Para ello se hace atención sostenida.
uso de un tabletop táctil para ejecutar actividades organizadas en • Cálculo: esta área cognitiva se refiere a la capacidad de
diversas categorías: memoria, atención, razonamiento, lenguaje y una persona de realizar cálculos matemáticos sin
orientación. instrumentos adicionales. Ejemplos de actividades de
Del estudio de los ejemplos anteriores se puede concluir que tanto este tipo se pueden encontrar en juegos de mesa como el
el tabletop como las actividades desarrolladas para éste acercan a Parchís (hay que ir contando los puntos cuando te
las personas con dificultades cognitivas a las nuevas tecnologías y comes una ficha), o el Dominó (se tienen que estar
les permiten seguir trabajando sus capacidades para evitar (o por contando constantemente los puntos que se llevan).
lo menos disminuir) su deterioro. Sin embargo, no se está • Lenguaje: capacidad de los seres humanos para
aprovechando todo lo que se podría las ventajas que ofrecen la comunicarse por medio de signos. Actividades que
combinación de la Interacción Tangible con los tabletop. potencian esta área pueden ser emparejar sinónimos y
Por ejemplo, a pesar de que Gamberini et al. 2006 [2] trabaja antónimos, u ordenar frases desordenadas.
diferentes áreas cognitivas, los lápices que se utilizan • Orientación espacio-temporal: es la toma de
simplemente como sustitutos de los dedos, por lo tanto no se conciencia de los movimientos en el espacio y el tiempo
tendría que considerar como Interacción Tangible. de forma coordinada. Se relaciona con el conocimiento
Con los cubos de Leiner et al. 2007 [6], o los objetos para hacer que tiene cada persona del tiempo en el que está (fecha
galletas de Kown et al. 2013 [5], sí que se está utilizando la y hora) y del lugar en el que está. Para trabajar la
Interacción Tangible. Sin embargo, ambos se han centrado en el orientación se suelen utilizar preguntas del tipo: ¿qué
desarrollo de una única actividad con juguetes ya predefinidos, hora es?, ¿en qué momento del día estamos?, ¿dónde
por lo que es muy limitada en los aspectos cognitivos cubiertos. estamos?

Por último, en el proyecto Sociable [11] se menciona un tabletop • Razonamiento: facultad que permite resolver
con una gran variedad de actividades cognitivas pero todas ellas problemas, extraer conclusiones y aprender de manera
están basadas en la interacción táctil y por lo tanto no aprovecha consciente de los hechos, estableciendo conexiones
las ventajas que tendrían esas actividades usando Interacción causales y lógicas necesarias entre ellos. Un ejemplo
Tangible. Basándonos en la variedad de actividades de proyecto claro de juego de razonamiento es el Sudoku.
Sociable el presente trabajo pretende explorar las potencialidades
Una vez detectadas la áreas y actividades más habituales, se llevó
que ofrecen los dispositivos tabletop y la Interacción Tangible en
a cabo un análisis de requisitos que se explican en la siguiente
el tratamiento de problemas cognitivos.
sección.
A continuación se presenta el estudio que se ha realizado de las
distintas actividades y juegos que se utilizan habitualmente para el 3.2 Análisis de requisitos
tratamiento de problemas cognitivos. Se llevó a cabo un análisis exhaustivo de las diferentes
actividades cognitivas expuestas en el anterior apartado, con
XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)
| 32 |
 Diseño de actividades de mejora de capacidades cognitivas para tabletops tangibles
C. Bonillo | J. Marco | E. Cerezo | S. Baldassarri
objeto de extraer los elementos comunes a todas ellas y poder determinado, durante el cual el jugador debe memorizarlo (R4 de
elaborar un conjunto de requisitos (Tabla 1) a cumplir por una la Tabla 1), por ejemplo, en el juego Simon™.
aplicación informática capaz de ejecutar cualquiera de estas
actividades en un dispositivo tabletop. A menudo, las actividades y juegos cognitivos se juegan como
una secuencia de tareas (R7 de la Tabla 1) con un nivel de
Tabla 1. Requisitos de la herramienta dificultad variable, o gradual. Conforme el jugador completa
tareas fáciles, se vuelve a plantear la misma actividad con un
Requisito Actividad grado de dificultad mayor. Por ejemplo, de nuevo el juego
R Se han de tener áreas interactivas en las Simon™, se juega como una sucesión de tareas en el que la
Todas
1 que poder poner uno o más objetos. secuencia a memorizar va aumentando gradualmente.
R En algunas actividades se ha de tener en
Una vez recopilados los requisitos comunes a las actividades
2 cuenta la orientación de los objetos Atención cognitivas y juegos de mesa, se ha creado una herramienta
colocados sobre el área informática que permite definir y ejecutar actividades basadas en
R Las áreas tienen una lista de objetos Interacción Tangible en un dispositivo tabletop. Para ello, primero
3 correctos y una lista de objetos Todas ha sido necesario crear un lenguaje de definición de actividades y
incorrectos juegos que cubra todos los requisitos anteriormente expuestos
R La imagen de fondo mostrada en pantalla usando una sintaxis entendible por una aplicación informática.
4 podrá cambiar pasado un cierto tiempo Memoria
que será configurable
R Las áreas interactivas pueden estar fijas
5 en una posición en pantalla o ir asociadas Todas
a objetos
R Se tiene que poder ofrecer algún tipo de
6 realimentación cuando se coloquen
Todas
objetos en las áreas dependiendo de si
están bien o mal
R Se tiene que poder definir tareas dentro Figura 1: Diferentes juegos de mesa. Izquierda: tablero de
7 de una misma actividad de modo que Parchís. Derecha: Dominó, un juego de mesa sin tablero
Todas
cuando se complete una tarea de la
misma, se pase a la siguiente 3.3 Lenguaje de modelado de actividades
R Las actividades pueden requerir objetos cognitivas
8 concretos diferenciados o fichas del Todas Partiendo del análisis de requisitos previo se ha extraído un
mismo tipo
vocabulario o conjunto de términos que siempre aparecen en los
juegos y actividades cognitivas. Un juego de tabletop es modelado
En el análisis se vio que en realidad los elementos comunes eran como una secuencia de tareas (ver fig. 2). Cada tarea es un
coincidentes con los de otros juegos puramente lúdicos que se objetivo que el jugador ha de alcanzar para poder avanzar en el
pueden jugar sobre la superficie de una mesa. Así, un juego típico juego. Cada tarea está compuesta por:
de tablero está compuesto por un conjunto de áreas fijas en la
superficie (R1 de la Tabla 1), un conjunto de piezas de juego (R8 • Un fondo: el tablero.
de la Tabla 1) y un conjunto de reglas que definen el significado
• Varias áreas: zonas del tablero o de una pieza de juego
de colocar piezas en cada área (R3 de la Tabla 1) y su efecto
en las que posicionar unas determinadas piezas de juego
asociado (R6 de la Tabla 1). Además de la colocación de la ficha
tiene un significado.
dentro del área, la orientación puede también tener un significado
concreto (R2 de la Tabla 1), por ejemplo, en el juego del • Varias piezas de juego: objetos utilizados en el juego.
Tangram. Juegos de tablero clásicos como el parchís (ver fig. 1 Se colocan en las áreas; fuera de ellas la pieza no tiene
izquierda) pueden describirse como un tablero dividido en varias ningún significado para el juego.
celdas en las que se colocan las piezas de juego siguiendo unas
determinadas reglas. • Retroalimentación: elementos gráficos o sonoros que
muestran las consecuencias de las acciones del jugador.
Por otro lado, hay muchos juegos populares de mesa que no usan
tablero y por tanto no tienen áreas predefinidas en la superficie de Siguiendo el R3 de la Tabla 1, cada área lleva asociada una lista
la mesa; p.e. Dominó (ver fig. 1 derecha) o juegos de cartas. En de piezas de juego correctas e incorrectas o, por defecto, todas las
este tipo de juegos cada pieza de juego se relaciona con las otras piezas no correctas son consideradas incorrectas. Cuando una
piezas por “proximidad”, la orientación puede tener un significado pieza de juego se sitúa dentro de un área, esta acción es
(R2 de la Tabla 1) y las reglas de juego se aplican cuando un pieza interpretada en el juego como correcta o incorrecta dependiendo
de juego está al lado de otra; en otras palabras, cada pieza juego de la lista en la que esté contenida la pieza de juego. El
tiene un área asociada a ella (R5 de la tabla 1), y cuando otras componente de retroalimentación informará al jugador de si la
piezas entran en dicha área se produce un efecto en el juego (R6 acción es correcta o incorrecta a través de una imagen, una
de la Tabla 1). animación y/o un sonido. Para completar una tarea del juego,
todas las áreas definidas en esa tarea han de contener todas las
En los juegos de memoria, es necesario mostrar inicialmente al piezas de juego correctas (y sólo las correctas). En ese caso, el
jugador una información que se oculta tras un tiempo componente de retroalimentación informará también de que la
tarea está completa y el juego avanzará a la siguiente tarea.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 33 |
 Diseño de actividades de mejora de capacidades cognitivas para tabletops tangibles
C. Bonillo | J. Marco | E. Cerezo | S. Baldassarri
Cuando se ejecuta, el intérprete lee todos los ficheros XML
almacenados en una carpeta específica del sistema informático del
tabletop. Cada fichero XML contiene un juego modelado con la
sintaxis detallada en la sección anterior. El reproductor muestra en
la superficie del tabletop una pantalla de menú con todos los
juegos de la carpeta (ver fig. 4 izquierda). De esta manera, el
jugador puede ejecutar un juego tocando su icono correspondiente
Cuando un juego ha sido elegido, el intérprete carga e interpreta el
fichero XML, y recupera todos los gráficos y ficheros de audio
requeridos para renderizar el juego sobre el tabletop. El intérprete
es capaz de cargar los formatos de imagen (BMP, GIF, JPG,
PNG…), y de audio (WAV, MP3…) más comunes. En caso de los
recursos de animación, el intérprete acepta ficheros SWF.
Finalmente, el tablero se muestra en el tabletop y el jugador puede
jugar situando las diferentes piezas de juego sobre la superficie
Figura 2: Jerarquía de los elementos
(ver fig. 4 derecha).
La estructura jerárquica de los diferentes elementos hace que la
forma más adecuada de modelar las actividades y juegos sea
mediante un lenguaje de marcas como el XML, ya que este tipo
de lenguajes son muy adecuados para representar jerarquías. La
figura 3 muestra la especificación en sintaxis XML del lenguaje
de modelado.
<game>
<task>
<background>
<color rgb="0xrrggbb"/>
<image path="path/filename"/>
<audio path=”path/filename”/> Figura 4: Intérprete del lenguaje de marcas. Izquierda:
</background>
Pantalla de menú del intérprete. Derecha: Intérprete
<area[associated_fid=”id”][orient=”yes/no”]>
<pos x="n" y="n" width="n" height="n"/> ejecutando un juego
<image path="path/filename"/>
<imageCorrect path="path/filename"/> 4. ACTIVIDADES DESARROLLADAS
<imageWrong path="path/filename"/> Se decidió comenzar con actividades de memoria, atención y
<fid id="1,2,…" correct="yes/no" razonamiento, en las que las potencialidades del uso de objetos
sound="path/filename" orient="n"/> eran más palpables. A continuación se detallan las actividades
<fid id="*” correct="yes/no" agrupadas por área de trabajo cognitivo.
sound="path/filename" orient="n"/>
</area>
…
4.1 Actividades de memoria
<feedback> Como actividades de memoria se desarrollaron las actividades
<pos x="n" y="n" width="n" height="n"/> “Lista de la compra”, “Viajes” y “Kraken”.
<imageCompleted path="path/filename"/>
<soundCompleted path="path/filename"/> En “Lista de la compra” se muestra en pantalla una imagen con
<imageCorrect path="path/filename"/> una lista de la compra con alimentos durante cinco segundos.
<soundCorrect path="path/filename"/> Pasado ese tiempo, se quita la imagen de la lista poniéndose una
<imageWrong path="path/filename"/> imagen con una bolsa de la compra y el usuario ha de seleccionar
<soundWrong path="path/filename"/> los alimentos correctos y situarlos en la bolsa. Los objetos que se
</feedback> utilizan en esta actividad son doce juguetes de comidas y bebidas:
</task>
agua, cebolla, guisantes, huevo, leche, manzana, naranja,
…
</game> pimiento, tomate, zanahoria, zumo de naranja y zumo de uva (ver
fig 5 arriba). La retroalimentación de esta actividad consiste en
Figura 3: Sintaxis XML del lenguaje de modelado una carita que se pone sonriente cuando se coloca un alimento
Una vez definido el lenguaje de marcas, se ha implementado una correcto, y triste si el alimento colocado no estaba en la lista.
aplicación informática capaz de interpretar ficheros XML que En la actividad “Viajes”, al usuario se le muestra un mapa de
siguen la sintaxis previamente expuesta para modelar los juegos, y Europa y se reproduce una grabación con un itinerario a seguir
ejecutarlos en un dispositivo tabletop. con distintos medios de transporte. El usuario ha de recordar el
itinerario y situar los transportes correspondientes en los países
3.4 Intérprete del lenguaje de modelado de correctos. Los objetos que se utilizan en esta actividad son tres
actividades cognitivas juguetes de medios de transporte: tren, avión y barco (ver fig 5
El intérprete es la aplicación encargada de cargar y ejecutar los abajo-izquierda). La retroalimentación de esta actividad consiste
juegos en el dispositivo tabletop. en una carita que se pone sonriente cuando se pone el medio de
tranporte en su correspondiente país, o triste en caso contrario.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 34 |
 Diseño de actividades de mejora de capacidades cognitivas para tabletops tangibles
C. Bonillo | J. Marco | E. Cerezo | S. Baldassarri
En la actividad “Kraken”, al usuario se le muestra una imagen de roja encima del número dependiendo de si este es correcto o
un mar dividido en casillas. Durante unos segundos, el usuario incorrecto.
puede ver en qué casillas están varios monstruos justo antes de
que la imagen inicial se oculte y sólo queden las casillas En “Marca los símbolos” se presenta un mapa con símbolos de
mostrando mar. El objetivo del juego es que el usuario (que gasolineras, restaurantes, hoteles, farmacias...y se pide al usuario
empieza en la casilla de la esquina inferior derecha) llegue a la que marque todos los que sean de un tipo concreto. Los objetos
isla del tesoro (situada en la esquina superior izquierda) evitando que se utilizan en esta actividad son fichas que el usuario tendrá
las casillas en las que había monstruos. El objeto que se utiliza en que colocar en los símbolos correspondientes (ver fig 6 abajo-
esta actividad es un juguete de un barco (ver fig 5 abajo-derecha). izquierda). La retroalimentación de esta actividad consiste en la
La retroalimentación de esta actividad consiste en la animación aparición de un tick verde o de una cruz roja encima del símbolo
del monstruo que aparece cuando el jugador pisa una casilla dependiendo de si este es correcto o incorrecto.
incorrecta. La actividad “Tangram” tiene dos modos: el fácil, en el que se ve
en qué posición está cada una las piezas que conforman la figura,
y el difícil, en el que solo se muestra el contorno de la figura. En
la modalidad difícil también se trabaja el razonamiento, al tener
que deducir dónde va cada pieza. Los objetos con los que
trabajamos en esta actividad son las siete piezas del Tangram (ver
fig 6 abajo-derecha). La retroalimentación de esta actividad
consiste en una carita que se pone sonriente cuando una pieza se
coloca de forma correcta, o triste si se coloca una pieza del puzle
en un lugar que no le corresponde.

Figura 5: Actividades de memoria. Arriba: Lista de la compra

Abajo-Izquierda: Viajes. Abajo-Derecha: Kraken

4.2 Actividades de atención

Como actividades de atención, se desarrollaron las siguientes:
“¿Cuántos hay?” (atención dividida), “Marca los símbolos”
(atención selectiva), y “Tangram” (atención sostenida).
En “¿Cuántos hay?” al usuario se le muestra una imagen con
números entre el 0 y el 9. Nada más empezar la actividad suena
una grabación que le dice al usuario qué número ha de buscar. El
usuario tendrá que ir situando fichas sobre todos los números que
encuentre de ese tipo, mientras que a la vez se estará
reproduciendo una grabación con una secuencia de golpes: por
ejemplo, sonarán cuatro golpes, habrá una pausa, sonarán ocho Figura 6: Actividades de atención. Arriba: ¿Cuántos hay?
golpes, habrá una pausa... de modo que el usuario, a la vez que Abajo-Izquierda: Marca los símbolos. Abajo-Derecha:
marca los números, durante la pausa tendrá que decir cuántos Tangram
golpes han sonado. Los objetos que se utilizan en esta actividad
son fichas que el usuario tendrá que colocar en los números 4.3 Actividades de razonamiento
correspondientes (ver fig 6 arriba). La retroalimentación de esta Como actividades de memoria, se desarrollaron las actividades
actividad consiste en la aparición de un tick verde o de una cruz “Relaciones sintagmáticas”, “Analogías”, y “Completa la
secuencia”.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 35 |
 Diseño de actividades de mejora de capacidades cognitivas para tabletops tangibles
C. Bonillo | J. Marco | E. Cerezo | S. Baldassarri
En “Relaciones sintagmáticas”, el usuario ha de seguir unas
instrucciones que guardan relaciones entre sí para rellenar una
composición de figuras geométricas con las figuras de la forma y
color adecuados. Esta actividad tiene dos modos: el fácil, con una
composición de cuatro figuras geométricas (dos cuadrados y dos
círculos) y el difícil, con una composición de seis figuras
geométricas (dos cuadrados, dos círculos y dos triángulos). Los
objetos que se utilizan son cuadrados, círculos y triángulos de
colores rojo, azul, verde o amarillo (ver fig 7 arriba). La
retroalimentación de esta actividad consiste en una carita que se
pone sonriente o triste dependiendo de si la acción del jugador es
correcta o incorrecta.
En “Analogías” se dispone de un conjunto de fichas con dos
imágenes cada una. A cada imagen le corresponde una pareja, que
estará en otra ficha diferente. El objetivo de la actividad es que
todas las fichas formen una cadena de modo que todos los
extremos de las fichas estén con su correspondiente pareja. Los
objetos que se utilizan en esta actividad son las diferentes fichas
(ver fig 7 abajo-izquierda). La retroalimentación de esta actividad
es únicamente sonora, diferenciando cuando dos fichas están bien
unidas y cuando no.
En “Completa la secuencia” se usan fichas de dominó para
mostrar una secuencia en la que faltan algunas piezas. El usuario
ha de seleccionar la pieza que falta para completar la secuencia.
Los objetos que se utilizan en esta actividad son las fichas de
dominó (ver fig 7 abajo-derecha). La retroalimentación de esta
actividad consiste en una carita que se pone sonriente cuando se
coloca una pieza en su lugar correcto, o triste en caso contrario.

Figura 7: Actividades de razonamiento. Arriba: Relaciones

sintagmáticas. Abajo-Izquierda: Analogías. Abajo-Derecha:
Completa la secuencia

5. EVALUACIÓN DE LAS ACTIVIDADES

Nuestro objetivo al plantear la evaluación fue responder a la
pregunta: ¿Cómo perciben los pacientes las nuevas actividades en
un tabletop Tangible?
La evaluación se llevó a cabo como resultado de la colaboración
con ASAPME, una asociación local que trabaja con adultos con
enfermedades mentales. La terapeuta de esta asociación usa
juegos tradicionales (sin ordenador) con sus pacientes para
trabajar y mejorar sus habilidades cognitivas, que pueden estar
afectadas como resultado de su enfermedad o como efecto
secundario de su medicación. Las áreas cognitivas que trabajan
son: atención, concentración, memoria, percepción, lenguaje,
cálculo, razonamiento y pensamiento lógico, y motivación entre
otras. El grupo de terapia estaba compuesto por cuatro adultos de
edades entre los 25 y 65 años con diferentes enfermedades
mentales: esquizofrenia, esquizofrenia y retraso mental, síndrome
frontal, psicosis maniaco depresiva, trastorno mental orgánico con
descontrol de impulsos y retraso mental, trastorno de ansiedad y
trastornos físicos.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 36 |
 Diseño de actividades de mejora de capacidades cognitivas para tabletops tangibles
C. Bonillo | J. Marco | E. Cerezo | S. Baldassarri

5.1 Metodología Además, en relación con el cuestionario AttrakDiff, la terapeuta

Después de enseñarle a la terapeuta las actividades expuestas en la nos dijo que este tipo de cuestionario resultaría difícil de rellenar
sección 4, ella mostró gran interés en aplicarlas con sus pacientes para los pacientes, así que se realizó una simplificación del
en sus sesiones de trabajo. Además, la actividad de razonamiento mismo: en primer lugar se eliminó la pregunta “predecible –
“Relaciones sintagmáticas” es una actividad que la terapeuta impredecible” por considerar la terapeuta que sus pacientes no
aplica habitualmente con sus pacientes sin uso de ordenador (ver iban a comprender a qué se refería este par de adjetivos; en
fig 8). segundo lugar, en vez de tener que puntuar utilizando dos
adjetivos antónimos como indica el cuestionario AttrakDiff
Se planearon dos sesiones de evaluación. Durante la primera original, la terapeuta eligió de entre los dos adjetivos
sesión, los pacientes probaron las actividades desarrolladas contrapuestos el más comprensible (por ejemplo, entre el par de
exceptuando “Relaciones Sintagmáticas”. Con esta evaluación se adjetivos “aislante” e “integrador” se eligió el segundo) de modo
quería valorar la experiencia del usuario a la hora de probar las que el paciente sólo tuviera que calificar dicho adjetivo entre 1 y
actividades en la mesa. Para ello se utilizó el cuestionario 7, significando el 1 “totalmente en descuerdo” y el 7 “totalmente
AttrakDiff [3], el cual es ampliamente usado para evaluar la de acuerdo” (ver fig. 9).
percepción de la usabilidad que los usuarios tienen al utilizar un
dispositivo o aplicación. El cuestionario AttralDiff considera tanto
dimensiones pragmáticas como hedónicas de usabilidad.
Durante la segunda sesión de evaluación, los pacientes probaron
la actividad “Relaciones Sintagmáticas”. El objetivo de esta
sesión era comparar la preferencia de los pacientes entre la
actividad con tabletop y sin tabletop.
Durante ambas sesiones estaban presentes la terapeuta, que
ayudaba a los pacientes mientras estos probaban las actividades, y
dos evaluadores, que eran los encargados de tomar nota de los
comentarios de los participantes. Ambas sesiones duraron dos
horas cada una y se llevaron a cabo en la sala donde los pacientes
realizaban normalmente sus actividades. Fue en esa misma aula
donde se instaló un tabletop NIKVision para que los pacientes
probaran las actividades. Figura 9: Cuestionario AttrakDiff adaptado que rellenaron
los pacientes
Durante esta sesión los participantes probaron por turnos algunas
de las actividades de memoria, atención y razonamiento
desarrolladas en el tabletop. Uno de los pacientes se negó a jugar
con el tabletop ya que comentó que no le gustaba el dispositivo.
Los demás completaron las actividades sin problemas aunque los
tiempos de completitud de las actividades variaron dependiendo
de la enfermedad mental del paciente, en un rango de quince
minutos y media hora.
Finalmente, los usuarios rellenaron su cuestionario AttrackDiff.
Uno de los pacientes necesitó para ello la ayuda de la terapeuta
porque no acababa de entender lo que había que hacer.

5.3 Sesión 2: Comparativa de la actividad con

tabletop y sin tabletop
Durante esta sesión, todos los participantes trabajaban en papel
con la versión no tecnológica de la actividad (ver fig. 8) y por
turnos también jugaban con la versión de tabletop.
El mismo paciente que se había negado a jugar con el tabletop
durante la sesión anterior tampoco quiso en esta sesión probar la
versión de tabletop de la actividad, pero el resto de pacientes
completaron el juego sin problemas, aunque necesitaron diferente
tiempo: los pacientes más afectados cognitivamente requirieron
Figura 8: Versión no tecnológica de la actividad “Relaciones media hora mientras que los menos afectados completaron el
sintagmáticas” juego en diez minutos.

5.2 Sesión 1: Uso de actividades por los Al finalizar todos los usuarios sus actividades de “Relaciones
Sintagmáticas”, se realizó una discusión en común de la opinión y
pacientes percepción que tienen sobre cada una de las versiones. Todos
Previamente a la sesión, la terapeuta decidió qué actividades iba a (menos el que no jugó), participaron en dicha discusión, durante
realizar cada uno de los pacientes, así como el grado de dificultad. 15 minutos.
XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)
| 37 |
 Diseño de actividades de mejora de capacidades cognitivas para tabletops tangibles
C. Bonillo | J. Marco | E. Cerezo | S. Baldassarri

5.4 Resultados Respecto a las notas que se tomaron, pudimos extraer algunas
En cuanto a los resultados de la primera sesión, en el cuestionario conclusiones para mejorar las actividades, como se comenta a
AttrakDiff cada adjetivo se engloba dentro de una de las cuatro continuación:
dimensiones siguientes: Casi todos los participantes encontraron las últimas tareas de la
• Cualidades pragmáticas (PQ): mide el producto como actividad de razonamiento “Completa la secuencia” demasiado
herramienta utilizada para realizar una tarea. difíciles, lo cual indica que es necesario bajar la dificultad de
dicha actividad.
• Cualidades Hedónicas-Estimulantes (HQ-S): mide en
También nos dimos cuenta de que en la actividad “Lista de la
qué medida el producto apoya la estimulación.
compra” el dibujo de la bolsa que aparecía en el área destinada a
• Cualidades Hedónicas-Identificación (HQ-I): mide en poner los alimentos podría no ser el más adecuado, ya que uno de
qué medida el producto permite al usuario identificarse los pacientes interpretó que al ser una bolsa, los alimentos tenían
con él. que meterse por la parte de arriba y por lo tanto concentraba todos
los alimentos únicamente en la parte del arriba del área en vez de
• Escala de atractivo global (ATT): mide el valor global en toda la zona de la bolsa. En esa misma actividad también
del producto basado en la percepción de sus cualidades descubrimos que el tiempo que se muestra la lista con los
hedónicas y pragmáticas. alimentos al principio de la actividad era demasiado corto para
A pesar de haber hecho una simplificación del cuestionario, uno de los pacientes.
analizando las respuestas dadas por los pacientes se han Por último, observamos que la posición de la retroalimentación
observado contradicciones a la hora de responder, puesto que un visual en determinadas actividades tenía que mejorarse, ya que en
paciente calificó el adjetivo “Integrador” con un 0 pero después actividades como la de “Viajes” o “Tangram” la retroalimentación
calificó “Me acerca a la gente” con un 5, otro paciente calificó (carita que se pone triste o contenta) está situada demasiado
“Ingenioso” con un 4 y “Soso” con un 6, y otro paciente calificó alejada de las áreas interactivas, lo que provoca que muchas veces
“Engorroso” con un 6 y “Claro” con otro 6. Por lo tanto, se puede el paciente no la perciba. En otras actividades, como “Busca los
llegar a la conclusión de que para este tipo de pacientes sería símbolos”, la retroalimentación sí que era adecuada ya que ésta
recomendable cambiar algunos de los adjetivos por otros más aparecía directamente sobre el área en la que jugaba el paciente y,
sencillos para que les resulten más fáciles de comprender y los por tanto, éste siempre lo veía.
resultados sean más fiables. A la hora de calcular los resultados,
los adjetivos contradictorios previamente mencionados no se En cuanto a los resultados de la segunda sesión, el grupo discutió
tuvieron en cuenta en ningún caso. durante 15 minutos con los evaluadores su valoración del juego de
tabletop. La discusión empezó preguntándoles qué actividad les
En la figura 10 se pueden ver las medias de los resultados de los gustaba más: la no tecnológica o la versión del tabletop. La
tres usuarios que participaron en la sesión agrupados en las cuatro actividad de tabletop fue, en general, bien recibida por los
dimensiones. Aunque todas las notas superan el 50%, se aprecia pacientes que jugaron en el dispositivo tabletop, y todos ellos
que la nota más baja es la pragmática (PQ), indicando que los eligieron la versión de tabletop de la actividad en vez de la de
pacientes no consideran que la herramienta destaque por su papel. Sin embargo, también expresaron algunas matizaciones
utilidad en la realización de actividades. Sin embargo, las notas de personales. Uno de los participantes apuntó que, aunque era más
las dos dimensiones hedónicas (HQ-S y HQ-I) son mayores, divertido hacer la actividad en el tabletop, prefería leer las
indicando que los usuarios destacan más del tabletop su capacidad instrucciones en papel en vez de en el tabletop ya que mirar a la
motivadora y atrayente. Por último, se puede ver que la dimensión pantalla de tabletop por mucho tiempo le cansaba la vista. Otro
con mejores resultados es la última (ATT), con un valor de casi el paciente dijo inicialmente que prefería hacer la actividad en papel
75%, lo cual indica que el valor global del producto es bastante porque cuando jugaba con la versión del tabletop, a veces le daba
satisfactorio. Por tanto, se puede extraer como conclusión que los retroalimentación errónea diciendo que la pieza de juego estaba
usuarios están satisfechos con el uso de las actividades sobre el mal puesta e inmediatamente después decía que estaba bien (este
tabletop, ya que aunque no consideran que éstas les aporten suceso era debido a un problema técnico que fue posteriormente
mucho más que el hacer sus actividades tradicionales, sí que se solucionado). Cuando se le preguntó al paciente qué elegiría en
sienten atraídos y motivados a la hora de realizarlas. caso de que el problema estuviera solucionado, dijo que elegiría la
actividad de tabletop en vez de la de papel porque era más
divertida y porque era más fácil saber si habías elegido la figura
correcta gracias al sonido, mientras que en la de papel tenía que
preguntar a la terapeuta además de borrar y colorear de nuevo
cuando fallaba. Finalmente, otro paciente nos dijo que prefería la
actividad de tabletop pero que no le gustaban los sonidos
incorrectos: él prefería que un sonido sólo se reprodujera cuando
hacía algo bien y que nada sonara cuando hacía algo mal. La
terapeuta nos dijo después que algunos pacientes se motivan con
retroalimentación positiva pero que se frustran fácilmente con la
retroalimentación negativa; por lo tanto, es un aspecto importante
a considerar en el futuro a la hora de diseñar actividades de
tabletop personalizadas según las preferencias de los pacientes.

Figura 10: Resultados del cuestionario AttrakDiff adaptado

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 38 |
 Diseño de actividades de mejora de capacidades cognitivas para tabletops tangibles
C. Bonillo | J. Marco | E. Cerezo | S. Baldassarri

6. CONCLUSIONES Y TRABAJO FUTURO [8] Marco, J., Cerezo, E., Baldassarri, S., Mazzonne, E., Read, J.
Tras analizar las áreas cognitivas más tratadas por los terapeutas 2009. Bringing Tabletop Technologies to Kindergarten
en pacientes mayores o con enfermedad mental se han Children. 23rd BCS Conference on Human computer
desarrollado las herramientas necesarias para el diseño y Interaction. Cambridge University.
ejecución de actividades de ese tipo en tabletops Tangibles [9] Muñoz-Marrón, E., Blázquez-Alisente, J.L., Galparsoro-
basados en visión como NIKVision: un lenguaje de marcas Izagirre, N., González-Rodríguez, B., Lubrini G, Periáñez-
flexible basado en XML y un intérprete de dicho lenguaje. Morales J.A., et al. 2009. Estimulación cognitiva y
Ambas han sido utilizadas en el desarrollo de nueve actividades rehabilitación neuropsicológica. Barcelona: UOC.
que han sido probadas por una terapeuta y su grupo de pacientes
[10] Rocha, K.B., Pérez, K., Rodríguez-Sanz, M., Borrell, C., y
adultos con enfermedad mental. La evaluación ha servido para
Obiols, E.J. 2010. Prevalencia de problemas de salud mental
reafirmar las potencialidades de este tipo de actividades
y su asociación con variable socieconómicas, de trabajo y
desarrolladas sobre un tabletop Tangible y también para encontrar
salud: Resultados de la Encuesta Nacional de Salud de
algunos problemas de usabilidad y de la propia metodología de España. Psicothema, 22, pp. 389-395.
evaluación con ese tipo de pacientes. Así, se ha detectado la
importancia de diseñar una correcta y adaptada retroalimentación [11] Sociable http://www.cognitivetraining.eu/?q=services-0
de las actividades, así como la necesidad de una mayor adaptación
de los cuestionarios de evaluación con objeto de obtener
resultados más fiables de los mismos.
Como trabajo futuro, se está trabajando en el desarrollo de una
interfaz gráfica que permita el desarrollo de actividades por parte
de los propios terapeutas.

7. AGRADECIMIENTOS
Agradecemos a la asociación ASAPME Huesca su participación
en las sesiones de evaluación. Este trabajo ha sido parcialmente
financiado por el Gobierno de España a través del contrato
DGICYT TIN2011-24660.

8. REFERENCIAS
[1] Drake, M.A. 2007. Evaluación de la atención. In Burin, D.I.,
Drake, M.A., and Harris, P. Evaluación neuropsicológica en
adultos. Buenos Aires, Barcelona, México: Paidós.
[2] Gamberini, L., Alcaniz, M., Barresi, G., Fabregat, M.,
Ibanez, F., and Prontu, L. 2006. Cognition, technology and
games for the elderly: An introduction to ELDERGAMES
Project. PsychNology Journal, vol. 4, pp. 285-308.
[3] Hassenzahl, M., Burmester, M., Koller, F. 2003. AttrakDiff:
A Questionnaire for Measuring the Perceived Hedonic and
Pragmatic Quality. Human & Computer 2003. Pp. 187-196.
[4] Kern, D., Stringer, M., Fitzpatrick, G., and Schmidt, A. 2006.
Curball - A Prototype Tangible Game for Intergenerational
Play. IEEE Press.
[5] Kwon, G.H., Kim, L., and Park, S. 2013. Development of a
cognitive assessment tool and training systems for elderly
cognitive impairment: A case study of a successful
development of a cognitive assessment tool and training
systems for the elderly people in South Korea. 7th
International Conference on Pervasive Computing
Technologies for Healthcare and Workshops.
[6] Leitner, M., Tomitsch, M., oltringer, T. K., Kappel, K., and
Greshenig, T. 2007. Designing tangible tabletop interfaces
for patients in rehabilitation. In Proceedings of Conference &
Workshop on Assistive Technologies for People with Vision
and Hearing Impairments: Assistive Technology for All
Ages. M. Hersh (Eds.), Spain. pp. 1-7.
[7] Loureiro, B., and Rodrigues, R. (2011) Multi-touch as a
Natural User Interface for elders: A survey. Information
Systems and Technologies, 6th Iberian Conference. IEEE.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 39 |
 Analyzing Learnability of Common Mobile Gestures used by Down Syndrome Users
A. Mendoza | F., J. Alvarez | R. Mendoza | F. Acosta | J. Muñoz

Analyzing Learnability of Common Mobile Gestures used

by Down Syndrome Users
Alfredo Mendoza G. Francisco J. Alvarez R. Ricardo Mendoza G.
Universidad Juarez Universidad Autónoma Instituto Tecnológico
Autónoma de Tabasco de Aguascalientes de Aguascalientes
Av. Universidad S/N, Av. Universidad 940, Cd. Av. A. López Mateos Ote. No. 1801,
Magisterial, Villahermosa, Tab. Universitaria, Aguascalientes, Ags. Bona Gens, Aguascalientes, Ags
(52) 993 312 4680 (52) 449 910 7400 (52) 449 910 5002
[email protected] [email protected] [email protected].

Francisco Acosta E. Jaime Muñoz A.

Universidad Juarez Universidad Autónoma
Autónoma de Tabasco de Aguascalientes
Av. Universidad S/N, Av. Universidad 940, Cd.
Magisterial, Villahermosa, Tab. Universitaria, Aguascalientes, Ags.
(52) 993 312 4680 (52) 449 910 7400
[email protected] [email protected]

ABSTRACT considered in the design of any user interface in order to be

The objective of this research was to analyze the learnability of friendly-used, since these limitations may affect the use of any
the 8 most common mobile gestures used by first-timers Down computational application when it has not been well designed.
Syndrome (DS) users. The study was performed by testing mobile The quality and mount of knowledge, which designers had
gestures usage by 10 DS teenagers that had never interacted with considered to the final product about users, will benefit the user
a mobile/gestural interface. They developed 18 tasks over a tablet experience. Understanding people, their characteristics,
computer. Results were measured by the Task Performance capabilities, commonalities and differences allow designers to
Learnability Metric which focuses in factors such as success, create more effective, safer, efficient, and enjoyable systems
optimality, and error decreasing. Although gestural interaction (Ritter et al., 2014).
was thought to be intuitive by design, DS users face limitations in This study focuses in the learnability of the most common mobile
fine-motor and visual skills, so as in eye-hand coordination, which gestures used by DS users. Learnability is one of the five
difficult manipulation of particular mobile gestures. This vein, components of Usability (Learnability, Efficiency, Memorability,
findings of this research helped to determine the most adequate Errors and Satisfaction) and is defined as how easy for users is to
interactions for mobile gestural interfaces used by DS users. complete basic tasks the first time they encounter with some
design (Nielsen, 2012). This means that the main goal of this
Categories and Subject Descriptors study is to analyze how easy to learn are the most common mobile
H.5.2 [Information Interfaces and Representation]: User gestures by the DS users in the very first encounter with mobile
Interfaces - Graphical user interfaces, Input devices and devices.
strategies
2. BACKGROUND
General Terms There are significant characteristics of DS individuals that may
Documentation, Performance, Design, Human Factors affect their performance as computers users. Based on the
complete list of characteristics found in (Champman & Hesketh,
Keywords 2000, Lauteslager, 2000), the most related with mobile gestures
UX, Learnability, Usability, Interaction, Disabilities are shown next:
1. INTRODUCTION • Short hands and broad fingers.
Down Syndrome (DS) is caused by a genetic anomaly where • Difficulties in fine and gross motor skills.
subjects have an extra copy of chromosome 21; such genetic
condition brings particular characteristics in learning, • Low muscle tone
communicating and perceiving skills. Worldwide, the estimated • Difficulties in vision capacities
incidence of DS is between 1 in 1,000 to 1 in 1,100 live births. • Poor eye- hand coordination
About 6,000 babies are born in the United States with DS each • Anxiety
year (Parker et al, 2010). The prevalence of Down syndrome is • Difficulties in comprehension of abstract concepts
about 8 people with DS per each 10,000 people in the United
States (Presson, 2013), 7 per 10,000 in England and Wales (Wu & Drs. Jinjuan Feng and Jonathan Lazar of Towson University had
Morris, 2013), and 8 per 10,000 in Spain (FESD, 2010), to quote leaded some important researches on Human-Computer
some data. The particular characteristics of DS people might be Interaction that involved DS users. In (Lazar, 2010), they applied

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 40 |
 Analyzing Learnability of Common Mobile Gestures used by Down Syndrome Users
A. Mendoza | F., J. Alvarez | R. Mendoza | F. Acosta | J. Muñoz
a survey about general usage of Computers to 561 DS children.
Among their most important findings, they show that 93% of the
children use mouse without much difficulty, they also remark that
they applied the survey only to children that already use
computers, but it is a remarkable finding because their Figure 1. Tap gesture
expectations based on the clinical information was overcome. Dr.
Lazar also analyzed the work-place related skills of adult expert 3.2 Double Tap
users with DS; he found that many of the participants were able to The double tap (figure 2), also known as double touch, consists in
efficiently use different programs, systems and devices at work rapidly touching twice the surface of the device with one fingertip
(Lazar et al, 2011). These papers leave an open question about in the same point (Apple, 2014; Android, 2015). The most
how easy to use were input techniques for DS users. In (Hu, et al, common functionality of this gesture is to zoom in (or out when is
2011), through an empirical study that involves eight users with already zoomed) a picture.
DS, the efficacy of keyboard-mouse, word prediction, and speech
recognition, where examined. All of them already were familiar
with the keyboard-mouse input technique, and seven of the eight
participants completed all the tasks, although none of them where
used neither word prediction nor speech recognition.
Figure 2. Double tap gesture
Within their findings, they confirm the fact that Dr. Lazar and
their team visualized previously: One design is unlikely to fit the 3.3 Swipe
needs of all DS users, instead, offering different design options Swiping is the action of pressing, moving and lifting at some point
targeting for specific user profiles might be a more effective of the screen. One particular characteristic of this gesture, in order
solution (Hu, et al, 2011). They mention two factors that influence to avoid misunderstandings with others, is that the fingertip that
the user’s performance level: the computer experience (number of triggers the gesture will never lose contact with the surface until
years using computers) and number of hours per week of the gesture is done. Figure 3 presents the graphic representation of
computer usage. swipe gesture.
In 2012, the research team of Towson University made an
empirical study about how usable were the tablets computers used
by expert adults with DS (Kumin et al, 2012). They found that
adults with DS are able to use multi-touch devices when already
had received formal computer training. Involving 10 participants, Figure 3. Swipe gesture
Social networking, Email, Calendar, Price comparison, and Text
entry/note-taking activities made in a tablet computer were tested. 3.4 Flick
It is important to remark that all users were experts and use Flick is defined as quickly brush the surface of the device with
computers in their everyday work activities. one fingertip (Villamor et al., 2010). The gesture finishes with the
One interesting finding in this study was that some users had fingertip away from the surface, that and the speed of doing it, are
problems with the sensitive nature of the device, mixing and the main differences between flick and swipe. Figure 4 is the most
confusing gestures, this gives the idea that mobile gestures are not common graphical representation of flick gesture. The most
as self-evident as it was thought. known action of this gesture is to advance (forwards or
backwards) rapidly in a list of elements.
One of the main advantages of the gestural interfaces used by
mobile devices over traditional keyboard-mouse interfaces is the
direct relationship with ordinary-living gestures; this implies that
users must not necessarily pass through a capacitation program for
learning the manipulation of such devices; being natural, last Figure 4. Flick gesture
generation of mobile devices interfaces, allows an intuitive
learning. Thus, the question discussed in this study is how natural 3.5 Hold and drag
are the mobile interfaces gestures for DS novice users? This gesture is composed of two elements: long touch and swipe.
By holding the touch with one fingertip over a specific element of
3. COMMON MOBILE GESTURES the screen, the user can drag it to another position. Commonly, by
This section describes the 8 most commonly used mobile gestures. holding and dragging an object of the screen, user may change its
It is important to mention that, with the exception of figure 5, all position, send it to trash or close it. In order to represent the hold
gestures representations are based on (Villamor et al., 2010). and drag gesture, authors has develop the figure 5, based on those
shown in (Villamor et al., 2010).
3.1 Tap
Tap gesture is the most basic one; it consists in pressing briefly
the surface of the device with one fingertip. The result could be
just the selection of an item of the screen or the trigger of a
predefined action. (Apple, 2014; Android, 2015), the typical
symbol to express this gesture is shown in figure 1. Figure 5. Hold and Drag gesture

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 41 |
 Analyzing Learnability of Common Mobile Gestures used by Down Syndrome Users
A. Mendoza | F., J. Alvarez | R. Mendoza | F. Acosta | J. Muñoz

3.6 Pinch
The pinch gesture consists in pressing the surface of the device Table1. Specs of device
with two fingers, move them inwards and lifting (Google, 2015).
The graphical representation shown in figure 6, presents the pinch Characteristic Samsun Galaxy
gesture made with thumb and index fingers. Display size 10.1’’
Screen resolution 1280x800
Internal memory 16GB
Processor 1.2 GHz Qd-Core
Figure 6. Pinch gesture Platform Android™ 4.4

3.7 Spread 4.3 Metrics

Spread gesture implies the inverse movement of pinch gesture; i. In (Grossman et al., 2009) a taxonomy of learnability evaluation
e. the gesture begins with pressing the device’s surface with the metrics was presented; there, six different categories group the
two fingers close-together, spreading and lifting. Figure 7 presents metrics according with their objective. For this study, the next
a graphical representation. points, taken form task performance metric, was chosen:
• Percentage of users who completed the task optimally
• Percentage of users who completed the task without any help
• Ability to complete task optimally after certain time frame
• Decrease in task errors made over certain time interval
Figure 7. Spread gesture • Time until user completes a certain task successfully

3.8 Rotate In order to clarify the way the metrics will be measured, an
The rotate gesture consists in touching the device’s surface with explanation for the key terms is presented next:
two fingers and move them in a clockwise or counterclockwise • A task is considered as optimally completed when the user
direction (Villamor, et al., 2010). A graphical representation is completes all the instructions at the first try, without any help.
shown in figure 8. • A task is considered as completed without help when the user
completes the task successfully, regardless the number of tries.
• A task is considered as completed successfully when the user
triggers the current gesture action without errors.
• An error occurs when the actions of the users do not trigger the
Figure 8. Rotate gesture
current gesture’s actions. A hesitation is not considered as
4. EMPIRICAL STUDY error.
• All the time measured starts when the first explanation is
4.1 Participants
completed, and stops when the activity is successfully
The study involves 10 DS teenagers between 15 and 20 years old;
all of them enrolled in a special education center where no completed.
computational instruction is given to them. All 10 participants
Thus, data were computed as next:
had no experience with mobile devices, touchscreens or
smartphones. It is worth mentioning that all parents of the 10 • Percentage of users who completed the task optimally
users involved in this research gave their consent by signing the • Percentage of users who completed the task without any help
institutional consent document based on the Informed Parental
• Percentage of users that acquired the ability to complete task
Consent for Research Involving Children of the World Health
Organization. optimally after certain time frame
• Percentage of users that decrease in task errors made over
4.2 Instruments certain time interval
The device selected for this study was the Samsung Galaxy Tab • Average time until user completes a certain task successfully
10.1; specifications are shown in table1.
4.4 Experiment design
28 users’ tasks were developed in order to test all 8 gestures. The
main premise was to define tasks where the cognitive load would
not affect the user performance of any gesture. From this fact, the
next rules where followed by each user task:
• Use the minimum number of colors as possible
• Use simple images without too many realism
• Avoid abstract images
• Avoid ambiguity

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 42 |
 Analyzing Learnability of Common Mobile Gestures used by Down Syndrome Users
A. Mendoza | F., J. Alvarez | R. Mendoza | F. Acosta | J. Muñoz
• Avoid words inside the images. Intermediate: Use flick and stop it.
• Use white background when possible. Advanced
• Use the most typical forms of all objects. (a) Use flick forwards and stop it to reach certain page.
(b) Use flick backwards and stop it to reach certain page.
Figure 9 shows an example of the application of the rules, both
images represent a house; left side picture is overloaded with 6. Rotate:
graphical information (trees, ground, fence, bench, etc.), right Basic: Rotate image
picture is a simple representation of a house with minimum Intermediate: Rotate and resize an image
colors. Advanced: Rotate 180° and zoom in an image.
7. Pinch:
Basic: Zoom out a picture
Intermediate: Users must reach the minimum zoom of a picture
as fast as they can.
Advanced: Users must zoom-out a picture and rotate it 180° in
clockwise at the same time.
8. Spread: All tasks will be developed using only one hand.
a) b)
Basic: Zoon in a picture
Figure 9. a) Overloaded Picture b) Simple picture Intermediate: Users must reach the maximum zoom of a
Every gesture was tested in three levels: basic, intermediate, and picture as fast as they can.
advanced; basic level implies the simplest tasks where the user Advanced: Users must zoom-in a picture and rotate it 180° in
only makes the gesture. Levels intermediate and advanced involve clockwise at the same time.
more complicated tasks that involve repetitions, combinations, or
Next table shows an overview of all the 28 tasks:
specified conditions of gestures. Next are shown the tasks defined
for each gesture: Table2. Overview of user tasks
1. Tap: In a grill given to user, he/she must tap the indicated Gesture Level Task
cell. The size of each cell is decreasing with each success. Basic Tap the indicated cell, cell
The three levels are tested with the cell size.
sizes:7’’and 4’’
2. Double tap: In a grill given to user, he/she must tap the Intermediate Tap the indicated cell, cell size:
indicated cell. The size of each cell is decreasing with each Tap
2’’and 1’’
success. The levels of difficulty are evaluated same way as
Advanced Tap the indicated cell, cell size: .5’’
the first task.
and .25’’
3. Hold and drag: For all levels of difficulty, a group of left- Basic Double-tap the indicated cell, cell
aligned elements is given to users; they must drag all of them size: 7’’ and 4’’
to their final position, the difference between levels is the way
the user must follow with the object. Double Intermediate Double-tap the indicated cell, cell
Basic: Objects must be moved in a straight line. Tap size: 2’’and 1’’
Intermediate level: Objects must be moved in an “L” line. Advanced Double-tap the indicated cell, cell
Advanced: size: .5’’and .25’’
a) Objects must be moved in a straight line, in certain
Basic Move objects in a straight line.
time. Hold and
Intermediate Move objects in an “L” line.
b) Objects must be moved in an “L” line, in certain time. drag
Advanced Move objects in an “S” line.
4. Swipe: All the next tasks will be done by side to side swipe. Basic Go to the next picture
Basic: Go to the previous one
Swipe
(a) Being in certain picture, user goes to the next one. Intermediate Use swipe continuously.
(b) Being in certain picture, user goes to the previous one. Advanced Use swipe with speed.
Intermediate: Users must go forward in a document, sheet by
Basic Flick forward
sheet until reach some pre-established page, then they must Flick backwards
return to the first one. Flick Intermediate Use flick and stop it.
Advanced: Users must reach the final page of a document as Advanced Use flick and stop it to reach
fast as they can, then they must return to the first one as the certain page.
same way. Basic Rotate image
Rotate Intermediate Rotate and resize an image
5. Flick Advanced Rotate 180° and zoom in an image
Basic: Use flick forward and backwards. Pinch Basic Zoom out a picture

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 43 |
 Analyzing Learnability of Common Mobile Gestures used by Down Syndrome Users
A. Mendoza | F., J. Alvarez | R. Mendoza | F. Acosta | J. Muñoz
Intermediate Reach the minimum zoom 4.5.2 Double tap
Advanced Zoom-out a and rotate 180° Double tapping involves the same windows and cells of single
tap, but users have to touch twice the surface in order to advance
Basic Zoom in a picture in the activity.
Spread Intermediate Reach the maximum zoom
Common mistakes involved accidental activation of other gestures
Advanced Zoom-in a and rotate 180° (such as swipe), multiple tapping. For some users were difficult to
make only two taps in the tiny cells.
4.5 Study development The same consideration for time measurement made with single
Although, all activities were planed and designed with a multi- tapping, were done for double tapping: only advance level were
disciplinary team involving psychologists, clinicians and experts measured. Corresponding results are shown in table 4.
in special education, all tests were made with the presence of the
psychologist in order to supervise that users understand the Table 4. Results for double tap
instructions and the activities procedure. Level Basic Intermediate Advanced
All 10 individual tests were videotaped in order to be analyzed Factor
and data were tabulated. All results obtained in the study are
Percentage of users who 90% 60% 30%
presented arranged by gesture. A table presents the corresponding
completed the task
value involving all 10 users’ information using the metric
optimally
presented in 4.3. Some data were note as N. M. (Not Measured)
Percentage of users who 100% 60% 30%
due to the low significance of measuring that factor.
completed the task
4.5.1 Tap without any help
The goal of all activities for tap gesture was tapping the indicated Percentage of users that 100% 90% 80%
(in green color) cell of a table. The size of all cells reduces at each acquired the ability to
success, whilst the number of cells increases; thus, the difficulty complete task
also increases. successfully after certain
time frame
For tap gesture, since this is a very easy task, the time was not Percentage of users that 100% 100% 90%
measured in basic and intermediate level, due to users tap the decrease in task errors
corresponding cell almost instantly. In advance level, the time made over certain time
was taken until user complete the tasks, and involves time for interval
locating the cell and time expended in all tries. The results for tap Average time until user N. M. N. M. 2.5
are shown in table 3. completes a certain task
As it is shown in the table, some users fail to complete tasks successfully (secs.)
where more precision were required; errors involved accidental
activation of swipe and double tap, keep holding and too much
force applied to make the tap (resulting in not detection of the
4.5.3 Swipe
gesture by the device). A document with several pages, each of them presenting only one
single basic image of fruits, vegetables, animals and other simple
Table 3. Results for single tap objects, was shown to the users; after a brief explanation of the
Level Basic Intermediate Advanced functionality to advance forward and backwards through the pages
of the document, user was ask to advance to the next, previous
Factor and certain image. Next, user was asked to go to certain
Percentage of users who 100% 60% 30% previously located image, so he/she must do the swipe gesture ten
completed the task times. Once user was familiar with this gesture, for advanced
optimally level, he/she was asked to go to certain object (10 pages away to
Percentage of users who 100% 80% 30% the current) as fast as he/she can. Single swiping did not represent
completed the task a challenge to any user, but when multiple swiping was required,
without any help users tend to make some mistakes. Among the most common
Percentage of users that 100% 100% 80% were: undetected swipe, due to incomplete or short gestures,
acquired the ability to diagonal swipe, that sometimes results in undetected gestures, and
complete task opposite swipe, due to no-lifting finger from the device’s surface
successfully after certain when swiping. All errors were most presented in advance level
time frame activities (See Table 5).
Percentage of users that 100% 100% 100% 4.5.4 Flick
decrease in task errors For flick gesture, the same document with simple images was
made over certain time used, after a brief explanation of its functionality users were asked
interval to do the activities. First, users made the flick to advance forwards
Average time until user N. M. N. M. 1.5 and backwards, in a preview-slide of the all document pages.
completes a certain task Then he/she was asked to stop the slide before stops itself. Once
successfully (secs.) user understands the gesture and its actions, he/she was asked to

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 44 |
 Analyzing Learnability of Common Mobile Gestures used by Down Syndrome Users
A. Mendoza | F., J. Alvarez | R. Mendoza | F. Acosta | J. Muñoz
stop the slide in a certain page. For this activity, the objective
page was the only one in colors, the rest of all pages were in gray
scale. The most common behavior of users was trying to Table 6. Results for flick
accelerate the slide by doing multiple slides. The result of this Level Basic Intermediate Advanced
action was commonly to stop the slide. Factor
Percentage of users who 20% 10% 10%
Table5. Results for swipe completed the task
optimally.
Level Basic Intermediate Advanced Percentage of users who 40% 60% 10%
Factor completed the task
without any help
Percentage of users who 90% 90% 70%
Percentage of users that 80% 80% 20%
completed the task
acquired the ability to
optimally
complete task
Percentage of users who 100% 90% 70%
successfully after certain
completed the task
time frame
without any help
Percentage of users that 80% 80% 20%
Percentage of users that 100% 100% 100%
decrease in task errors
acquired the ability to
made over certain time
complete task
interval
successfully after certain
Average time until user 4.6 8.6 13
time frame
completes a certain task
Percentage of users that 100% 100% 100%
successfully (secs.)
decrease in task errors
made over certain time
interval 4.5.5 Hold and drag
Average time until user N. M. 12.25 7.4 This activity involves two steps: tap over one object of the screen
completes a certain task and hold it, and then, move the object to another place. It was
successfully (secs.) especially difficult to user to understand the moment when he/she
can move the object; that is why none of them complete the
The advanced level activity was very hard to achieve by all users. activity optimally. Once the user hold the tap over one object, it
At first, researchers thought it was caused of vision problems, but takes less than a second to presented a colored-transparent square
after a brief analysis of the visual capacities of users it was that covers the object, then it is enabled to be moved. At first,
discarded. Researchers and experts support that the difficulty was users were asked to move certain object in a straight line and
due to a coordination problem that in general affects DS place in a previously located square. In general, all three levels of
individuals. difficulty involve moving an object to a predefined place by
following the indicated trajectory. The most common mistakes
Table 6 shows the information tabulated that involve all 10 users’ made in hold and drag gesture were trying to move the object
tests for flick gesture. without hold, and stop holding in the way to the final position.
Sometimes, users lift the finger in the middle of the trajectory;
they were asked to do it again but without releasing the object.
Only those users that did not make this mistake were tabulated as
optimally completed. All results are shown in table 7.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 45 |
 Analyzing Learnability of Common Mobile Gestures used by Down Syndrome Users
A. Mendoza | F., J. Alvarez | R. Mendoza | F. Acosta | J. Muñoz

Table 7. Results for hold and drag Table 8. Results for Rotate
Level Basic Intermediate Advanced Level Basic Intermediate Advanced
Factor Factor
Percentage of users who 0% 0% 0% Percentage of users who 80% 60% 40%
completed the task completed the task
optimally optimally
Percentage of users who 0% 0% 0% Percentage of users who 80% 80% 90%
completed the task completed the task
without any help without any help
Percentage of users that 50% 50% 40% Percentage of users that 100% 100% 100%
acquired the ability to acquired the ability to
complete task complete task
successfully after certain successfully after certain
time frame time frame
Percentage of users that 50% 50% 40% Percentage of users that 100% 100% 100%
decrease in task errors decrease in task errors
made over certain time made over certain time
interval interval
Average time until user 10.5 33.6 15.6 Average time until user N. M. 4.5 5.6
completes a certain task completes a certain task
successfully (secs.) successfully (secs.)

4.5.6 Rotate 4.5.7 Pinch

In general, users can do pinch, spread, and rotate with two fingers After a brief explanation of making the pinch and the spread
of one hand or with one finger of each hand. Both ways of doing gestures over a picture, user was asked to do it. Once he/she
these gestures was shown to users, when users seem to have understand both gestures in general, particular activities to each
troubles with one way; then were asked to do it in the other way. were asked to be done. Users were asked to zoom-out a picture as
Optimally data was record when users made it with one of the two it can be done in order to test the continuous pinching over one
options, even if fail with the first one. Users were asked to rotate a point, and then were asked to increase the speed: next picture,
picture with no restrictions, and then were asked to rotate 90° and more speed. Once the gesture was understood, it was easy for
180° a picture. Pictures involve single images of simple objects, users to make multiple gestures slowly; but when speed factor
animals and fruits that were easy understandable not only their entered, mistakes begin to happen. Common mistakes when users
contents but their position, this was verified by experts. The tried to do multiple pinch rapidly were: undetected gesture,
application used for these activities restricts the exact position of undecireble swipe, spread, and rotate gestures. Results are shown
the picture, no help is on positioning. Sometimes users achieve the in table 9.
goal of the activities but taking various steps, first rotating the Table 9. Results for Pinch
picture 90°, stopping, replacing hands and rotating another 90°;
these users were not taken into account as optimally completed. Level Basic Intermediate Advanced
All results for this gesture are shown in next table: Factor
Percentage of users who 60% 20% 0%
completed the task
optimally
Percentage of users who 70% 60% 40%
completed the task
without any help
Percentage of users that 90% 80% 80%
acquired the ability to
complete task
successfully after certain
time frame
Percentage of users that 90% 80% 80%
decrease in task errors
made over certain time
interval
Average time until user N. M. 15.5 5.4
completes a certain task
successfully (secs.)

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 46 |
 Analyzing Learnability of Common Mobile Gestures used by Down Syndrome Users
A. Mendoza | F., J. Alvarez | R. Mendoza | F. Acosta | J. Muñoz

4.5.8 Spread 5. Conclusions

Pinch and spread gestures cannot be explain to users in separated People with DS commonly have problems with fine motor skills
activities, since they involves related movements of fingers and (coordination, manipulation), skills required to interact with any
produce very similar results in the test application. Nevertheless, mobile/gestural interface. It was found that there are some
activities for each gesture were applied separately. Activities gestures that are simple and very understandable by DS users, but
involved zooming-in basic pictures, reach the maximum size of it others are still hard to discover, learn and apply. Interfaces where
by making continuous spread gestures, and do it with speed. this fact is not considered on their design will diminish the user’s
Results shown in table 10 denote that it is a little more difficult to performance.
DS users to make, rapidly and continuously, the spread gesture
than the pinch gesture, probably because pinch movement is more Tap gesture, for example, is the most basic and simple gesture; for
commonly used in real life than it is spread. Users have the same DS users it was evident how to make a tap over an interface
errors at advanced level as they have in pinch: undetected gestures object with great performance and almost no trouble; but due to
and mistake gestures. the common visual problems in DS users, plus the characteristics
of hands and fingers (see part 2) it turns to be difficult for them to
Table 10. Results for Spread make tap gesture over the tiny ones. These factors affect the same
Level Basic Intermediate Advanced way the development of double-tap gesture.
Factor It was also evident the usage of swipe gesture when used in a
normal way. When DS users were asked to swipe continuously
Percentage of users who 70% 20% 0%
and rapidly, they tend to make mistakes.
completed the task
optimally It turned out that flick is one of the most difficult gestures to
Percentage of users who 70% 60% 20% make, especially when users were asked to stop the slide. It
completed the task requires great eye- hand coordination that becomes a real
without any help challenge to them.
Percentage of users that 90% 80% 40%
There were some troubles with Hold and drag gesture, but only
acquired the ability to
when the required time for hold was too long. The default time of
complete task
new systems such as Android 5.0.1 works just fine.
successfully after certain
time frame The last three gestures, rotate, pinch and spread, did not present
Percentage of users that 90% 80% 40% greatest troubles in the basic actions. Errors appeared when users
decrease in task errors start to mix them and repeat them continuously. Along with flick
made over certain time gesture, they required a more extended explanation than the first
interval four, to be completely understood by users.
Average time until user N. M. 17.6 6
completes a certain task Based on these findings, the next recommendations are proposed:
successfully (secs.) • For very-first-timers users, require only simple gestures that
involve one simple movement, trigger only one evident and
By calculating the joint probability between the three levels of distinguishable action, and that go according to real life, not
difficulty of all the first four factors, the probability that one only for their movement but also for their action that trigger.
individual succeed in at least one factor (independently of the
• Add a flexible error handling rule for gestures considering that
level) is shown in Figure 10. With this graphic, it is easy to
identify which gestures are difficult and which are not. users may accidentally make undesirable gestures.
• Avoid, as much as possible, gestures that involves great eye-
hand coordination such as flick, especially for novice users.
• Avoid little tiny selectable objects even for expert users.

These recommendations do not tend to asseverate that DS users

could not be able to use some robust mobile interfaces; with the
proper capacitation plan, users can understand, use, and even
master almost any activity, in fact, throughout the study, all users
increase their performance as they use the mobile devices.

6. ACKNOWLEDGMENTS
Authors want to thank all students, parents, instructors and
clinicians that collaborate with this research. Specially thanks to
Dr. Cristian Russu and Rodolfo Inostroza from the Pontificia
Universidad Católica de Valparaiso, to share their experience and
hard-work.

Figure 10. Probability of success on each gesture

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 47 |
 Analyzing Learnability of Common Mobile Gestures used by Down Syndrome Users
A. Mendoza | F., J. Alvarez | R. Mendoza | F. Acosta | J. Muñoz

7. REFERENCES [8] Lauteslager, P., 2000, Children with Down’s Syndrome:

[1] Apple Inc., Apple Developer Site, Motor Development and Intervention, Thesis University
https://developer.apple.com/, last visited February 2015. Utrecht, The Netherlands.
[2] Chapman, R., and Hesketh, L., Behavioral phenotype of [9] Lazar, J., Kumin, L., Feng, J., 2011, Understanding the
individuals with Down syndrome, Journal: Mental computer skills of adult expert users with Down syndrome:
Retardation and Developmental Disabilities Research an exploratory study, The proceedings of the 13th
Reviews - MENTAL RETARD DEV DISABIL RES , vol. 6, international ACM SIGACCESS, 51-58, ACM New York,
no. 2, pp. 84-95, (2000). NY, USA.
[3] FESD (Federación Española de Síndrome de Down), 2010, II [10] Parker SE, Mai CT, Canfield MA, Rickard R, Wang Y,
Plan de Acción para Personas con Síndrome de Down, 2009- Meyer RE, Anderson P, Mason CA, Collins JS, Kirby RS,
2013. Down España, Madrid (ISBN:9788469315651). Correa A; National Birth Defects Prevention Network.
Updated national birth prevalence estimates for selected birth
[4] Google Inc., Android Developers Site, defects in the United States, 2004-2006. Birth Defects Res A
http://developer.android.com/, last visited February 2015. Clin Mol Teratol. 2010;88:1008-16.
[5] Grossman, T., Fitzmaurice, G., Ramtin, A., A survey of [11] Persson, H., Ahman, H., Yngling, A., Gulliksen, J. Universal
software learnability: metrics, methodologies and guidelines, design, inclusive design, accessible design, design for all:
CHI '09 Proceedings of the SIGCHI Conference on Human different concepts one goal?, Universal Access in the
Factors in Computing Systems, 649-658, ACM New York, Information Society, Springer, (2014).
NY, USA, (2009).
[12] Strothotte, C., Strothotte, T., Seeing Between the Pixels:
[6] Hu, R., Feng, J., Lazar, J., Kumin, L., 2013, Investigating Pictures in interactive Systems, Springer-Verlag Berlin
input technologies for children and young adults with Down Heidelberg (1997).
syndrome, Universal Access in the Information Society,
March 2013, Volume 12, Issue 1, pp 89-104. [13] Villamor, C., Willis, D. and Wroblewski, L., Touch Gesture:
Reference Cards, LukeW Ideation + Design, (2011).
[7] Kumin, L., Lazar, J., Feng, J., Wentz, B., and Ekedebe, N.
(2012). Usability Evaluation of Workplace-Related Tasks on [14] Wu, J., Morris, J., 2013, The population prevalence of
a Multi-Touch Tablet Computer by Adults with Down Down’s syndrome in England and Wales in 2011, European
Syndrome. Journal of Usability Studies 7(4), 118-142. Journal of Human Genetics 21, p. 1016–10

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 48 |
 Multi-touch Technology in Early Childhood: Current Trends and Future Challenges
V. Nacher | J. Jaen

Multi-touch Technology in Early Childhood: Current

Trends and Future Challenges
Vicente Nacher, Javier Jaen
ISSI Group, Dpto. Sistemas Informáticos y Computación
Universitat Politècnica de València.
[email protected] [email protected]

ABSTRACT
The advantages of the direct manipulation style make the multi-
touch technology an ideal mechanism to support learning
activities for children. Moreover, although pre-kindergarten
children are becoming frequent users of the technology little work
has been done in the area to assess their actual abilities. This
paper goes over the state of the art of multi-touch technology
targeting pre-kindergarten children and its use for educational
purposes. In addition, in this work we present future challenges
that should be faced in the area in the near future to establish the
basis on which designers will develop educational applications for
children that fully exploit the multi-touch technology according to
the actual abilities of pre-kindergarten children.

Categories and Subject Descriptors Figure 11. Target age 2009 vs 2011 for Education cat. in the
H.5.2. [Information interfaces and presentation]: User Interfaces - Apple Store [22].
Interaction Styles. However, the increasing interest in multi-touch technology has
not given rise to studies on the design of multi-touch systems for
General Terms the youngest age range [10]. Moreover, the lack of standardized
Performance, Design, Experimentation, Human Factors, and universally accepted interactions for these challenging users
Standardization. makes the design of well-designed multi-touch interactions even
Keywords more crucial [11].
Multi-touch interaction; pre-kindergarten children; gestures; According to these facts, in this paper we carry out a review of the
future challenges current state of the literature of multi-touch technology with pre-
kindergarten children and provide a set of future challenges to be
1. INTRODUCTION addressed in the future. The end goal is to define a research
Multi-touch technology has rapidly evolved in recent decades and agenda to make this technology usable by pre-kindergarten
nowadays it has widespread acceptance [5] because it provides children and give application designers the necessary guidelines to
users with a more natural and intuitive way to interact [23]. As develop touch applications according to the actual skills of pre-
pointed out in [7] children between zero and eight years old are kindergarten children.
frequent users of digital media in the USA and they meet with
touch technology often before they can even speak. Supporting 2. STATE OF THE ART
these ideas the Horizon report [12] places mobile devices, such as Until very recently, there have been no research efforts addressing
tablets and smartphones, as one of the two emerging technologies multi-touch interaction with pre-kindergarten children (aged less
suitable for children aged under 2 years. than three years). Possibly this has been the case because age is a
limiting factor for experimental studies; young children do not
The three basic ideas behind the direct manipulation concept were
have the verbal and cognitive skills to express their likes and
listed by Shneiderman and Plaisant [21]: (1) the visibility of
preferences [15] and, they are not able to carry out tasks for long
objects and actions of interest; (2) the replacement of typed
periods or are easily distracted [8]. However, according to Piaget
commands by pointing-actions on objects of interest; and (3) the
[19], children are in a preoperational stage from 2 years old
rapid, reversible and incremental actions that help children to keep
onwards, i.e., they begin to think in terms of images and symbols,
engaged, giving them control over the technology and avoiding
and develop symbolic play with imaginary objects, which means
complex instructions.
they could be candidates for multi-touch technology. In addition,
A detailed analysis of applications in the Apple store made by being aware of children developmental abilities is critical when
Shuler [22] reveals the growing use of educational applications designing software for the very young [26].
for children based on touch devices. This is specially the case for
This has motivated recent studies that assess the abilities of pre-
preschool children (Figure 11) who were the target users of nearly
kindergarten children to perform basic touch gestures. The work
60% of these applications by 2011.
by Nacher et al [18] reveals that the advantages and features of
multi-touch technology is not being fully exploited in existing
commercial applications since only the drag and tap gestures are

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 49 |
 Multi-touch Technology in Early Childhood: Current Trends and Future Challenges
V. Nacher | J. Jaen
being used and no support for collaboration is given. Hence, the
work evaluates a set of basic touch gestures with children aged
from two to three years (Figure 12) concluding that even these
young users are able to perform a set of basic touch gestures such
as tap, drag, scale (up & down) and one finger rotation. In
addition, in this work a set of design guidelines is proposed to
deal with complex gestures such as double tap and long pressed.
These assisted strategies were evaluated later in [16] and results
show that even these more complex gestures can be successfully
performed by pre-kindergarten children with basic assistance.
Figure 13. Child performing double drag gestures (extracted
from [24]).
Usability is not the only dimension that has been addressed by
recent studies in the field. Another topic of interest is that of
communicability when pre-kindergarten children are considered.
The work in [17] report a first approach to evaluate mechanisms
for applications to communicate pre-kindergarten children the
expected multi-touch gestures at a given moment. In this study,
the authors present and evaluate two visual approaches (iconic
and animated) to communicate touch gestures (see Figure 14).
Three touch gestures are considered: the tap representing in-place
gestures (i.e. those in which the hand does not actually describe a
trajectory but taps at a very specific pace or in a specific way); the
drag representing one-contact point gestures that require a
movement following a specific trajectory; and the scale up gesture
representing two-contact point gestures that require movement.
The results show, firstly, that none of the evaluated languages is
Figure 12. Examples of scale up and drag tests (extracted
effective to communicate in-place gestures and, secondly, that the
from[18]).
animated approach overcomes the iconic one for gestures that
In the experimental studies in [1, 2] four applications are require movement of contacts reaching success rates above 85%.
considered and the interaction needed to play with them is This fact suggests that even pre-kindergarten children are able to
evaluated. The gestures under test are the tap, drag, rotate, drag interpret the direct mapping between the visual stimuli (i.e. a hand
and drop, pinch, spread and flick and the experiment involves sliding on the surface) and the gesture to be performed. Therefore,
children aged 2 to 4 years. According to the results, the authors this work suggests that visual languages could be an effective way
conclude that children aged four are able to perform all the to enable pre-kindergarten children autonomous interaction.
evaluated gestures. Those aged three only find problematic the
spread task. Finally, those aged two are able to perform the tap
and drag gestures properly, learn quickly the flick gesture but they
are less effective with the more complex gestures such as drag and
drop and pinch.
Another interesting study is the work of Vatavu et al [24] who
evaluate touch gestures with children between 3 and 6 years using
tablets and mobile devices. They conclude that despite there is a
significant performance growth with age, in general, children have
good performance with the tap, double tap and single hand drag
and drop gestures. However, in the case of the double hand drag
and drop gesture (see Figure 13) they do not reach these good Figure 14. Description of the animated visual (top) and the
results and the success rate drops to 53.7%. In addition, authors iconic (bottom) language for the scale up gesture (extracted
correlate the results with a sensorimotor evaluation based on from [17]).
children’s finger dexterity and their graphomotor and visuospatial Exploring the educational dimension and the suitability of multi-
processing abilities. The correlation shows that children with touch surfaces to support educational activities there are several
higher visuospatial skills (i.e. having better skills for works that strengthen the idea that this technology provides
understanding relationships between objects, as location and benefits for pre-kindergarten children education. For example, the
directionality) perform better in the drag and drop tasks. work by Bebell et al [4] shows a nine week study comparing the
Furthermore, the study also reveals that children with more improvement of kindergarten children taking early literacy lessons
developed visuospatial skills tap closer to the center of targets in when using tablets for learning or using a traditional non-
the tap and double tap tasks since they have a better understanding technological method. Their results show that the group that
of the targets’ geometries. learned with tablets scored higher on early literacy assessments,
particularly these higher scores are present in the ability to
recognize sounds and represent sounds and letters. Following the
same line, Chiong and Shuler [6] conduct an experiment involving
touch devices and audiovisual material adapted to children aged

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 50 |
 Multi-touch Technology in Early Childhood: Current Trends and Future Challenges
V. Nacher | J. Jaen
three to seven years and their results show that children obtain of work to do to assess the best approaches. This preliminary
remarkable gains in vocabulary and phonological awareness. study points out that animated languages can be effective to
Moreover, Knoche et al [14] point out that the interaction of communicate touch gestures to pre-kindergarten children and help
children aged between 16 and 33 months with interactive them to be autonomous when using the multi-touch devices.
elements in a tablet does not reduce their comments in dialogic However, as pointed out by the authors, only three gestures were
reading activities. Another example is provided by Zaranis et al evaluated, hence, it remains to be evaluated whether the inclusion
[27] who conduct an experiment to study the effectiveness of of additional gestures has an impact on the overall performance
digital activities on smart mobile devices (tablets) to teach and effectiveness of the languages. Moreover, the gestures were
mathematical concepts such as general knowledge of numbers, tested in isolation; therefore, languages should be studied when
efficient counting, sorting and matching with kindergarten the interaction area is cluttered with many touchable elements and
children. Their results show that the tablet-aided learning their corresponding visual cues or with elements that may be
provided better learning outcomes for the students than the manipulated with several different gestures. With respect to
traditional teaching method. Kammer et al [13] present three communicability of touch gestures, another possible future work
applications to foster the development of cognitive and motor can be the evaluation of other languages both iconic and animated
skills on a multi-touch tabletop with children aged from four to in order to find out which type of language fits better to
six years. The conducted experiment shows that even preschool communicate touch gestures to infants. In addition, an interesting
children are able to use this technology and they enjoy the task future work could be to design and evaluate a similar approach to
and collaborate in the multi-user activity. the method used by Balonian et al [3] for children between five
and six years old. In this work, each gesture was associated to a
The results of these works suggest that pre-kindergarten children
specific character in a way that the gestures were “recallable”.
are prepared to use multi-touch technology and the intuitive and Metaphors such as a walking ladybug for a drag gesture or a
natural interaction of direct manipulation style of the multi-touch jumping grasshopper for a double tap were used. Finding suitable
technology makes it ideal to support pre-kindergarten children metaphors that pre-kindergarten children can understand could be
interaction and, hence, educational activities targeted to them. very useful to develop autonomous educational applications.
Moreover, these works conclude that these particular users are
able to perform a set of touch gestures successfully and future The studies in the literature point out that pre-kindergarten
applications designed for them do not need to be restricted to only children have the necessary skills to make use of multi-touch
basic interactions such as the drag and tap gestures. However, technology. However, these works implement assistive techniques
these works reveal that there is no consensus or standardization of to deal with precision issues during the initiation and termination
the multi-touch interaction style for users in these early ages. phases of gestures since they assume that pre-kindergarten
children are no able to perform the gestures with high accuracy.
3. FUTURE CHALLENGES This causes that the implemented interaction styles do not allow
In this section we present a compilation of future works that we children to have the control over the termination of the gestures
consider interesting and necessary to complete the literature and despite they are in the process of developing their motor skills and
shed light on the specific needs of pre-kindergarten children when some of them may have already the proper cognitive abilities to
using multi-touch technology. perform the gestures with higher levels of precision. As a result,
existing applications designed under these assumptions do not
On the one hand, according to Hinrichs and Carpendale [9] who
benefit from the use of multi-touch technology to help children to
point out that there is evidence that some events are affected by
develop their precision-related cognitive and motor skills.
previous and subsequent ones and given that all the analyzed
According to this observation it would be interesting to evaluate
studies consider and evaluate the gestures in isolation, an
dynamic gestures (those that require movement of contacts over
interesting strand of future work would be to evaluate these same
the surface) demanding high levels of accuracy to users,
gestures that pre-kindergarten can do in isolation but when several
specifically, it would be interesting to evaluate how accurately
interactive elements are shown simultaneously. Users should
they can rotate an object, how close they can drag an object to a
perform different preset sequences of these gestures in order to
target and whether they are able to perform scaling gestures (up
determine whether the cluttering of elements in the interaction or
and down) with enough accuracy to stop the stroke of the gesture
the task chaining affect their performance. These results will help
in a specific moment to reach a desired size. This would certainly
to develop more complex applications which are not restricted to
help in understanding the limitations on accuracy that should be
only one possible interaction in each phase of the game.
fulfilled in applications targeting pre-kindergarten children. In
Another interesting issue that remains to be addressed is the addition, the data collected during the experimentation could be
processing of the unexpected touch events when children are used to develop assistive strategies to deal with precision issues in
holding the tablet with a finger resting on the display or when part an adaptive way for the users that actually need it and not in an
of the palm also touches the surface. This issue is difficult to exhaustive way for every child as current systems do.
address because children may not be aware of such unintentional
Finally, once known the actual capabilities and abilities of pre-
contacts with other parts of their body when their fingers approach
kindergarten children, the gestures that they can perform, the
the screen resulting in an unexpected effect. It would therefore be
accuracy that they can achieve and evaluated the communicative
interesting to explore potential improvements in multi-touch
strategies suitable for them; interesting future works could be
usability, for instance by determining and filtering out unexpected
developed for the definition, construction and evaluation of
blob contacts wherever applicable.
environments based on multi-touch technology that foster
Addressing the topic of the definition of effective mechanisms to creativity and allow collaboration between peers. These
communicate which actions are expected from the user; and environments could be integrated into classrooms and be used
taking as a starting point the work in [17], there is still a great deal with educational purposes to allow children to develop their

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 51 |
 Multi-touch Technology in Early Childhood: Current Trends and Future Challenges
V. Nacher | J. Jaen
creativity skills and allow educators to monitor the progress of [8] Egloff, T.H. Edutainment: a case study of interactive cd-rom
their students and create appropriate content for them. As Rushton playsets. Computers in Entertainment 2, 1 (2004), 13.
[20] points out, the creation of classroom environments that [9] Hinrichs, U. and Carpendale, S. Gestures in the wild:
enhance students expression and selection opportunities provides studying multi-touch gesture sequences on interactive
a proper atmosphere towards helping children to learn at their own tabletop exhibits. Proc. of CHI’11, 3023–3032.
pace. A starting point could be taking advance of (1) the [10] Hourcade, J.P. Interaction Design and Children.
portability of tablets that allow children and teachers to use Foundations and Trends® in Human-Computer Interaction
various locations in the classroom enabling the creativity and 1, 4 (2007), 277–392.
collaboration of small groups of children; and (2) the more [11] Ingram, A., Wang, X., and Ribarsky, W. Towards the
sophisticated interface of tablets that allows both individual and establishment of a framework for intuitive multi-touch
mutual interaction between two or more users [25]. interaction design. Proc. of AVI’12, 66–73.
[12] Johnson, L., Adams, S., and Cummins, M. The NMC
4. CONCLUSIONS Horizon Report: 2012 K-12. The New Media Consortium,
The contributions of this paper are twofold; firstly, a review of the Austin, Texas, 2012.
state of the art of multi-touch technology for pre-kindergarten [13] Kammer, D., Dang, R., Steinhauf, J., and Groh, R.
children was made. The literature suggests that the use of this Investigating interaction with tabletops in kindergarten
technology is an ideal way to develop applications for these very environments. Proc. of IDC’14, 57–66.
young users. In addition, the presented studies show that pre- [14] Knoche, H., Rasmussen, N.A., and Boldreel, K. Do
kindergarten children are able to perform more touch gestures Interactions Speak Louder than Words ? Dialogic Reading
than could be expected when analyzing existing commercial of an Interactive Tablet-based E-book with Children
applications. On the other hand, pre-kindergarten children are between 16 Months and Three Years of Age. Proc. of
developing their motor skills and, hence, in order to use the multi- IDC'14, 285–288.
touch technology, designers should be aware of the actual [15] Kremer, K.E. Conducting Game User Experience Research
capabilities of children when developing interfaces for them. The with Preschoolers. Workshop on Games User Research:
second contribution is a collection of future challenges to be faced practice, methods, and applications (collocated to CHI’12).
in the near future when building multi-touch technology for [16] Nacher, V., Jaen, J., Catala, A., Navarro, E., and Gonzalez,
preschool children. These challenges are related to P. Improving Pre-Kindergarten Touch Performance. Proc.
communicability, adaptability and usability of multi-touch of ITS ’14, 163–166.
applications designed for these challenging users. We hope that [17] Nacher, V., Jaen, J., and Catala, A. Exploring Visual Cues
the usability studies and design guidelines compiled from these for Intuitive Communicability of Touch Gestures to Pre-
works and the future challenges discussed in this paper will allow kindergarten Children. Proc. of ITS ’14, 159–162.
designers of future applications for pre-kindergarten children to [18] Nacher, V., Jaen, J., Navarro, E., Catala, A., and González,
fully exploit the potential of multi-touch technology to support P. Multi-touch gestures for pre-kindergarten children.
their cognitive and motor development. International Journal of Human-Computer Studies 73,
(2015), 37–51.
5. ACKNOWLEDGMENTS [19] Piaget, J.The Child and Reality. Grossman, New York,
Work supported by the MINECO (grants TIN2010-20488 and 1973.
TIN2014-60077-R) and from GVA (ACIF/2015/075). [20] Rushton, S. and Juola-Rushton, A. Classroom Learning
Environment, Brain Research and The No Child Left
6. REFERENCES Behind Initiative: 6 years Later. Early Childhood Education
[1] Abdul Aziz, N.A., Batmaz, F., Stone, R., and Paul, C. Journal 36, 1 (2008), 87–92.
Selection of touch gestures for children’s applications. Proc. [21] Shneiderman, B., Plaisant, C., Cohen, M., and Jacobs, S.
of SIC'13, 721–726. Designing the User Interface: Strategies for Effective
[2] Abdul Aziz, N.A., Mat, N.S., Batmaz, F., Stone, R., and Human-Computer Interaction. Prentice Hall, 2009.
Paul, C. Selection of Touch Gestures for Children’s [22] Shuler, C. iLearn II: An Analysis of the Education Category
Applications: Repeated Experiment to Increase Reliability. of the iTunes App Store. The Joan Ganz Cooney Center at
International Journal of Advanced Computer Science and Sesame Workshop, New York, 2012.
Applications 5, 4 (2014), 97–102. [23] Smith, S.P., Burd, E., and Rick, J. Developing, evaluating
[3] Baloian, N., Pino, J. a., and Vargas, R. Tablet gestures as a and deploying multi-touch systems. International Journal of
motivating factor for learning. Proc. of ChileCHI'13, 98– Human-Computer Studies 70, 10 (2012), 653–656.
103. [24] Vatavu, R., Cramariuc, G., and Schipor, D.M. Touch
[4] Bebell, D., Dorris, S., and Muir, M. Emerging Results From interaction for children aged 3 to 6 years : Experimental fi
The Nation’s First Kindergarten Implementation of iPads. ndings and relationship to motor skills. International
Auburn, 2012. Journal of Human-Computer Studies 74, (2015), 54–76.
[5] Buxton, B. Multi-touch systems that I have known and [25] Wakefield, J. and Smith, D. From Socrates to Satellites:
loved. 2013. iPad Learning in an Undergraduate Course. Creative
http://billbuxton.com/multitouchOverview.html. Education 03, 05 (2012), 643–648.
[6] Chiong, C. and Shuler, C. Learning: Is there an app for [26] Wolock, E., Ann Orr, E.D., and Buckleitner, W. Child
that? Investigations of young children’s usage and learning development 101 for the developers of interactive media.
with mobile devices and apps. New York, 2010. Active Learning Associates, Inc., 2006.
[7] Common Sense Media. Zero to Eight: Childrens Media Use [27] Zaranis, N., Kalogiannakis, M., and Papadakis, S. Using
in America 2013. 2013. Mobile Devices for Teaching Realistic Mathematics in

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 52 |
 Multi-touch Technology in Early Childhood: Current Trends and Future Challenges
V. Nacher | J. Jaen
Kindergarten Education. Creative Education 04, 07 (2013),
1–10.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 53 |
 Vibrotactile Vest and The Humming Wall: “I like the hand down my spine”
A. Morrison | C. Manresa-Yee | H. Knoche

Vibrotactile Vest and The Humming Wall: “I like the hand

down my spine”
Ann Morrison Cristina Manresa-Yee Hendrik Knoche
Department of Architecture, Design Department of Mathematics Department of Architecture, Design
and Media Technology, and Computer Science and Media Technology,
Aalborg University Universitat de les Illes Balears Aalborg University
Rendsburggade 14, Crta. Valldemossa km 7.5 Rendsburggade 14,
9000, Aalborg, DENMARK 07122, Palma, SPAIN 9000, Aalborg, DENMARK
[email protected] [email protected] [email protected]

ABSTRACT use and is placed in Utzon Park (Aalborg, Denmark), with seating
Vibrotactile information can be used to elicit sensations and at each end on both sides (that also act as wind protection).
encourage particular user body movements. We designed a For the vest wearer though, the interactive experience is unique.
vibrotactile vest with physiological monitoring that interacts with Several actions performed on the wall (knocks, a swipe action
a vibroacoustic urban environment, The Humming Wall. In this downwards, upwards; sideways left to right, right to left) are
paper, we describe the first field trial with the system held over a replicated in actuators vibrating in the vest. Further, people feel,
5-week period in an urban park. We depict the participants’ see and hear their own heartbeat in one zone of the wall and feel
experience, engagement and impressions while wearing the and hear their own breath in another.
vibrotactile vest and interacting with the wall. We contribute with
positive responses to novel interactions between the responsive The aim of this work is to describe participant experience and
environment and the vibrotactile vest, with a system designed to engagement with the paired tactile interaction between vest and
calm, activate, guide and warn the participants. wall, evaluated in field trials held over a 5-week period in an
urban park with 39 people. As this was the first execution of these
Categories and Subject Descriptors paired vibro-tactile-acoustic systems, we kept the interaction
H.5.2. Information interfaces and presentation: User interfaces. paradigms relatively simple.

General Terms The paper is structured as follows: Section 2 compiles works on

Design, Human Factors, Measurement vibrotactile interfaces related to activation, navigation and
calming purposes as well as responsive environments. Section 3
Keywords describes the design of the vest and The Humming Wall. Section 4
Wearable computing; Vibrotactile; The Humming Wall; Audio presents the evaluation and section 5, the results of the field trial.
vibrations; Haptic patterns; Sensations; Vibroacoustics Finally, we summarise the findings and discuss their implications.

1. INTRODUCTION
Touch is highly connected to our emotions [1] and vibrotactile
cues can cause different sensations (e.g. calming, pain) depending
on factors such as frequency or rhythm [2, 3]. Further, touch is a
powerful communication channel that is not used as frequently as
sight or hearing. By structuring vibrotactile patterns, we can build
a vibrotactile language to convey information for different
applications such as enhancing communication [4] or helping to
perceive the interlocutor’s facial expression [5]. We are especially
interested in those works using vibrotactile cues working with
whole body navigational cues and instigating body motion as well
as vibrations used for activation/calming purposes.
Working with a kinesiologist-neurophysiologist, we designed Figure 1. General public at the wall, hearing and feeling
novel placement and patterns for vibrotactile actuators on the the vibration and playing with the wall
body to activate, guide, warn and to emulate natural touch
sensations such as stroking up or down the back to relax a 2. RELATED WORK
person—‘there, there, it’s okay’.
Work related to the vibrotactile vest and The Humming Wall
In order to enrich and yield a positive experience for the vest includes (i) vibrotactile languages working with whole body
wearer, we built a responsive urban environment The Humming navigational cues and instigating body motion, (ii) vibrotactile
Wall, designed to enrich participants experience with the interfaces initiating activating/calming sensations and (iii)
environment and invite open-ended, embodied and playful responsive vibroacoustic environments in public spaces where
interactions [6-8]. The Humming Wall is a 12 x 3.5 x 2.7 meter visual is not the privileged sense. As we cannot cover all of these
(long x wide x high) vibroacoustic wall (see Fig.1) with a calming in detail here, we cover an abridged version.
low-level hum. The curving wall is designed for general public

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 54 |
 Vibrotactile Vest and The Humming Wall: “I like the hand down my spine”
A. Morrison | C. Manresa-Yee | H. Knoche
In order to motivate and improve motor skills, Spelmezan et al. actuators are operated in overlapping patterns to provide different
[9] developed a tactile language to assist athletes to improve their haptic synesthesia sensations such as sense of movement, shiver
motor skills in snowboarding. In this line of work, McDaniel et al. and states of activation and/or calming as well as providing
[10] proposed a framework, MOVeMENT (Mapping Of navigational cues. We explored the positioning and combinations
Vibrations to moveMENT), that used vibrotactile stimulation to of the actuators with a kinesiologist trained in neurophysiology.
b
teach motor skills and to motivate fundamental movements
(flexion, extension, abduction, adduction and rotation). Further, c
Rosenthal et al. [11] presented a belt to provide vibrotactile
spatio-temporal patterns to teach choreographed dance. Morelli et
al [12] and Yuan and Folmer [13] also used vibrotactile cues to
inform users with sight loss or blindness to carry out certain
movements to play videogames.
Several research projects use vibrotactile cues for navigation.
Tsukada et al. [14] designed a belt that transmits directional
information. Van Erp et al. [15] also tested a directional belt but
with different distance-coding schemes. Other wearable supports
have been used, such as the Vibrotactile Glove [16] that offered
users with disabilities information by indicating warning signals,
spatial representation, and directional guidance using a glove.
Vibrations have been used for therapies to generate soothing and
calming sensations to users. Vaucelle et al. [17] presented Touch Figure 2. The vibrotactile vest: adjustable harness (a)
Me, a flexible vibrotactile motor array in a soft enclosure that can front and (b) back and (c) outer shell and skirt-apron.
be applied to large areas of the body and remotely actuated by a
care-giver through a remote switch array to provide soothing
touch simulation over an entire part of the body. Paredes and For the initial patterns, 29 actuators were used (see Fig. 3). These
Chan [18] employed two vibrotactile motors in a bracelet that patterns try to emulate the hands on work that a kinesiologist-
stimulates acupressure points in the wrists and the chest (when the neurophysiologist does in activating sequential points of the body.
wrist is held to the sternum) to reduce stress. And using the same In addition, we emulated the natural touching we give each other
bracelet, they coached participants to breathe according to well- for calming-comforting (e.g. stroking the back to calm or comfort
known deep-breathing techniques. Arafsha et al [19] designed a a person), guidance-navigation (e.g. placing hands on nape of
haptic jacket to enhance comfort with 6 affective haptic back and shoulder as if to support and guide an elder) or
components: chest and neck vibration, neck warmth, heartbeat information-instruction (e.g. stopping the body with pulses to the
simulation, arms vibration and shivering. solar plexus). The reader is directed to [22] for a more detailed
description of the patterns.
Prior work on responsive vibroacoustic environments in public
spaces for developing embodied, multi-user interactive
experiences include works such as MEDIATE, an environment
that uses real time visual, aural and vibrotactile stimuli both for
sensing and giving feedback for education with autistic children
[20] or ListenTree, an audio-haptic display embedded in a tree in
a natural environment. A visitor notices a faint sound that appears
to emerge from a tree, and feels a slight vibration under their feet
as they approach. Future versions will add touch input to achieve
a more responsive installation [21].

3. THE SYSTEMS
There are two systems. First, a vibrotactile vest which produces
vest-to-human interactions as felt sensations for the wearer.
Second, The Humming Wall produces two types of human-to-
Wall-to-human interactions, one type between the vest wearer and
the wall and the other enables interaction between non-vest Figure 3. Placement of actuators
wearers-to-vest wearers via the wall and also interactions for non-
vest wearers with the wall itself. In Table 1, examples of patterns are described. Ten patterns were
designed:
3.1 Vibrotactile vest • Calming: Up, Down,
The vest is made of two layers, an inner layer consisting of one-
size-fits all adjustable harness (see Fig. 2a, 2b) and an outer layer • FeelGood: WaistLeftToRight, WaistRightToLeft,
consisting of an enclosing vest (See Fig. 2c). The harness houses MidFrontToBack
32 actuators, moveable to ensure they are placed exactly on each
different shaped body. The lower harness fits around the legs, to • Activation: Shiver, Tarzan
safeguard that the harness stays pulled down and keeps the • Navigation: TurnLeft, TurnRight
actuators in place while the participants move around. The

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 55 |
 Vibrotactile Vest and The Humming Wall: “I like the hand down my spine”
A. Morrison | C. Manresa-Yee | H. Knoche
• Warning: Stop d) Swipe—3 panels that respond to up, down, left and right
gestures. These played as audio and vibration at the wall with
The Zephyr BioHarness 3 [23] is integrated into the harness and a corresponding gesture played on the vest—e.g. swiped up or
vest system to read the physiological state of the wearer in real- down the back and swiped across the waist—left to right or
time, specifically their heartbeat and breath rate. Two custom- right to left—depending on direction of gesture (Fig. 4d).
made electronic boards are also included in the wearable system:
one controls communication and the other powers and controls the e) Knock & Swipe—5 panels that respond to knock and swipe at
actuators. the wall and on the vest. The swipe gestures run along the 5
sections with audio, vibration and lights. The knocks are
Table 1. Examples of patterns relayed alternatively to all actuators and felt all over the body.
Actions Pattern Knocking on two panels activated two actuators at once,
knocking on three activated three actuators (Fig. 4b).
Activating:
4x(FcL, FcR), pause, FcM
Tarzan The wall was placed in an outside environment at Utzon Park
Calming: (BsL6, BsR6), (BsL5, BsR5), (BsL4, BsR4), (Aalborg, Denmark) between two of Utzon’s architectural
down (BsL3, BsR3), (BsL2, BsR2), (BsL1, BsR1) structures, the Utzon Center and Utzon student housing, and
Navigating: Bw7, (BhR1, BhR2), (Fw3, Fw2), (Fw2, Fw1), encompasses Utzon’s aesthetics and design principles. The design
turn left (Fw1, Bw8) includes harmonious modular repeating segments with internal
Alert: Stop (FspL, FspR) voids large enough for arms to access electronics and cables to
enable interactivity, serving both the aesthetics and pragmatics.

The outer shell, made in 3 adjustable sizes, is a padded layered

stretchable vest designed to keep the actuators tightly in place
against the bowed areas of the body, especially the lower back and
chest, so the vibrations are evenly felt in all areas. We offered an
adjustable skirt-apron for the sake of modesty and/or aesthetics,
particularly for those wearing dresses.

3.2 The Humming Wall

A 12m long, 3.5m wide and 2.7m high was cut from EPS150
foam blocks with a hot wire and glued together to form 11
segments. Each segment is coated with a PolyUrea coating for
strength and durability. The wall is fitted with an array of 2-
channel DMX controlled warm and cold white lights. These lights
are used for ambient and indicative visual effects on the wall. An
Figure 4. a) Heartbeat: Vest transmits to Wall; Audio,
array of tactile transducers placed in the wall generates different
Vibration and Light emulate vest wearers heartbeat at
haptic sensations, audio cues and haptic feedback. Each segment
wall. (b) Knock and Swipe: Wall transmits to Vest; Audio,
has an 80W Bass-Shaker mounted with a frequency range suitable
Vibration and Light respond to knock and swipe. (c)
for haptic and audio feedback, and amplified so that each segment
Knock: Wall to Vest (d) Swipe: Wall to Vest (e) Breath
functions as a vibrating speaker. The Humming Wall has its own
Rate: Vest to wall
baseline vibration and audible hum. A combination of piezo- and
capacitive electronics to read and analyse touches, enables the
differentiation of knocking (or tapping) and swiping gestures. 4. EVALUATION
To evaluate the tactile interaction, people were invited to
The wall is divided into five zones that interact with different
participate in pairs asking a familiar, friend or partner to take part
input from the data registered in the vest or from gestures done on
with them, as early tests proved the interaction was best done in
the wall and trigger different output in the wall and on the vest
pairs and personal. The trial was held over an extended period of
simultaneously. In the two seating areas, the physiological data
5 weeks from June 23-July 25, 2014. The duration of the trials per
from the vest is relayed to the wall:
pair took between 1.5 to 4 hours, averaging 2.5 hours per pair.
a) Heartbeat—displayed as audio, vibration and light (at the The initial programmed 3-week period proved too short given
same beat) in the wall (Fig. 4a). emergent circumstances and the programmed 2 hours for each set
of paired participants was not enough time as we constantly had to
b) Breath rate—displayed as audio and vibration in the wall push-rush the participants to move onto the next zone.
(Fig. 4e). Subsequently, from participant 16 onwards, participants were told
Then, there are three zones where sensations relayed at the wall there were 7 activities and they could either choose their own
are transferred to the vest: timing or we could prompt. All chose to set their own timing.

c) Knock—3 panels respond to knocking with audio and 4.1 The Participants
vibration in the wall and knocks are felt in individual We enlisted 39 volunteers with ages from 12 to 65 years (average
actuators of the vest. Right panel knocks are transferred to the age 39), 20 females and 19 males. 19 participated in mixed gender
right side of the body—front and waist; middle panel groups, 11 in female/female and 9 in male/male groups (uneven
transferred to the back; and Left panel transferred to left side numbers are due to only one participant from a pair wearing the
of the body—front and waist (Fig. 4c). vest). Most people took part with close or good friends (11), their

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 56 |
 Vibrotactile Vest and The Humming Wall: “I like the hand down my spine”
A. Morrison | C. Manresa-Yee | H. Knoche
partners (10), colleagues (10), family members (5), or social cut into manageable chunks and coded focusing on where
friends or acquaintances (3). 22 self reported basic or average IT activities and interactions occurred. We transcribed the talk aloud
skills with 17 advanced or above. 28 spent on average more than responses of the participants to each vibration and coded them
20 hours on a computer each week and 12 enjoyed playing a numerically. For each zone we measured the amount of time the
musical instrument regularly. 31 had tertiary level qualifications participants spent, coded to what degree they explored the
with 14 knowledgeable about wearable technology. interrelation of knocking and swiping on the wall and
corresponding sensations for the vest wearer.
4.2 Procedure and venues
Each pair of participants was accompanied throughout the trial by We used 5 level Likert-type scales (from completely disagree to
two researchers and all sessions were videoed. The trial was completely agree) to measure items adopted from IMI, FSS and
divided in 4 steps: MEC-SPQ. The levels of agreement were mapped to values from
1 to 5. The semantic differential scale items featured 5 numerical
(1) Utzon Library Fitting: fitting was done in a side room of the levels with the differential labels at the extreme values (e.g.
Utzon Library with a change screen for privacy. Demographic Impersonal 1,2,3,4,5 Personal).
info was collected and the BioHarness, harness and vest was fitted
and tested to place correctly actuators on the body (15-35 mins). 5. RESULTS
In this section we report on findings from questionnaires, vest
(2) Utzon Park Patterns: participants were guided by a researcher activity logging, video footage analysis, time spent at each section
and introduced to the ten vibration patterns 3 times while asked to of the wall and semi-structured interviews.
talk aloud their responses to the sensations. Then participants and
researchers stopped, each vibration pattern was played again and 5.1 Questionnaires
responses discussed again (10-15 mins). In Table 2 we find a summary of the results for a subset of the
(3) The Humming Wall Interaction: participants interacted with 5 questions. Most responses show that users enjoyed interacting
zones of activity (with 2 of the zones repeat visits = 7 total with The Humming Wall. Participants were relaxed when carrying
activities) at The Humming Wall. The participants moved out the tasks (Q19) and they found it pleasant to complete the
clockwise around the wall from one zone to the next. The different activities at the wall (Q2, Q16). They did not report
activities were: (1) heartbeat, (2) knock, (3) breath rate, (4) swipe discomfort associated with the vibrations from the wall (Q4), and
(5) 2nd visit to breath, (6) knock & swipe and (7) 2nd visit to most users felt that they lost track of time during the trial (Q15).
heartbeat (20-90 mins). When responding to the items regarding the vest, they did not feel
much discomfort associated with the vibrations in the vest (Q4),
(4) Utzon Library Evaluation: participants took off the vest, filled they felt it was possible to be active in the surroundings (Q6) and
in questionnaires and undertook semi-structured interviews (15-40 the vest vibrations made it easier to understand the tasks and
mins). responses (Q17). In general users stated that they were
concentrating on the sensations in the vest (Q1) and thought
4.3 Data Collection and Processing whether these sensations could be useful for other activities (Q7).
Data was gathered using quantitative and qualitative methods. Most participants found that some sensations held greater
Before the trial, the participants filled in informed consent forms meaning than others (Q3).
and a demographic questionnaire with questions on fitness levels,
their general IT level and experience with vibrotactile technology, Results summarising the overall experience with the vest and the
embodied interaction, large public displays and playing musical wall are shown in Fig. 5. Participants reported to be relaxed and
instruments. Activity data was logged for each participant from calmed even though they reported to be active as well. Further,
the actuators and from the BioHarness, the heartbeat and breath users felt that it was a personal experience, but at the same time,
rate. The actions done on the wall were also registered, knocking they felt sociable at the installation.
and swiping, together with the segment and direction and the time When crossing results of the questionnaires with the demographic
spent in each zone. data using non-parametric Mann-Whitney tests we found the
After walking around the wall, participants completed shortened following statistically significant differences. Males agreed
adapted versions of questions from MEC Spatial Presence significantly more than females that some sensations made more
Questionnaire (MEC-SPQ) [24], Flow State Scale (FSS) [25] and sense than others but enjoyed less learning how to do the tasks
Intrinsic Motivation Inventory (IMI) [26], to gauge reactions to suggested by the sensations. They felt less warm and personal
the sensations in the vest and the interaction with The Humming than the women. Participants who exercised 2 or more times per
Wall. The questionnaires comprised 21 Likert-type items to week reported significantly warmer and more relaxed experiences
analyse and cross-check users' perceptions. For Presence, we than those exercising less. Participants with average and lower IT
measured concentration, errors, activated thinking, and imagining knowledge found the experience significantly: easier to respond to
space. For IMI, we measured interest/enjoyment, perceived vibrations, found their understanding of the sensations improving
competence, pressure/tension, and effort/importance. For Flow, more as the trial progressed and felt warmer and more sociable
we measured challenge-skills balance, goals, concentration on than the participants with high IT knowledge. Those participants
task, and sense of control. The overall experience was measured with more experience at large displays agreed significantly more
through 10 semantic differential scale items (See Table 2). Lastly, that some of the vest vibrations were uncomfortable when
each participant described their experience, highlighting chosen compared to the less experienced. Musical participants who
aspects in semi-structured recorded interviews. enjoyed playing an instrument reported a different experience
from participants who did not. Non-instrument players had a
Data post-processing tasks included transcribing user interviews significantly more sociable and open experience. We found no
and translating them into English for further analysis. Videos were significant differences between those with experience with

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 57 |
 Vibrotactile Vest and The Humming Wall: “I like the hand down my spine”
A. Morrison | C. Manresa-Yee | H. Knoche
embodied games, e.g. Wii, compared to those with little or none.
Those less knowledgeable with vibrotactile technology enjoyed
learning how to do the tasks suggested by the sensation
significantly more and had a more personal experience than the
savvy ones. Those knowledgeable in wearable technology
reported a significantly more relaxed experience compared to
those with little or no knowledge in this area.

Figure 6: Comparative Times (guided or not) for Wall

Figure 5: Overall experience results for 39 participants.

Semantic differential items

5.2 Logging of wall activity at zones

As commented before, from participant 16 onwards, participants
could choose their own pace. As a consequence, all interaction Figure 7: Comparative Time spent in Physiological and
durations from that point at the wall lengthened with an average Knocking and Swiping zones (guided or not)
of 9.8 extra minutes per participant when interacting at the wall.
In Fig. 6, the time spent at each zone is shown. The larger The total time spent in the physiology zones was significantly
differences can be seen at Heartbeat zone 1st visit, Breath rate predicted by the following three factors in the regression
zone 1st visit and Knock zone when participants could decide on analysis: participants spent more time in those zones when they
their timings. Participants spent more time in the physiology could advance at their own pace, were less knowledgeable about
zones when they decided their own pace (54.8%) than in the vibrotactile technology and if they had tried to find out the
Knocking and Swiping zones (45.2%) (Fig. 7). On average, mapping of the knocking to the vest. These three factors
participants deciding on their own timings stayed almost 6 extra accounted for 58% of the variance in time spent in the physiology
minutes in the physiology zones and around 4 minutes more in the zones.
action zones comparing with participants that were guided.
5.2.1 Knocking
In addition and regardless of this impact, there were two 56% of the participants tried to map knocking to the vest wearer’s
significant predictors of the time spent in the knocking and body. That is, they assumed a direct mapping between body and
swiping zones: participants spent more there, the stronger their wall, e.g. that knocking higher would map closer to the head,
relationship was with their pair and if they tried to find out the knocking in the middle section would trigger actuators around the
mapping of the knocking to the vest. Together these factors waist and lower would be felt in the lower back (see Fig. 8).
accounted for 32% of the variance in time spent in these zones.
5.2.2 Swiping
The main swipe movements were up, down, left and right.
However, participants also experimented with other swiping
directions. 64% of the participants made waving-swipe-circular
movements with one or two hands, as if massaging a back when
swiping at the wall (See Fig. 9).

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 58 |
 Vibrotactile Vest and The Humming Wall: “I like the hand down my spine”
A. Morrison | C. Manresa-Yee | H. Knoche
feeling and it was stressful… Later I got used to it’ and ‘a bit
scary, intimidating at start, but then more natural when I
understood how it felt’. Some tried to compare to other
experiences: ‘front vibrations felt like standing in room with loud
music, connection vibrations going one side to another like being
in the ocean, and then stroking hand down back’. Others were
uncertain how to discuss it, especially the whole body sensation:
Figure 8. (a) Knocking at lower wall, (b) at mid wall. (c) at ‘haven’t felt before… single vibrations from mobile… but, not all
high wall and looking at partner to see response. over body and flowing… not sure how to describe… is new’. And
discussed: ‘all immersed, when all over body—could feel like me
vibrating all over, especially ones on stomach’. Others talked
about it acting like a real touch: ‘Putting up the same emotion as
touch—bringing out the same sense of well-being or happiness as
when somebody touches or pats or hugs you… the hug [Shiver,
Activating] was very surprising’.

Figure 9: Participant is making circular wave style swipes Some of the patterns made people ticklish, especially around the
waist [Waist, FeelGood], and at the sides: ‘It was like somebody
was tickling me’. And ‘I like the hand down my spine’ [Back,
5.3 Vibration Patterns Calming] or ‘The tickling on the side [Waist, FeelGood]… gave
The talk aloud responses to the 10 patterns that were registered in me goosebumps from the outside—like you didn't have the
the Utzon Park Patterns stage were coded numerically, i.e. a sensation but got the goosebumps anyway’. For most, the waist
response to each individual pattern was judged as either negative and the back sensations were the most pleasant: ‘I liked the ones
(-1), neutral (0) or positive (1) - see Fig. 10. We used the around the waist the most’. And ‘the waist was very pleasant and
numerically coded responses to the 10 patterns in a non- it tickles a bit’ [Waist, FeelGood] or ‘Those on the back were
parametric Friedman test rendering a Chi-square value of 51.9, pleasant… nothing was uncomfortable… it was all very gentle’
which was significant (p<.001). Post-hoc pairwise comparisons [Back, Calming] and ‘Preferred up and down—lifting you up with
showed that Up, Down, and WaistRightToLeft all were up swipe…Up more than down, felt more natural for me’. Yet
significantly more positively evaluated than all other patterns, there was differences in interpretations and preferences between
whereas the Tarzan-activation pattern was significantly evaluated elements of the patterns: ‘Solar plexus [Front, Warning], not so
more negatively than all others (see Fig. 10). bad once got used to it—at first didn’t like it—was heavier—then
saw it is a warning of don't do that’, whereas for another ‘Front—
not as pleasant—not as natural—vibrating there’. But others
preferred a different set of patterns again: ‘The feeling in the chest
was best, feel it better also at lower back---very nice’ and
different intensities ‘All could be stronger’ and ‘Tarzan [Front,
Activating], was pleasant. Just as pleasant for waist etc… Some
also suggested certain movements took longer to learn: ‘turn left
or right, took time to get more used to them–needed to do again a
few times at first—and was harder to figure out at the beginning,
but become almost normal quickly’.
The Humming Wall also provided new sensations with responses
such as: ‘Knew touching wall, but no idea how it would feel,
exciting to know how it felt—big difference to having the feeling
of it all over body and not just on fingers or one part.’ Many
participants explored mapping the body to the wall: ‘I thought I
was able to knock high and feel high in body, middle to middle
Figure 10: Means of coded responses to the vibration and once or twice—there was a specific connection between the
patterns with error bars (standard error of the mean) two… it became difficult to find the good spots like you do with
massage’. Sensations felt better in certain spots, so participants
5.4 Semi-structured interviews kept seeking a logic, often ordering a partner to repeat or go back
In the semi-structured interviews we asked a series of questions, to find a sensation again.
for example ‘How did you find the sensations in the vest?’ ‘Were Many found the wall helped them relax with participants often
any unpleasant?’ in order to engage and to get participants talking referring to the wall as: ‘therapeutic’ or ‘good to help take care of
about their experience and express their views in their own terms. health’ and ‘Love the wall, soothing and relaxing listening to
People were generally very forthcoming and the interviews lively. humming sounds and feeling vibrations---also to play it when sat
Participants often describe the interactions as interesting, fun, there—calming humming sound—like it in my couch’. The wall
unusual or new and different. For example, participants described appeared to be a safe place where people felt protected and even
the experience as ‘Funny - different –a new experience’; ‘I like to though in a public space, somehow private. ‘Did not mind to hear
try new things, so it was new to me’. Others took some time to my physiology in a public space… Enjoyed to sit in each others
adjust before they felt comfortable: ‘In the beginning it was a new [physiology]…. tried to breathe in the same rhythm and heart

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 59 |
 Vibrotactile Vest and The Humming Wall: “I like the hand down my spine”
A. Morrison | C. Manresa-Yee | H. Knoche
beats also’. Most participants seemed to relax well on the wall massaging a body. The time spent in the physiology zones was
(see Fig. 11). longer when we offered participants to walk around the wall at
their own pace; they were less knowledgeable with vibrotactile
technology and if they had explored the mapping of the actions
done in the wall to the vest.
From the interviews we found that people had very diverse
reactions to the patterns: some requested more activating (Tarzan
or Shiver) but most preferred the calming sensations (waist and
back). Overall, for many participants this was a new experience,
Figure 11. Participants relaxing and listening to their for some daunting at the beginning, others uncertain how to ‘name
heartbeat at the end of the interaction sequence. it’, but a whole body experience and for most an activating and
enjoyable one. There was an obvious connection to the wall as a
5.5 Observations whole body and playful experience and one that had put their
The participants enjoyed exploring and playing with the paired inside body available to the wall in a public space.
vibrotactile interaction. In the Breath rate zone, they played
around with their breathing by holding their breath or doing deep
7. DISCUSSION AND CONCLUSIONS
slow breathing. In the Heartbeat zone, they followed the We designed a vibrotactile vest with physiological monitoring that
suggestion of jumping to increase their heartbeat and then sitting interacts with a vibroacoustic urban environment, The Humming
down to ‘feel-hear’ the heartbeat rhythm slowing down. In the Wall. Different vibrotactile patterns were designed together with a
Swipe and Knock zones, participants played their own little kinesiologist-neurophysiologist for the vest to emulate calming
musical rhythms, running up and down, or trying to touch 4 and activating sensations and to guide or warn the vest wearer.
panels at once and/or generally experimenting with what was Further, actions on the wall are replicated on the vest, and in two
possible and playing together at the wall. areas of the wall the participants ‘feel’ (viboracoustically) and
hear their heartbeat and breath rate. We conducted a field trial
When interviewing the participants, we noted that for many of held over a 5-week period in Utzon Park, a public space in
them, it was difficult to describe the sensation experienced. Aalborg, Denmark. In this work we described the experience of
Additionally, sensations in more than just one area of the body at 39 participants interacting with the vest and the responsive
the same time generally confused people. environment and their engagement with the system.

6. SUMMARY OF FINDINGS The participants had a novel, unusual and generally activating, yet
The results of the first field trial with participants wearing the calming experience. It was very physical, tangible, embodied—
vibrotactile vest and interacting with The Humming Wall were not that they did particularly use those words—but they talked
very positive. Answers to the questionnaires denoted that about feelings of well-being, hunger, goose bumps on the skin and
participants found the experience enjoyable and pleasant. They touch as if it was real enough—‘I like the hand down my spine’.
were highly engaged and concentrated with this novel interaction Overall, they had a very personal experience and people were
with the vest and the environment, as they did not notice the ‘touched’ and often moved by it.
passing of time. In addition, they were activated, motivated but Allowing participants to control their own timing increased the
relaxed and understood what was required of them. As well, time spent and potentially the quality of ownership of the
participants found this a personal, warm, sensitive, sociable, experience. Participants built proprietorship from their speculative
active and open experience. play and experimentation—despite their personal heartbeat and
Crossing demographic information with the questionnaires, we breath rate being made publicly available in the public space.
observed that women reported to have a more personal and warm For this first implementation, delays and unforeseen factors meant
experience than men. Fitter people and particularly those with we pared back and simplified many of our initial planned
average or lower IT knowledge had the best experiences—the interactions in the interests of robustness. Our concerns then were
latter even understanding the sensations better as the trial that the experience would hold a short novelty factor and that
progressed. Further, participants less familiar with vibrotactile there would be ‘not enough to do’—breath rate, heartbeat, swipe,
technology had a more personal experience and enjoyed learning knock—people would be finished in 5 minutes! In addition, we
how to do the tasks suggested by the sensations more than experts had deliberately worked without the visual, (beyond minimal
in this domain. responsive lighting—which many did not notice) in order to
Regarding the patterns and the sensations they elicited from the investigate the potential of vibrotactile interaction. Given the
vest wearer, we found, particularly for males, that some sensations newness for many of our participants, their warm, social
made more sense than others, and this was confirmed across responses, and that they wanted to stay much longer than we had
genders in the video logging and semi-structured interviews. envisaged—or even allowed time for—anything more
Participants responded more positively to Up, Down, complicated could have undermined this initial investigation.
WaistLeftToRight, WaistRightToLeft, (calming patterns) than to We found the nature of the relationships—the intimacy—that the
e.g. Shiver, Tarzan (activating) or Turn Left (navigation) and Stop wall enabled was surprising. Pairs who play well generally found
(warning). a good place to play and discover at the wall. The experience was
For the participants’ relation to The Humming Wall interaction, abstract, there were no real tasks, no game as such, rather a place
most participants experimented with mapping the knocking to the to explore—which people did and made their own fun.
vest wearer’s body and with different types of swipes—emulating

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 60 |
 Vibrotactile Vest and The Humming Wall: “I like the hand down my spine”
A. Morrison | C. Manresa-Yee | H. Knoche
As future work, we will redevelop some of the internal electronics mapping vibrotactile stimulations to fundamental body
and place The Humming Wall in a public environment to conduct movements. In Proc. HAVE’10, IEEE, 13-18.
a longitudinal study including general use. We found a wide array [11] Rosenthal, J., Edwards, N., Villanueva, D., Krishna, S.,
of personal preferences and responses to the sensations in the McDaniel, T., and Panchanathan, S. 2011. Design,
vibrotactile patterns and we need to address these. Calming implementation, and case study of a pragmatic vibrotactile
sensations were clearly more popular and easily understood than belt. IEEE Trans. Instrum. Meas., 60, 1, 114–125.
the activating, warning or guiding ones—but understandably the
later would not necessarily be designed to be as pleasant. People [12] Morelli, T., Foley, J., Columna, L., Lieberman, L., and
were easily confused by sensations in more than one area of the Folmer, E. 2010. VI-Tennis: a vibrotactile/audio exergame
body, so we have work to do on uncovering best placements of for players who are visually impaired. In Proc. FDG '10.
actuators for guiding sensations on e.g. the shoulder and nape of ACM, New York, NY, USA, 147-154.
the back that are more easily understood—yet this is something [13] Yuan, B., and Folmer, E. 2008. Blind hero: enabling guitar
we see and use in everyday life when guiding elders or children. hero for the visually impaired. In Proc. Assets '08, ACM,
Further, it would be interesting to evaluate the system with blind New York, NY, USA, 169-176.
(or blindfolded) users to analyse their perceived sensations when
[14] Tsukada, K., and Yasumura, M. 2004. ActiveBelt: Belt-Type
using the system.
Wearable Tactile Display for Directional Navigation. In
This work demonstrates potential for responsive urban Proc. Ubicomp’04. LNCS 3205, Springer, 384-399.
environments that are nurturing for and assistive to the [15] Van Erp, J. B. F., Van Veen, H. A. H. C., Jansen, C., and
physiological states of their inhabitants. Dobbins, T., 2005. Waypoint navigation with a vibrotactile
waist belt. ACM Trans. Appl. Percept. 2, 2, 106-117.
8. ACKNOWLEDGMENTS
We thank the volunteers for the field trials and Aalborg Kommune [16] Uchiyama, H., Covington, M. A. and Potter, W. D. 2008.
for support. This work is supported by the EU funded project Vibrotactile Glove guidance for semi-autonomous
CultAR (FP7-ICT-2011-9 601139), and mobility grant wheelchair operations. In Proc. ACM-SE 46. ACM, New
CAS12/00199, Programa José Castillejo granted by the Spanish York, NY, USA, 336-339.
MECD. [17] Vaucelle, C., Bonanni, L. and Ishii H. 2009. Design of haptic
interfaces for therapy. In Proc. CHI '09. ACM, New York,
9. REFERENCES NY, USA, 467-470.
[1] Field, T. 2014 Touch. Second ed. MIT press.
[18] Paredes, P. and Chan, M. 2011. CalmMeNow: exploratory
[2] Réhman, S., Liu, L., and Li, H. 2008. Vibrotactile Rendering research and design of stress mitigating mobile interventions.
of Human Emotions on the Manifold of Facial Expressions, In Proc. CHI EA '11. ACM, New York, NY, USA, 1699-
J. of Multimedia, 3,18-25. 1704.
[3] Seifi, H., and MacLean, K. E. 2013. A first look at [19] Arafsha, F., Alam, K. M., and El Saddik, A., 2012.
individuals' affective ratings of vibrations. In EmoJacket: Consumer centric wearable affective jacket to
Proc. WHC’13, 605-610. enhance emotional immersion. In Proc. IIT’12, IEEE, 350-
[4] Chang, A., O’Modharain, S., Jacob, R., Gunter, E. and Ishii, 355.
H. 2002. ComTouch: design of a vibrotactile communication [20] Pares, N., Carreras, A., Durany, J., et al. 2004. MEDIATE:
device. In Proc. DIS '02. ACM, New York, NY, USA, 312- An interactive multisensory environment for children with
320. severe autism and no verbal communication. In Proc.
[5] Krishna, S., Bala, S., McDaniel, T., McGuire, S. and IWVR’04, Lausanne, Switzerland.
Panchanathan, S. 2010. VibroGlove: an assistive technology [21] Dublon, G., and Portocarrerro, E. 2014. ListenTree: Audio-
aid for conveying facial expressions. In Proc.CHI EA '10. haptic display in the natural environment. In Proc. ICAD’14,
ACM, New York, NY, USA, 3637-3642. June 22-25, 2014, New York, NY.
[6] Alves Lino, J., Salem Bernard, B. and Rauterberg, M. [22] Morrison, A., Knoche, H., and Manresa-Yee, C. 2015.
2010. Responsive environments: User experiences for Designing a Vibrotactile Language for a Wearable Vest. In
ambient intelligence. Journal of Ambient Intelligence and Proc HCII’15, LNCS, Springer.
Smart Environments, 2 (4). 347-367.
[23] Zephyr Technology Corporation. 2015.
[7] Bullivant, L. 2006. Responsive Environments: architecture http://zephyranywhere.com/ Last accessed March 10, 2015.
art and desing. Ed. Victoria & Albert Museum.
[24] Vorderer, P., Wirth, W., Gouveia, F.R., et al. 2004. MEC
[8] Morrison, A., Viller, S., and Mitchell, P. 2011. Building spatial presence questionnaire (MEC-SPQ): Short
sensitising terms to understand free-play in open-ended documentation and instructions for application. Report to the
interactive art environments. In Proc. CHI’11, ACM, New European Community, Project Presence. (IST-2001-37661).
York, NY, USA, 2335-2344.
[25] Jackson, S.A., Marsh, H.W. et al. 1996. Development and
[9] Spelmezan, D., Jacobs, M., Hilgers, A. and Borchers, J. validation of a scale to measure optimal experience: The
2009. Tactile motion instructions for physical activities. In Flow State Scale. J. Sport Exerc. Psychol. 18, 17-35.
Proc. CHI’09. ACM, New York, NY, USA, 2243-2252.
Deci, E.L. and Ryan, R.M. 2000. The“ what” and“ why” of goal
[10] McDaniel, T., Villanueva, D., Krishna, S. and Panchanathan, pursuits: Human needs and the self-determination of
S. 2010. MOVeMENT: a framework for systematically behavior. Psychological inquiry 11, 4, 227-268

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (I)

| 61 |
NATURAL USER
INTERFACES (II)
 Exploración del busto humano en tiempo real mediante interacción natural con fines educativos
R. Méndez | J. Flores | R. Arenas

Exploración del busto humano en tiempo real mediante

interacción natural con fines educativos
Roi Méndez Julián Flores Rubén Arenas
CITIUS CITIUS CITIUS
Universidade de Santiago de Universidade de Santiago de Universidade de Santiago de
Compostela Compostela Compostela
[email protected] [email protected] [email protected]

ABSTRACT denominada Chispas compuesta por 13 módulos interactivos, que

En este trabajo se presenta el desarrollo de un sistema interactivo recorren cada una de las 8 inteligencias descritas por Gardner [4]
de visualización del interior del cráneo humano para La Domus y algunos aspectos de la anatomía y fisiología del cerebro. Se
(Casa del Hombre de A Coruña). La interacción del usuario con el plantea crear un contenido interactivo que permita al visitante
mismo se realiza a través de una interfaz natural e intuitiva observar las características anatómicas del cerebro, explorarlo y
(desarrollada ad-hoc) basada en sensores de movimiento Prime visualizar las diferentes partes del mismo, así como los elementos
Sense. Se presentan el ciclo de vida del desarrollo del proyecto orgánicos del busto humano. Por lo tanto, este módulo cubre la
basado en prototipos y por tanto los diferentes prototipos parte de la exposición centrada en la anatomía del cerebro.
implementados así como las pruebas realizadas sobre los mismos
hasta llegar a la implementación final que, a día de hoy, se
2. OBJETIVOS
El objetivo principal de este proyecto es desarrollar un sistema
encuentra operativa en el museo.
que permita la exploración anatómica del cerebro humano
Categories and Subject Descriptors mediante una interacción natural por parte del usuario.
I.3.6 [Computer Graphics]: Methodology and Techniques – Para lograr este objetivo se plantearon dos requisitos funcionales
Interaction Techniques principales:
General Terms • Implementar una herramienta de visualización de
Design, Human Factors. imagen médica (TAC, RMI o similar) que permita
explorar el modelo mediante un plano de corte en los
Keywords tres ejes cartesianos del espacio, pudiendo seleccionar la
Interacción, Kinect, Prime Sense, museo interactivo, anatomía posición del plano y el eje de corte del mismo en cada
humana, interacción natural. momento. De esta forma se podrá visualizar el interior
del cráneo desde una perspectiva cenital, frontal o
1. INTRODUCCIÓN lateral permitiéndole al usuario la exploración del
En los últimos años los museos, en especial los científicos, han modelo desde diferentes ángulos y puntos de vista.
sufrido una gran evolución pasando de meras salas de exposición
a espacios donde el usuario puede interactuar con los elementos • Implementar un sistema de interacción intuitivo, natural
que se muestran. Incluso museos clásicos como el Louvre han y atractivo del usuario con el sistema, evitando el uso de
incorporado elementos como la Nintendo 3DS o el sensor Kinect periféricos clásicos como teclados, ratones y, cada vez
para crear una experiencia más interactiva con el visitante. En este más, pantallas táctiles. Este sistema debe permitir que el
tipo de entornos tiene una importancia capital el hecho de usuario defina en todo momento el plano de corte
mantener la atención del visitante y transmitirle conocimientos de deseado y su posición en el espacio. Igualmente debe de
una forma clara, amena y entretenida, promoviendo un definirse la posición física del usuario respecto al
aprendizaje por experimentación que haga más atractiva la sistema así como las acciones que este debe realizar
información que se desea transmitir. Si, por otra parte, los para un correcto funcionamiento del mismo.
potenciales visitantes son niños debemos desarrollar paradigmas
educativos que permitan difuminar, en la medida de lo posible, la 3. DISEÑO DEL CONTENIDO
frontera entre el aprendizaje y el juego. El paradigma utilizado para el diseño del sistema se basa en la
arquitectura clásica de un expositor de museo. Estos expositores
El caso de estudio se centra en el diseño e implementación de un son urnas en cuyo interior se expone el elemento de interés. En
contenido interactivo para la Domus, Casa del Hombre de A nuestro caso se sustituye la urna por un monitor y se introduce un
Coruña [2], museo interactivo, encuadrado en el conjunto de componente de interacción del usuario con el contenido que
museos científicos de la ciudad, que centra su actividad en permita incrementar el valor expositivo y docente del mismo. De
diferentes aspectos del ser humano como especie, esta forma, el diseño debe tener una arquitectura similar, tal como
proporcionando, entre otras cosas, un recorrido completo a través se muestra en la Figura 1. Está formado por una pantalla en la que
de su anatomía. Se trata de un museo orientado principalmente a realiza la visualización de las imágenes del cráneo y uno o varios
un público infantil y familiar, por lo que ha de tenerse en cuenta el sensores para permitir la interacción natural del usuario con el
aspecto lúdico a la hora de desarrollar la instalación. Esta sistema.
instalación se enmarca dentro de una nueva exposición

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 63 |
 Exploración del busto humano en tiempo real mediante interacción natural con fines educativos
R. Méndez | J. Flores | R. Arenas
motor gráfico orientado al render en tiempo real, decantándonos
por OSG (Open Scene Graph) [11] por tratarse de uno de los más
extendidos y potentes, así como por la posibilidad de integración
que ofrecía tanto con ITK (Insight Segmentation and Registration
Toolkit) [6] como con DCMTK (DICOM Toolkit) [10],
herramientas que permiten leer e interpretar variados formatos de
imagen médica y en especial DICOM, formato que se ha
convertido en un estándar en este campo y que presenta grandes
dificultades de lectura, interpretación y generación debido a su
complejidad interna [1]. Con esta herramienta se obtuvo una
visualización en tiempo real (sin variaciones en la calidad del
renderizado) y con un frame rate estable, condiciones
indispensables para el sistema planteado.
En este punto se planteó la substitución de las imágenes iniciales
en formato DICOM (en escala de grises) por unas equivalentes en
formato png (en color) ya que la información cromática permite
una mejor percepción de los diferentes tejidos y estructuras
presentes en el modelo además de aumentar la espectacularidad de
la visualización. Al disponer del dataset completo del Visible
Human Project [12] (incluidas las imágenes reales en color de los
cortes realizados) y utilizar OSG como motor de render el cambio
fue directo, ya que este motor gráfico permite la importación de
volúmenes como conjuntos de imágenes en diferentes formatos.
Figura 1. Esquema de diseño del módulo Este cambio permitió, además, aumentar la resolución de la
visualización al tener las imágenes en color un mayor detalle que
Puesto que se trata de un contenido interactivo se define una zona su equivalente DICOM. Se hubo de ajustar el tamaño del volumen
de control en la que el usuario puede moverse manteniendo el de entrada para mantener el frame rate dentro de los parámetros
control del sistema, fuera de la cual el usuario perderá el control del tiempo real ejecutando la aplicación sobre una tarjeta gráfica
del mismo. Esto es necesario debido a que, al estar situado el Nvidia Geforce 9600GT.
módulo en una zona con tránsito de visitantes se debe de discernir
cual de los posibles usuarios en la zona cercana al quiosco tiene el
control. Esta diferenciación se realizará mediante la restricción del
funcionamiento del sistema a la zona de control especificada.

4. TECNOLOGÍAS EMPLEADAS Y
DISEÑO HARDWARE
Se realizó un estudio previo de cuáles eran las mejores
alternativas para llevar el proyecto a cumplir sus objetivos de
forma eficiente y satisfactoria. Siguiendo las necesidades básicas
planteadas se realizó un estudio de las diferentes herramientas de
software libre tanto para la visualización como para la interacción
del usuario con el sistema. Como fue comentado con anterioridad
el sistema consta de dos partes: un entorno gráfico que permite la
visualización del modelo y un sistema de interacción para
controlar dicha visualización.
Figura 2. Imagen de un corte procesado.
4.1 Sistema de Visualización Las imágenes en color precisaron de un preprocesado ya que la
Se ha de construir una imagen tridimensional a partir de la zona exterior al cráneo estaba completamente cubierta por la
información contenida en 275 imágenes con una resolución de mezcla de gelatina y agua empleada para la congelación del
512x512 pixels pertenecientes al Visible Human Project [12]. sujeto. A efectos prácticos esto supone que al realizar la
Para realizar este proceso se planteó el uso de un motor gráfico visualización exterior lo que observa el usuario es un bloque azul.
especializado en esta tarea: VTK (The Visualizaton Toolkit)[7]. Para su procesado se definió una zona de interés (en la que estaba
No obstante, los resultados de rendimiento con las imágenes del contenido el busto) y descartar todos los elementos exteriores a
dataset desaconsejaron su uso, ya que los frame rates obtenidos no esta zona. Una vez realizado este proceso se procedió a la
eran estables y caían en ocasiones por debajo del tiempo real eliminación de la región exterior azul mediante un proceso de
necesario para toda aplicación interactiva. Asimismo, la propia umbralización. Al ser la parte anatómica de la imagen una región
implementación del visor de VTK realiza una simplificación del cerrada y al tener los tejidos de la misma una pigmentación bien
modelo cuando está siendo manipulado (cuando la imagen diferenciada de la región a eliminar bastó con seleccionar el color
visualizada no es fija) para evitar caídas en el frame rate, lo que azul y aplicarle una serie de umbrales. El resultado puede verse en
deriva en una visualización poco precisa y en un efecto de la Figura 2. En cuanto a las zonas que contienen mucosas en el
popping cuando se deja de manipular el modelo y se muestra el interior del cráneo (aparecen también azules debido a la
render sin simplificar. Teniendo esto en cuenta se optó por un penetración del líquido de congelación en su interior) se decidió

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 64 |
 Exploración del busto humano en tiempo real mediante interacción natural con fines educativos
R. Méndez | J. Flores | R. Arenas
no procesarlas ya que la pigmentación resultado de la congelación necesidad de usar patrones ni marcadores de ningún
permite a un usuario no experto detectarlas con mayor facilidad. tipo. Emplea cámaras de color e infrarrojos para
El conjunto de datos final está formado por 275 imágenes con una detectar usuarios en su zona de acción y reconstruye
resolución de 1024x608 pixels. automáticamente un esqueleto del mismo, siguiendo sus
movimientos en el espacio.
El proceso seguido a la hora de construir un modelo 3D a partir de
una serie de cortes 2D es el siguiente: Primero se cargan las Teniendo en cuenta que las condiciones lumínicas del museo no
imágenes en memoria. Una vez cargadas se construye una textura podían ser controladas sin afectar a toda la exposición se descartó
3D que se visualizará a través de un proceso de Raycasting. Para el uso de cámaras RGB por los problemas que presentan ante
realizar los cortes requeridos en los tres ejes cartesianos se hace cambios de iluminación. Se descartó la manipulación de un objeto
uso de tres planos de clipping (uno por cada eje) que, combinados como interfaz de control (un mando de consola o elemento
con la textura 3D y el raycasting permiten renderizar todos los manipulable con captura de movimiento) por los problemas de
detalles tanto exteriores como interiores del volumen visualizado. mantenimiento que este podría presentar así como por la
posibilidad de hurto del mismo. Se descartó, por tanto, el uso de
4.2 Sistema de Interacción cámaras de infrarrojos o mandos de control. Se decidió, por tanto
Al tratarse de un museo principalmente dirigido a público infantil emplear sensores PrimeSense para crear una interacción natural
y familiar el diseño de una interacción intuitiva y al mismo tiempo entre el usuario y el sistema a través de los movimientos de su
atractiva resulta indispensable. cuerpo.
En los últimos años han aparecido una serie de sensores El esquema de interacción es el siguiente: El visitante del museo
avanzados que permiten enriquecer, a través una captura de llega al módulo, se sitúa de pie, en una posición indicada
movimiento de bajo coste, la interacción del usuario con el mediante un gráfico en el suelo, de forma frontal a la instalación y
ordenador, creando nuevas interfaces naturales y metáforas de el sensor lo detecta y obtiene de sus gestos la información
acción que añaden una tercera dimensión además de aumentar la necesaria para manipular el sistema. Finalmente, la imagen que se
implicación física del usuario en la interacción con la máquina muestra en pantalla varía siguiendo los gestos realizados por el
[3]. Persiguiendo un control más natural del sistema se plantea la usuario. Para la detección de gestos se decidió optar por un
utilización de estos sensores con el fin de evitar el uso de desarrollo ad-hoc en lugar de emplear gestos estándar ya que se
periféricos tradicionales como el ratón y el teclado buscando entendió que estos se adecuarían mucho mejor a una interacción
simplificar la realización de una serie de tareas por parte del natural y efectiva [9].
usuario. Los tipos de sensorización planteados fueron los
siguientes: 5. IMPLEMENTACIÓN
La implementación del sistema se planteó como un desarrollo por
• Sensorizaciones mediante cámaras de infrarrojos: Se prototipos, se crearon diferentes prototipos y se probaron en
planteó el uso de sistemas de cámaras de infrarrojos sujetos experimentales, hasta obtener una implementación
para sensorizar al usuario. El usuario tendría una intuitiva del sistema, de modo que este a penas necesitase de
herramienta que manejaría en el aire y a través de la instrucciones de uso y la interacción con el mismo surgiese de
cual podría interactuar con el sistema. Esta herramienta forma natural. También se tuvieron en cuenta diversos factores
sería seguida por el sistema y el renderizado de las para evitar la fatiga del usuario [5] como una posición natural y
imágenes 3D se realizaría consecuentemente a esta relajada del mismo durante la interacción, la no necesidad de
interacción. Los principales inconvenientes que llevaron mantener una tensión postural innecesaria en manos o cabeza para
a decartar este tipo de sensorización fue, por un lado, la tener una interacción efectiva, etc.
ausencia de librerías libres y fiables para el tipo de
hardware disponible y, por otro, la necesidad de que el 5.1 Prototipo 1
usuario manejase una herramienta, con las posibilidades La exploración tradicional de modelos 3D se basa en el zoom, la
de hurto y rotura que esto implica. rotación y la traslación [8] convirtiendo la interacción del usuario
• Sensorizaciones mediante cámaras RGB y marcadores: con el sistema (generalmente a través de interfaces 2D) en algo
El extenso uso que el grupo de investigación ha hecho complejo y que requiere de un aprendizaje. Sin embargo, el
de sensorizaciones con webcams empleando marcadores paradigma planteado reduce éste al desplazamiento de planos a
llevó a que se planteara una interacción basada en esta través de los ejes cartesianos, lo que simplifica la interacción al no
tecnología. El problema planteado era la dificultad de, ser dependientes las transformaciones en unos de las realizadas en
con un usuario manipulando un objeto en el aire, evitar otros. Por ser tres los planos de corte planteados, se definieron tres
oclusiones en los patrones, así como obtener la zonas de acción distintas que permiten al usuario operar en cada
configuración óptima de iluminación de la sala sin, con una sobre un plano de corte distinto.
ello afectar al resto de la exposición o, al igual que en el
caso de la sensorización con cámaras de infrarrojos la
presencia del elemento físico manipulable que puede ser
hurtado o que se puede deteriorar con el uso.
• Sensorización mediante el uso de sensores PrimeSense:
Se planteó el uso de sensores PrimeSense como Kinect
o Asus Xtion debido a las grandes posibilidades que
ofrecen y lo novedoso de la interacción que plantean. La
ventaja principal de esta tecnología es que trabaja sin la

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 65 |
 Exploración del busto humano en tiempo real mediante interacción natural con fines educativos
R. Méndez | J. Flores | R. Arenas
• La indefinición entre zonas de acción: Cuando la mano
pasa de una zona de acción a otra la perspectiva del
modelo que visualiza el usuario cambia al modificar el
plano de corte sobre el que actúa, produciendo
confusión al no ser capaz de identificar los límites de las
diferentes zonas de acción, situación a la que contribuye
el tamaño reducido de las mismas.

5.2 Prototipo 2
Dados los problemas que planteaba el prototipo 1 se decidió
implementar dos cambios básicos en la concepción del sistema.
Por un lado se decidió aumentar el recorrido de la mano
ampliando las zonas de acción que pasaron del entorno de la
cabeza a ocupar todo el entorno del tronco (de 25 cm a 50 cm de
recorrido) para permitir al usuario tener un mayor control del
detalle que deseaba ver en cada momento (5,5 imágenes por cada
centímetro de recorrido de la mano). Al ser más grande el rango
Figura 3. Zonas de acción del prototipo 1.
de movimiento de la mano el plano se moverá más lentamente
Se definieron las zonas de interacción que pueden verse en la respecto al movimiento de la misma permitiendo observar detalles
Figura 3 y cada una de ellas se asoció con un eje. Este primer que, de otra forma, serían imposibles de visualizar y reduciendo,
prototipo restringe las zonas de acción al entorno de la cabeza del de esta forma, la intensidad de los temblores en la imagen debidos
usuario asumiendo una metáfora exploratoria basada en que si se al ruido de la señal del sensor.
explora el busto, la forma más simple de interacción para el
usuario era asumir que lo que está visualizando es su propio
cráneo y que, al desplazar las manos en torno a él, puede explorar
su interior. La zona 1 se situó frente al pecho (para evitar poner
las manos delante de los ojos y entorpecer la visión) y permite
realizar cortes en el modelo en el sentido horizontal (eje X). La
zona 2, a la derecha o izquierda de la cabeza, permite realizar
cortes en sentido vertical (eje Y) y la zona 3, situada delante de la
zona 2 permite explorar el interior del busto en profundidad (eje
Z).

El prototipo se probó con 6 usuarios experimentales (entre los que

se encontraban los organizadores de la exposición) que utilizaron Figura 4. Zonas de acción del prototipo 2.
el sistema en base a objetivos (detectar el nervio óptico, visualizar
el cerebelo, etc.). A pesar de las ventajas que presentaba esta Por otro lado, se decidió también hacer una mayor diferenciación
propuesta, como el ser válida tanto para zurdos como para diestros de zonas de trabajo, para que en cada zona espacial sucediese sólo
(las zonas de interacción estaban situadas a ambos lados de la una cosa, buscando simplificar, de esta forma, la comprensión de
cabeza) o lo acertado de la metáfora de interacción que facilitaba qué se manipulaba en cada zona. Se definieron las tres zonas
la comprensión del funcionamiento al usuario, empíricamente se como se muestra en la Figura 4. A la derecha del hombro derecho
pudieron detectar una serie de problemas operativos: se permitía al usuario interactuar con el eje vertical de corte
desplazando su mano en sentido vertical, en la zona delantera del
• Problemas derivados del poco recorrido que tiene el torso y cabeza se interactuaba con el eje horizontal de corte
movimiento de la mano entorno a la cabeza: Ya que el desplazando la mano en sentido horizontal y a la izquierda del
modelo 3D se construye a partir de 275 cortes y el hombro izquierdo, acercando y alejando la mano del mismo, se
desplazamiento de la mano para mover el plano a través realizaba la interacción del usuario con el eje de profundidad de
del mismo se reducía a 25 centímetros, lo que corte.
dificultaba la visualización de pequeños detalles al
avanzar el plano de corte una gran distancia con un Para probar el funcionamiento de este prototipo se realizaron
movimiento sutil de la mano (11 imágenes por pruebas a 7 usuarios no expertos. Estas pruebas estaban
centímetro de recorrido de la mano). compuestas por 9 tareas cronometradas (localizar el nervio óptico,
localizar el bulbo raquídeo, etc.) y un breve cuestionario para
• El ruido en la señal producida por el sensor: El sensor evaluar el nivel de frustración del usuario así como la usabilidad
introducía una gran cantidad de ruido, lo que derivaba del sistema. Las tareas fueron realizadas con éxito por la totalidad
en un movimiento continuo de los planos de corte, de los usuarios, que invirtieron un tiempo medio de 13 segundos
impidiendo una visualización clara de los mismos. Este en realizar cada una de ellas. De las respuestas al cuestionario se
ruido se veía incrementado, además, por la cercanía de dedujo que los usuarios consideraban el prototipo atractivo, así
las manos al cuerpo derivada del tipo de interacción como intuitivo una vez habían interiorizado la mecánica de
propuesta, que dificultaba la labor de seguimiento del funcionamiento del mismo. Sin embargo, en los momentos
sensor. iniciales, se sentían perdidos al no saber qué dirección exploraban
con cada mano, lo que les producía frustración al pasar de una
XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)
| 66 |
 Exploración del busto humano en tiempo real mediante interacción natural con fines educativos
R. Méndez | J. Flores | R. Arenas
tarea a otra. También salió a relucir la dificultad de algún usuario posicionamiento de los diferentes elementos del sistema (Figura
diestro para manejar el plano de profundidad (Z) con la mano 5).
izquierda, ralentizando de manera notable su desempeño en las
tareas que requerían el uso de esta vista.
Al ser la primera impresión y el manejo intuitivo del sistema de
capital importancia en un museo interactivo, así como la
accesibilidad y que la aplicación sea usable para el usuario
independientemente de su pericia con su mano menos hábil, se
decidió cambiar radicalmente el enfoque que se estaba dando a la
interacción, eliminando por completo las zonas de actuación
diferenciadas.

5.3 Prototipo 3
Ante los problemas planteados por las zonas diferenciadas de
interacción (frustración en el usuario al no saber que tarea realiza
en cada zona, dificultades de manejo de la aplicación con la mano
menos hábil, etc.) se cambió el enfoque de la interacción
planteada, centrando la misma en la dirección principal de
movimiento de la mano. Esta solución consiste en que la
visualización es modificada y el objeto explorado en la dirección
del eje en que la mano del usuario presente un mayor
desplazamiento. Esta solución plantea tres ventajas principales
respecto a las dos anteriores:
• Permite una diferenciación más intuitiva del eje de corte
que se está utilizando en cada momento y de por qué es
ese el eje sobre el que se actúa al no importar en que Figura 5. Diseño final del sistema con medidas.
parte del espacio se encuentre la mano, sino la dirección
en la que se mueve. 6. PRUEBAS
En la fase de prototipado también se observó la necesidad de dar
• Permite la utilización de ambas manos
un punto visual de referencia para que el usuario supiese en todo
indiferentemente, solo hay que seleccionar cual tiene
momento que plano de corte estaba visualizando y en qué
preferencia en caso de que ambas estén interactuando.
posición respecto al busto se encontraba. Para esto, en una
• Ofrece la posibilidad de tener un mayor rango de primera aproximación se optó por replicar el modelo en
movimiento en los tres ejes, al no estar este limitado por perspectiva y situarlo en el interior de un cubo transparente a
su correspondiente zona acción. través del cual se desplazaban los planos de corte de forma visual
y solidaria con el plano de corte real. Esta visualización, a pesar
Se repitieron las pruebas que se habían realizado en el prototipo de lo vistoso no resultó efectiva para conocer la posición exacta
anterior sobre los mismos 7 usuarios. De nuevo las 9 tareas fueron del plano respecto al modelo (por motivos de perspectiva) por lo
completadas con éxito por todos los participantes no observándose que se decidió optar por una solución más esquemática y que
diferencias notables en los tiempos empleados en uno y otro caso transmitiera la información de una forma más simple y directa. Se
(entorno a los 13 segundos de media) para completarlas. Sin emplearon dos siluetas (una de frente y otra de perfil) que se
embargo si se apreciaron diferencias en las respuestas al visualizan en paralelo a la imagen del modelo 3D. Según el plano
cuestionario. Se subrayaba la facilidad para el manejo del mismo, de corte que se esté explorando en cada momento varía la silueta
lo intuitivo que resultaba y lo preciso que era en el movimiento. que se está visualizando y, dependiendo de la posición del plano
Se señalaba también alguna dificultad a la hora de cambiar de tipo que actúa sobre el modelo, se desplaza una línea a través de la
de vista, pero la valoración general, tanto de usabilidad como de silueta indicándole al usuario qué zona del busto está
atractivo, fue muy positiva eliminando la frustración inicial que si visualizando. Dos flechas indican la dirección en la que se está
habían sentido en la utilización del anterior prototipo. desplazando el plano en cada momento simplificando el control
Se determinó, por tanto que este sería el prototipo final y se pasó a por parte del usuario así como la comprensión del funcionamiento
su implementación final para ser probado con una muestra del sistema (Figura 6).
poblacional más representativa de los futuros usuarios que el Aprovechando una serie de visitas de escolares al centro de
sistema tendría en el museo. investigación y dado el enfoque familiar del museo se realizaron
En esta implementación final se determinó que el sistema una serie de pruebas de uso del sistema en las que participaron 41
reaccionaría a movimientos de las manos de los usuarios cuando estudiantes de entre 8 y 9 años. La prueba consistió en dejarlos
estas estuviesen por encima de la cintura y que, en caso de no ser manipular libremente el sistema sin ponerles límite de actuación o
así, el sistema permanecería en reposo. Asimismo se decidió que tiempo y una puesta en común posterior de la experiencia. Se debe
en caso de interacción de las dos manos al mismo tiempo se le de subrayar que el tiempo de utilización no fue totalmente libre al
daría preferencia a la derecha por ser la de mayor frecuencia encontrarse la prueba en medio de una visita que debía de cumplir
poblacional. También se definieron las medidas y unos tiempos aunque sí que es un escenario que se puede asemejar
a una visita de escolares en grupo al museo en la que hay un
tiempo limitado y numerosas instalaciones para visitar.
XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)
| 67 |
 Exploración del busto humano en tiempo real mediante interacción natural con fines educativos
R. Méndez | J. Flores | R. Arenas
El tiempo medio que estuvieron interactuando con el sistema fue Más allá de los datos obtenidos, los escolares afirmaron sentir una
de 61 segundos de los cuales un 75% del tiempo lo hicieron mayor curiosidad por el sistema que si hubiese sido manejado a
utilizando la mano derecha, el 11% la mano izquierda y un 14% través de un periférico clásico y que la interacción les había
no realizando ninguna actividad. Estos datos confirmaron que ante resultado más amena. Insistiendo repetidamente en que les habría
la utilización del sistema los usuarios se mantenían activos y gustado probar un sistema similar a través del que se pudiese
probando sus funcionalidades. Destaca en este aspecto el uso de la observar el cuerpo humano al completo.
mano izquierda ya que solo un usuario era zurdo. El porcentaje
del 11% se explica a través del intento de los escolares de probar
todas las posibilidades del sistema “retándolo” a funcionar con las
dos manos.

Figura 7. Resultados del proceso de pruebas en porcentajes de

tiempo.
Desde el punto de vista de una supervisión visual exterior se
puede afirmar que, en general, este tipo de interacción atraía la
atención de los usuarios pero, en ciertos momentos esta
interacción se convertía más en una finalidad que en un medio,
ocupando el juego de “retar” a la máquina con movimientos
rápidos o ondulantes una mayor parte de la atención del niño que
la visualización propiamente dicha.

Figura 8: Instalación final en el museo

7. CONCLUSIONES Y TRABAJO FUTURO

Figura 6. Diferentes visualizaciones presentes en la aplicación Se ha desarrollado un sistema de visualización y exploración del
final. cerebro humano a partir de una visualización basada en texturas
Dentro de la fase de actividad (ya sea con la mano derecha o la 3D, controlado de una manera natural e intuitiva a través de una
izquierda) los usuarios interactuaron con la vista frontal (eje Z) un interfaz natural para su exhibición en el museo de divulgación
29% del tiempo, con la vista superior (eje Y) un 40% del tiempo y científica Casa Domus. El funcionamiento del sistema se ha ido
con la vista lateral (eje X) un 31% (Figura 7). Destaca probando a lo largo de su desarrollo por prototipos y se ha
ligeramente la vista superior al ser la que posee una mayor validado en una prueba final con usuarios potenciales del mismo.
resolución, al tratarse de los cortes originales a partir de los cuales El sistema ha sido instalado en el museo, con gran aceptación por
se realizó la reconstrucción del modelo 3D. Esta mayor nitidez se parte de los organizadores y visitantes de la exposición.
refleja en un mayor interés de los usuarios por los pequeños
detalles visibles en ese eje. Como trabajo futuro se plantea la mejora del cambio de plano de
corte utilizado en cada momento, ya que, debido al ruido de la

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 68 |
 Exploración del busto humano en tiempo real mediante interacción natural con fines educativos
R. Méndez | J. Flores | R. Arenas
señal del sensor se descartan los pequeños cambios en la dirección Consumer Electronics (ICCE), 2011 IEEE International
del movimiento para esta finalidad, dificultando, en ocasiones, la Conference on, 269–270, 2011.
interacción del usuario con el sistema. [6] Kitware. Itk - segmentation and registration toolkit, 2014.
También se plantea la necesidad de introducir información visual Retrieved January 15, 2013 from Kitware.
y auditiva sobre los diferentes órganos y tejidos que se encuentran http://www.itk.org/
en el busto, indicando su nombre, una breve explicación de su [7] K. Martin, W. Schroeder, and B. Lorensen. Vtk - the
funcionalidad etc. Esto incrementaría, tanto el valor divulgativo visualization toolkit, 2015. Retrieved November 13, 2012
de la instalación, como el interés del usuario. from Kitware. http://www.vtk.org/
8. AGRADECIMIENTOS [8] S. A. Lacolina, A. Soro, and R. Scateni. Natural exploration
Domus, Casa del Hombre de A coruña. of 3d models. In Proceedings of the 9th ACM SIGCHI Italian
Chapter International Conference on Computer-Human
9. REFERENCIAS Interaction: Facing Complexity, CHItaly, 118–121, New
[1] W. D. Bidgood and S. C. Horii. Introduction to the York, USA, 2011.
ACR_NEMA DICOM standard. RadioGraphics, 12(2):345– [9] M. Nielsen, M. Storring, T. Moeslund, and E. Granum. A
355, 1992. procedure for developing intuitive and ergonomic gesture
[2] Domus | Museos Científicos Coruñeses (=mc2), 2015. interfaces for hci. In Antonio Camurri and Gualtiero Volpe,
Retrieved April 15, 2015 from Museos Científicos editors, Gesture-Based Communication in Human-Computer
Coruñeses (Ayuntamiento de A Coruña). Interaction, volume 2915 of Lecture Notes in Computer
http://mc2coruna.org/es/ Science, pages 409–420. Springer Berlin Heidelberg, 2004.
[3] R. Francese, I. Passero, and G. Tortora. Wiimote and kinect: [10] Offis. Dicom software made by offis - dcmtk - dicom toolkit,
Gestural user interfaces add a natural third dimension to hci. 2014. Retrieved January 21, 2013 from Offis.
In Proceedings of the International Working Conference on http://dicom.offis.de/index.php.en
Advanced Visual Interfaces, AVI ’12, 116–123, New York,, [11] Openscenegraph, 2014. Retrieved January 15, 2013 from
USA, 2012. ACM. Openscenegraph. http://www.openscenegraph.org/
[4] H. Gardner. Frames of mind: The theory of multiple [12] V. Spitzer, M. J. Ackerman, A. L. Scherzinger, and D.
intelligences. New York: Basic Books, 1983. Whitlock. The visible human male: A technical report.
[5] Y. Kim, G.A. Lee, D. Jo, U. Yang, G. Kim, and J. Park. Journal of the American Medical Informatics Association,
Analysis on virtual interaction-induced fatigue and difficulty 3:118–130, 1996.
in manipulation for interactive 3d gaming console. In

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 69 |
 Performance evaluation of gesture-based interaction between different age groups using Fitts’ Law
D. Carvalho | L. Magalhães | M. Bessa | E. Carrapatoso

Performance evaluation of gesture-based interaction

between different age groups using Fitts’ Law
Diana Carvalho Maximino Bessa
INESC TEC and UTAD INESC TEC and UTAD
4200-465 Porto, Portugal and 4200-465 Porto, Portugal and
5000-801 Vila Real, Portugal 5000-801 Vila Real, Portugal
+351 {222094000, 259350000} +351 {222094000, 259350000}
[email protected] [email protected]

Luís Magalhães Eurico Carrapatoso

Centro ALGORITMI, UMinho INESC TEC and FEUP
4800-058 Guimarães, Portugal 4200-465 Porto, Portugal
+351 253510180 +351 222094000
[email protected] [email protected]

ABSTRACT Keywords
The recent advances made in human-computer interaction have HCI, natural user interfaces, gestural interaction, performance,
allowed us to manipulate digital contents exploiting recognition- Fitts' law, Microsoft Kinect, Leap Motion, target acquisition,
based technologies. However, no work has been reported that selection tasks.
evaluates how these interfaces influence the performance of
different user groups. With the appearance of multiple sensors and 1. INTRODUCTION
controllers for hand gesture recognition, it becomes important to For many years, the traditional mode of interaction with
understand if these groups have similar performance levels computers was based on a WIMP interface (Windows, Icons,
concerning gestural interaction, and if some sensors could induce Menus, Pointing device), which allowed us to interact with the
better results than others when dealing with users of different age machine via specific pointing devices, usually a computer mouse.
brackets. In this respect, it could also be important to realize if the With the paradigm shift to Post-WIMP interfaces [5], we have
device’s sensor accuracy in terms of hand / full body recognition turned towards a user-oriented and task-oriented approach that
influences interaction performance. We compare two gesture- attempts to simplify the usability of the interface [17], giving
sensing devices (Microsoft Kinect and Leap Motion) using Fitts’ preference to the users innate skills [16] and allowing us to take
law to evaluate target acquisition performances, with relation to advantage of recognition-based technologies that understand
users’ age differences. In this article, we present the results of an human behaviors, such as gestures or touch. However, there
experiment implemented to compare the groups’ performance seems to be a constant addition of new modes of interaction
using each of the devices and also realize which one could yield without the proper awareness as to which could be the most
better results. 60 subjects took part in this study and they were adequate for different user profiles (e.g., children, elderly users,
asked to select 50 targets on the screen as quickly and accurately people with different levels of digital literacy, people with
as possible using one of the devices. Overall, there was a disabilities) and also regarding which types of tasks (selection,
statistically significant difference in terms of performance insertion, manipulation).
between the groups in the selection task. On the other hand, users’
A previous study of ours [3] presented insights on how users
performance showed to be rather consistent when comparing both
interact using different input modalities and which interface holds
devices side by side in each group of users, which may imply that
the best results in terms of usability testing regarding three user
the device itself does not influence performance but actually the
groups with different age brackets. Our preliminary findings
type of group does.
indicated that the gestural interface presented worse results than
Categories and Subject Descriptors the other ones, and we thus acknowledged that the device itself
used for the gesture-recognition – the Kinect – could be
H.5.2 [Information Interfaces and Presentation (e.g., HCI)]: negatively influencing the user due to the accuracy required by the
User Interfaces – evaluation/methodology, input devices and task. Different optical sensors, which allow human body
strategies (e.g., mouse, touchscreen). acquisition with respected accuracy, have been released and
comparable controllers in the same price range include the Kinect
General Terms and Leap Motion, a sensor with declared sub-millimeter precision
Performance, Design, Human Factors. that claims to obtain higher levels of accuracy than the Kinect
[24]. Therefore we think that it may be important to understand if
another device could perform better than the one used in the
previous experiment. After all, when it comes to gestural
interfaces the precision of the sensor is said to be vital.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 70 |
 Performance evaluation of gesture-based interaction between different age groups using Fitts’ Law
D. Carvalho | L. Magalhães | M. Bessa | E. Carrapatoso
In this context, this study aims at throwing some light on the 3. CASE STUDY
indices of performance of three groups of users with different age- The aim of this work is to compare two different gestural input
brackets concerning the use of two different gestural sensors. We devices in order to understand if different age groups display
intended to understand: (1) if the gestural interaction presents similar indices of performance between them, and which device
significantly better results on specific groups opposed to others could lead to better target acquisition performance rates for
with regards to their age; (2) if the devices used and their declared different user groups, using Fitts’ law.
precision influence or not the reaction times and indices of
performance in each group. To accomplish this, we compared the use of Microsoft Kinect and
Leap Motion Controller for motion and gesture control with users
2. SCIENTIFIC BACKGROUND from three distinct age groups: children, young adults and older-
adults. A total of 60 participants who were naïve to the purpose of
2.1 Natural Interfaces the experiment took part in this case study, and were grouped by
Active and passive input modes of interaction [19] have been age and device: (1) 20 children from 9 - 12 years old (10 for each
implemented in countless fields. However, published studies have device); (2) 20 young adults from 20 - 35 years old (10 for each
not yet provided an understanding of how different user groups device); (3) 20 older-adults from 45 - 60 years old (10 for each
perceive distinct tasks and if their performance is directly device).
influenced by the interaction modality.
We intended to emphasize the distinct age groups and work with
Little is known about how the different interfaces affect one’s users that had noticeable differences concerning cognitive
performance when it comes to age-related issues. There are no performance levels and dexterity and, as such, we tried to select
transversal comparisons of different age groups in one same study distinct groups. First, we did not consider users younger than 9
where more than one natural interface is evaluated. Work has years old because: (1) they are not so used to the computer and do
been developed in this area, but not as a systematic approach. not use it on a daily basis, and also they do not have the dexterity
Sambrooks and Wilkinson compare gestural, touch and mouse capabilities as improved as older children; and (2) these ages are
interaction with 15 participants aged between 22 and 34 years old encompassed by the fifth and sixth year of primary school, being
[21]. They reached the conclusion that touch and mouse presented these school years the ones that show greater variance in terms of
better results, but this interaction performance was not compared reasoning by the children [12]. As this variance tends to
between other groups of users and thus it just clarified that the significantly fade towards high school, we selected graduate
gestural performance was indeed worse than the other interfaces students, being these users already at a different stage concerning
regarding that specific niche of participants. Other studies, as [2, cognitive performance when compared with the group of children.
6, 9], compared the interaction performance when using Finally, the group of older-adults consisted of active workers of a
traditional mouse inputs or touchscreens, but not between other secretariat department. This group could present worse dexterity
natural recognition-based interfaces as a hole, and they only or consider gestures to be more challenging.
compare at the most two groups.
Also, we ascertained that all of the participants had the same level
Concerning the gestural controllers, tests have been performed in of computer proficiency and used the computer on a daily basis.
order to understand the accuracy of any given device [20, 24], but However, when it comes to gestural interaction, this modality was
they have not been compared amongst different user profiles, such not familiar to the participants, who were conscious of this
as distinct age groups. gesture-based interface mainly due to having seen gaming
consoles with support to this modality. In fact, only seven users
2.2 Fitts’ Law: An Overview had already tried gestural interaction once.
Fitts’ law is a human performance model that has been widely
applied to describe the relationship between movement time, The majority of participants were right-handed (right-handed: 58;
distance and accuracy within specific pointing tasks, as a way to left-handed: 2), and the setup was calibrated to accommodate each
formally evaluate pointing devices [15] and compare their arm as needed: the left-handed participants used the left arm to
efficiency [23]. The original experimental paradigm Fitts’ Law [7] perform the tests, and the right-handed used the right arm.
measures the rate of transmission in “bits per second” of a pre-
established movement with different indices of difficulty. Also, 3.1 Apparatus
ISO 9241-9 “Ergonomic requirements for office work with visual We conducted the experiment in a closed room with artificial light
display terminals (VDTs) -- Part 9: Requirements for non- and the tests were performed in a specific setup assembled for the
keyboard input devices” [10, 11] provides a standard approach for purpose of this research. The system consisted on a 22’’ screen
input device evaluation based on Fitts’ law. Fitts’ performance placed on a desk in front of the user, with a resolution of
model can also be applied when comparing and evaluating 1280x800 pixels; a Microsoft Kinect sensor mounted on a tripod
pointing devices. Task differences, selection techniques, response behind the screen, about 25 cm above it and facing the user; a
irregularity and test conditions applied may influence Leap Motion sensor placed on top of the desk, between the user
experimental variations. Understanding these variables increases and the screen, and facing upwards. The distance between the user
the validity of across-studies comparisons regarding input and the Kinect sensor was about 75 cm, and was calibrated to see
techniques [15]. In this sense, researchers use this model to the user’s upper body only. The user was seated facing the setup
measure multiple movement times and then determine how the at all times, and this setup was not altered throughout the different
different conditions or devices affect the coefficients of the trials. Also, a purpose-built application based on Fitts’ law was
equation, that is, performance. developed in Python with the support of: the Kivy Framework,
TuioKinect and the Leap Motion Python API.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 71 |
 Performance evaluation of gesture-based interaction between different age groups using Fitts’ Law
D. Carvalho | L. Magalhães | M. Bessa | E. Carrapatoso

3.2 Experiment Design

The experiment made use of an independent-measures design and
the tasks were performed in a controlled environment. Aside from
the discrete data collected during the tests, we also gathered
qualitative observational analysis on the participants’ behavior. At
the beginning of each phase of this study, we performed a
questionnaire regarding the participants’ previous experience with
gestural interfaces. Also, at the end of each test we proposed a
questionnaire with qualitative Likert Scales [13], in order to Figure 1. Feedback for the selection strategy.
understand the users’ preferences and their views regarding: ease
of use of the device in question, fatigue effect, level of user
comfort / frustration, and users’ degree of concentration. 3.3 Procedure
We analyzed the users’ performance considering only movements
The tests were performed on different days and with different along the “x” axis and used a reciprocal one-dimensional pointing
users. As such, the participants only tried one of the devices, and task based on Fitts’ original experiments [7]. This consisted on a
not both of them, as we did not intend for the use of one device to horizontal movement between two vertical bars displayed on the
influence the movement times recorded on the other due to screen, representing an initial location and a target (as shown on
performance improvement over trials. Figure 2). According to MacKenzie [15], three-dimensional
In order to understand the users’ index of performance when movements may follow the same predictive model as a one-
considering target acquisition tasks using gestural interaction with directional task, as a possible substitute for target width when the
the Kinect sensor versus Leap Motion, we resorted to Fitts’ law angle of the approach varies can be the distance through the
evaluation paradigm, a frequently used model for measuring targets along the approach vector. However, we only intend to
movement performance. Fitts quantifies the movement tasks' understand the one-dimensional movement with homogeneous
difficulty, also known as Index of Difficulty (ID), by the metric shapes.
"bits" and calculates it by using the value of the distance, or
amplitude (A), between two specific targets and their width (W)
or tolerance area:
ID = log2(A/W + 1). (1)
Although not in its original form, this formulation is a more stable
model for the Fitts’ Law suggested by Mackenzie [15], since it is
resilient to negative values. The Index of Difficulty can be
obtained by varying the values of the width of the targets and their
distance, thus providing a range of task difficulties. We used six
different amplitudes: A = 200, 460, 750, 770, 930, 940 pixels; and
five different target widths: W = 16, 26, 50, 120, 200 pixels.
These choices attain seven levels of Fitts' Law’s index of
difficulty, from 1.72 bits to 5.20 bits. The order of appearance of
the seven indices of difficulty was randomized. We also included Figure 2. Layout of the targets displayed on the screen
two target directions (left, right) in the study because it could be
relevant to understand if the contactless motion was influenced by
different directions [4, 8]. Each test proceeded as follows. At the beginning of the test, the
participant would be required to select a “start” button shown on
In summary, the target acquisition data consisted of: 60 the screen, therefore maintaining a consistent original position.
participants divided in three groups; two input devices (Microsoft Afterwards, two vertical bars would be revealed: a grey bar
Kinect; Leap Motion); two target directions (left, right); 25 representing the initial position, and the green bar being the
repetitions per index of difficulty; and seven indices of difficulty intended target to be selected. Participants then selected the target
(1.72, 2.25, 3.12, 3.14, 4.04, 4.89, and 5.20 bits). Thereby, bar in each successive trial, which was at all times organized in
altogether there were 28 experimental conditions and an overall of the opposite direction (from left to right and from right to left).
42 000 trials. Also, there was a visual feedback to indicate the gesture position
The gestural-based cursor illustration was a target badge (Figure was on top of the target object: the bar changed its color to blue.
15) grounded on the validated “point and wait” strategy [22] for The start time was recorded from each target bar to the next and
selection. This cursor was shown on the screen when a pointing stopped when the next target was selected. At the end of all of the
gesture was interpreted by the system. The center of the target trials, the bars disappeared and a “successfully completed task”
badge represented the point from which the selection coordinates message appeared on the screen.
were referenced by the application. When the system detected the Participants were given a practice attempt before starting the test.
gesture to be motionless, a visual feedback was triggered to make This period consisted in completing a continuous task [15] with
the user aware of the selection progress: a green circle started 10 consecutive trials, in order for the participant to feel at ease
growing inside the cursor representation until it reached the whole with the interaction device.
target and thus the selection was made. Here, the user had to point
and wait for 0.8 seconds for the system to recognize the intention.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 72 |
 Performance evaluation of gesture-based interaction between different age groups using Fitts’ Law
D. Carvalho | L. Magalhães | M. Bessa | E. Carrapatoso

3.4 RESULTS AND DISCUSSION pronounced as the children, being the Kinect the one with the
In this section, we present the results of this case study. We highest movement times (Kinect, average rank of 13.20; Leap
believe that we may be able to throw some light on how or if: (1) Motion, average rank of 7.80). Also, we determined a statistically
the different user groups exhibit significant differences in terms of significant difference when comparing the older-adults’
interaction performance when compared between each other; (2) interaction using both sensors (U = 21, p = .028), where the Leap
the accuracy of the devices influences gestural interaction when it Motion remains faster (Kinect, average rank of 13.40; Leap
comes to target acquisition tasks, concerning users of different age Motion, average rank of 7.60). This may indicate that according to
brackets. the movement times all the groups display lower results when
interacting with the Leap Motion sensor than with the Kinect,
3.5 Movement Times and Errors which may imply that for gestural interaction concerning selection
We present the movement times recorded during each trial of the tasks, the Leap Motion could induce quicker reaction times
experimental tests with the Kinect and Leap Motion devices regardless of the users’ age.
(Figure 17). A boxplot analysis of the data revealed a presence of Overall, considering a comparison between the three groups, the
outliers from standardization failure [18] on some of the young adults presented the lowest mean time with both devices,
movement times regarding the calibration of the equipment, and which may indicate that when it comes to gestural interaction the
thus these results were removed in order to prevent distortion of young adults are quicker in completing target acquisition tasks
estimates. than the other groups. Since our group sample was limited to 10
In terms of movement times recorded, the device that registered participants for each group and each device and some data were
the fastest mean results throughout all the three groups was the not normally distributed, a non-parametric independent-samples
Leap Motion: children (1.94 s); young adults (1.76 s); and older- Kruskal-Wallis 1-way ANOVA was run to determine whether this
adults (2.24 s). On the other hand, the Kinect sensor registered difference in terms of mean times throughout the groups was
higher mean movement times: children (2.46 s); young adults indeed significant. The distribution of the times of all the groups
(2.03 s); and older-adults (2.64 s). using the Leap Motion was considered the same and thus not
statistically different (χ2(2) = 5.635, p =.060), presenting a mean
rank time of 15 for the group of children, 11.10 for young adults,
and 20.40 for older-adults. On the other hand, considering the use
of the Kinect, the Kruskal-Wallis test showed that except for the
children and older-adults (p = .875), there was a statistically
significant difference in mean times between the different groups
(χ2(2) = 12.742, p =.002), with a mean rank time of 17.30 for the
group of children, 7.75 for young adults, and 21.45 for older-
adults. Hence, the difference of movement times using of the
Kinect was statically significant between the group of children
versus young adults (p = .046), and the group of young adults
versus older-adults (p = .002).
In this sense, when using the Leap Motion, the groups did not
present statistically different results concerning movement times,
which may suggest that their reaction times are not that disparate.
However, when using the Kinect sensor the groups show
inconsistency with regard to movement times.
Another aspect that we looked at was error rate. We assumed a
Figure 3. Mean movement times of each group with both speed–accuracy tradeoff approach [25] when dealing with errors,
devices as the selection rested on the “point and wait” strategy and thus
did not trigger false selections caused by user mistakes. The more
precisely the task was performed, the longer it took to be
We assessed the normality of data with the Shapiro-Wilk Test in completed, and vice-versa. Here, the selection would only be
order to understand if the data were normally distributed and triggered when the system detected the gesture to be motionless
could thus be considered for statistical analysis. We elected this and precisely on top of the target bar. Otherwise, no selection was
test over the Kolmogorov-Smirnov due to our narrow sample. As made.
we found that the data were not normally distributed across all of
the groups and since we were working with small samples, we 3.6 Indices of Performance
performed a non-parametric Mann-Whitney U test to understand Figure 18 shows the Index of Performance achieved by the three
if the devices presented significantly different results for each groups for each interaction device. The chart clearly shows the
group in terms of movement time. Indeed, every group presented children as having the worst performance of all the groups, either
a statistically significant difference between the use of each with the Kinect (IP = 1.64 bits/s) or with the Leap Motion (IP =
device. In this regard, the group of children revealed slower 2.03 bits/s); followed by the young adults with an IP of 2.98 bits/s
movement times using the Kinect sensor, with an average rank of with the Kinect, and 2.47 bits/s with the Leap Motion. The group
13.80, while the Leap Motion had an average rank of 7.20, being of older-adults presented the highest throughput regarding
this difference significant (U = 17, p = .013). Concerning the gestural interaction for target acquisition tasks with both devices:
group of the young adults, there was also a statistically significant the Kinect with an IP of 3.03 bits/s and the Leap Motion with 3.25
difference between the devices (U = 23, p = .041), but not as bits/s.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 73 |
 Performance evaluation of gesture-based interaction between different age groups using Fitts’ Law
D. Carvalho | L. Magalhães | M. Bessa | E. Carrapatoso
A Shapiro-Wilk Test showed us that the data were not normally 3.7 Participants’ Behavior and Feedback
distributed and, as such, we performed a Mann-Whitney U test to During the experiment, participants shared the same behavior
understand if there were significant differences between each concerning both devices. They were attentive to the task: trying to
group’s performance concerning the device used. None of the maintain a certain pattern with the arm’s motion and trying to
groups presented a significant difference between the performance avoid any sudden movements. The most perceptible behavior we
using each device: children (U = 34, p = .226); young adults (U = noticed was that some of the participants would push the arm
38.50, p = .384); older-adults (U = 44, p = .650). foreword in order to try to elevate the target badge on the screen,
instead of lifting it upwards. This situation would lead the
participants to a higher level of frustration with either device
because the badge would not go up unless the arm was raised.
Also, at the end of the experiment the participants answered a
questionnaire about their preferences. We discuss below the
responses with the majority of the preferences. In terms of ease of
use, 50% of the children focused their choice in that the Kinect
was relatively difficult to use, being the other answers scattered
throughout the Likert scale. In contrast, the Leap Motion had 37%
of the children saying it was relatively easy. The older-adults also
shared this opinion (40% signed the Kinect as relatively difficult;
and 45% said the Leap Motion was very easy). Although the
results do not show this tendency, some participants felt that the
Leap Motion sensor seemed easier to use than the Kinect. The
young adults stated that both devices were very easy to use (50%
of the responses regarding each device).
On the other hand, in terms of ease of learning and improvement,
Figure 4. Groups’ Index of Performance concerning both both children and young adults thought both devices were
devices relatively easy, but the older-adults argued that the Leap Motion
was easier to learn than the Kinect. Regarding fatigue, all of the
groups thought both devices were relatively demanding except the
An independent-samples Kruskal-Wallis 1-way ANOVA test was older-adults’ group that considered the Leap Motion less
also run to determine if there was a statistically significant challenging than the Kinect. Finally, all groups considered the
difference in terms of performance between the groups. Kinect as a device that requires further concentration to complete
Considering the Kinect, the performance of the children versus the trials, as opposed to the Leap Motion that was simpler in terms
young adults (p = .031) as well as versus older-adults (p = .013) of attention to the task.
was significantly different. Contrarily, the group of young adults
and older-adults did not register significant differences (p = 4. CONCLUSIONS AND FUTURE WORK
1.000). On the other hand, considering the Leap Motion sensor, This investigation was intended to understand (1) if there are age-
the only significant difference presented was between the groups related differences regarding gestural selection tasks when using
of children versus older-adults (p = .034), and not between one device to the detriment of the other; (2) if two different
children and young adults (p = .824) or young adults versus older- gestural sensors, the Kinect and the Leap Motion, could
adults (p = .453). differently influence the users’ interaction performance and target
acquisition times.
We might reach the conclusion that, in terms of performance,
neither the Kinect nor the Leap Motion sensor directly influences We are aware that conducting further tests may help ascertain
gestural interaction. However, they appear to have different whether the devices do truly influence the participant’s movement
results regarding users of different age brackets, which implies times and respective index of performance. Also, it is possible that
that these sensors could affect accuracy of interaction depending by continuing the experiment the groups’ index of performance
on the user’s age. The group that revealed the best performance would increase, and we therefore need to understand if practice
results was the older-adults’ one, which has already shown to could be considered as a way to attenuate the learning curve, and
have worse results when it came to reaction times during selection consider running more tests to set aside potential hardware
tasks. Here, we may infer that since other pointing devices like the problems on detection that could be negatively affecting accuracy.
mouse or even touch inputs may request more accuracy than
gestural commands, these allow older-adults to overcome some Nevertheless, we acknowledge it appears to be evidence that for
problems of dexterity that may exist in this age bracket, and thus selection and target acquisition tasks a gestural interface may not
demonstrate improved performances over other groups of users. be the best approach, as it shows low indices of performance
However, we will not further explore this assumption on the throughout the three groups. We may presume that for selection
course of this paper. In contrast, children may need more time tasks that require a higher level of precision neither device
practicing to overcome the learning curve to better understand the displayed proof that it influenced the user to complete the trials
relation between the physical environment of the gesture itself and with better performance. In fact, the devices’ accuracy did not
its visual feedback. appear to have a direct relation to improved performance.
As stated earlier, mean movement times should not be considered
the only variable when exploring the users’ performance, and here

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 74 |
 Performance evaluation of gesture-based interaction between different age groups using Fitts’ Law
D. Carvalho | L. Magalhães | M. Bessa | E. Carrapatoso
we can relate to why: for example, when comparing the finger, stylus and mouse. International Journal of Human-
movement times and respective index of performance, children Computer Studies. 70, 3 (Mar. 2012), 218–233.
exhibited mean times similar to the other groups, but not nearly as [5] Dam, A. van 1997. Post-WIMP user interfaces. Commun.
good performance as the others. Children have as good response ACM. 40, 2 (1997), 63–67.
times as other groups, but they do not present a consistent
interaction, which may reveal that they are faster but not as [6] Findlater, L. et al. 2013. Age-related differences in
precise when interacting with gesture-based interfaces. On the performance with touchscreens compared to traditional
other hand, the group of older-adults was indeed the one that mouse input. Proceedings of the SIGCHI Conference on
exhibited the best results concerning performance with both Human Factors in Computing Systems - CHI ’13 (New York,
devices. However, it was also the one group to present higher New York, USA, Apr. 2013), 343.
movement times, which may indicate that they have a consistent [7] Fitts, P.M. 1954. The information capacity of the human
interaction. Regardless of their movement times being not as fast motor system in controlling the amplitude of movement.
as the other groups, they tend to be more precise throughout the Journal of Experimental Psychology. 47, 6 (1954), 381–391.
several levels of difficulty of each set of trials.
[8] Goldberg, K. et al. 2013. A New Derivation and Dataset for
This is a work in progress, and more tests will be conducted to Fitts’ Law of Human Motion.
further attest these findings, but we may suggest that: (1) in terms [9] Inai, Y. 2013. Age-related differences in pointing movements
of index of performance the groups displayed statistically in restricted visual tasks and their design implication. 2013
different results: the group of older-adults held the best gestural ICME International Conference on Complex Medical
performance results, and the children the worst, when compared Engineering (May 2013), 439–443.
between each other; (2) both devices behave in a similar manner
for selection tasks and there are no statistically significant [10] ISO 2007. ISO 9241-400:2007 - Ergonomics of human--
differences concerning their accuracy when comparing each of the system interaction -- Part 400: Principles and requirements
three groups individually. However, in terms of user preference, for physical input devices: International Organization for
some participants felt that the Leap Motion sensor was easier to Standardization.
use and less demanding in terms of concentration than the Kinect. [11] ISO 2000. ISO 9241-9 International standard: Ergonomic
requirements for office work with visual display terminals
Aside from more tests to be conducted, in the future we also
(VDTs) -- Part 9: Requirements for non-keyboard input
intend to broaden our research to other elemental tasks (as
devices: International Organization for Standardization.
insertion and manipulation) using gestural interfaces and observe
performance considering our targeted age groups. Also, we intend [12] Lemos, G.C.E.M.P. de 2006. Habilidades cognitivas e
to follow other evaluation methods and test the 7 measurements of rendimento escolar entre o 5.o e 12.o anos de escolaridade.
McKenzie [14] and Steering Law [1] for tracking performance Universidade do Minho.
evaluation for these pointing devices. [13] Likert, R. 1932. A technique for the measurement of
attitudes. Archives of Psychology. 22 140, (1932), 55.
5. ACKNOWLEDGMENTS
We would like to acknowledge the support and contribution of [14] MacKenzie, I.S. et al. 2001. Accuracy measures for
“Universidade de Trás-os-Montes e Alto Douro” and the schools evaluating computer pointing devices. Proceedings of the
that took part in our study: “Monsenhor Jerónimo do Amaral”, SIGCHI conference on Human factors in computing systems
“Escola Secundária Morgado de Mateus”, and the studies center - CHI ’01 (New York, New York, USA, Mar. 2001), 9–16.
“Super-Heróis”, all in Vila Real, Portugal. [15] MacKenzie, I.S. 1992. Fitts’ Law as a Research and Design
Diana Carvalho has a PhD fellowship granted by FCT – Fundação Tool in Human-Computer Interaction. Human-Computer
para a Ciência e a Tecnologia (SFRH/BD/81541/2011). Interaction. 7, 1 (Mar. 1992), 91–139.
[16] Mayer, R.E. and Moreno, R. 2003. Nine Ways to Reduce
6. REFERENCES Cognitive Load in Multimedia Learning. Educational
[1] Accot, J. and Zhai, S. 1999. Performance evaluation of input Psychologist. 38, 1 (Mar. 2003), 43–52.
devices in trajectory-based tasks. Proceedings of the SIGCHI
[17] Nielsen, J. 1993. Noncommand user interfaces.
conference on Human factors in computing systems the CHI Communications of the ACM. 36, 4 (Apr. 1993), 83–99.
is the limit - CHI ’99 (New York, New York, USA, May
1999), 466–472. [18] Osborne, J.W. and Overbay, A. 2004. The power of outliers
(and why researchers should always check for them).
[2] Bobeth, J. et al. 2014. Tablet, gestures, remote control? Practical Assessment, Research & Evaluation. 9, 6 (2004).
Proceedings of the 2014 ACM international conference on
Interactive experiences for TV and online video - TVX ’14 [19] Oviatt, S.L. 2007. Multimodal Interfaces. The Human-
(New York, New York, USA, Jun. 2014), 139–146. Computer Interaction Handbook: Fundamentals, Evolving
Technologies and Emerging Applications, Second Edition. A.
[3] Carvalho, D. et al. 2014. Different interaction paradigms for Sears and J.A. Jacko, eds. CRC Press. 413–432.
different user groups: an evaluation regarding content
selection. Proceedings of the XV International Conference on [20] Ren, Z. et al. 2011. Robust hand gesture recognition with
Human Computer Interaction - Interaccion’14 (Puerto de la kinect sensor. Proceedings of the 19th ACM international
Cruz, Tenerife, Spain, Sep. 2014), 40. conference on Multimedia - MM ’11 (New York, New York,
USA, Nov. 2011), 759.
[4] Cockburn, A. et al. 2012. Understanding performance in
touch selections: Tap, drag and radial pointing drag with [21] Sambrooks, L. and Wilkinson, B. 2013. Comparison of
gestural, touch, and mouse interaction with Fitts’ law.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 75 |
 Performance evaluation of gesture-based interaction between different age groups using Fitts’ Law
D. Carvalho | L. Magalhães | M. Bessa | E. Carrapatoso
Proceedings of the 25th Australian Computer-Human years of Fitts’ law research in HCI. International Journal of
Interaction Conference on Augmentation, Application, Human-Computer Studies. 61, 6 (Dec. 2004), 751–789.
Innovation, Collaboration - OzCHI ’13 (New York, New [24] Weichert, F. et al. 2013. Analysis of the accuracy and
York, USA, Nov. 2013), 119–122. robustness of the leap motion controller. Sensors (Basel,
[22] Schapira, E. and Sharma, R. 2001. Experimental evaluation Switzerland). 13, 5 (Jan. 2013), 6380–93.
of vision and speech based multimodal interfaces. [25] Zhai, S. et al. 2004. Speed–accuracy tradeoff in Fitts’ law
Proceedings of the 2001 workshop on Percetive user tasks—on the equivalency of actual and nominal pointing
interfaces - PUI ’01 (New York, New York, USA, Nov. precision. International Journal of Human-Computer
2001), 1. Studies. 61, 6 (Dec. 2004), 823–856..
[23] Soukoreff, R.W. and MacKenzie, I.S. 2004. Towards a
standard for pointing device evaluation, perspectives on 27

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 76 |
 Including multi-stroke gesture-based interaction in user interfaces using a model-driven method
O. Parra González | S. España | M. Bessa | O. Pastor

Including multi-stroke gesture-based interaction in user

interfaces using a model-driven method
Otto Parra González 1 Sergio España Oscar Pastor
1
Universidad de Cuenca Universitat Politècnica de València Universitat Politècnica de València
Av. 12 de Abril s/n y A. Cueva Camino de Vera s/n Camino de Vera s/n
01.01.168, Cuenca, Ecuador 46022, Valencia, Spain 46022, Valencia, Spain
+593 07 405 1000 +34 96 387 7000 +34 96 387 7000
[email protected] [email protected] [email protected]

of gesture-based capabilities has to be done entirely in the source

ABSTRACT code by programmers with specific technical skills [3].
Technological advances in touch-based devices now allow users
to interact with information systems in new ways, being gesture- With the aim of overcoming this situation, this paper introduces a
based interaction a popular new kid on the block. Many daily Model-Driven Development (MDD) approach to gesture-based
tasks can be performed on mobile devices and desktop computers information systems interface development. The method is
by applying multi-stroke gestures. Scaling up this type of intended to allow software engineers to focus on the key aspects
interaction to bigger information systems and software tools of gesture-based information system interfaces; namely, defining
entails difficulties, such as the fact that gesture definitions are gestures and specifying gesture-based interaction. Coding and
platform-specific and this interaction is often hard-coded in the portability efforts are alleviated by means of model-to-text (M2T)
source code and hinders their analysis, validation and reuse. In an transformations. We focus on multi-stroke gestures as they are
attempt to solve this problem, we here propose gestUI, a model- expressive and currently wide-spread.
driven approach to the multi-stroke gesture-based user interface Our proposal has the following benefits: (a) the solution is
development. This system allows modelling gestures, integrated with the existing source code of user interfaces; (b) the
automatically generating gesture catalogues for different gesture- solution can be used on any target platform (platform-
recognition platforms, and user-testing the gestures. A model independence).
transformation automatically generates the user interface
components that support this type of interaction for desktop We also implemented a tool prototype to support our approach
applications (further transformations are under development). We that defines multi-strokes gestures and includes them in a user
applied our proposal to two cases: a form-based information interface. This tool prototype is coded in Java and Eclipse
system and a CASE tool. We include details of the underlying Modelling Framework.
software technology in order to pave the way for other research The contributions of this paper are the following:
endeavours in this area.
• We present gestUI, an MDD method for gesture-based
Categories and Subject Descriptors IS interface development consisting of: (i) a modelling
H.5.2 [Information Interfaces and Presentation]: User language to represent multi-stroke gestures, and (ii) a
interfaces. – Graphical User Interfaces, Interaction Styles, Input set of multi-platform model transformations.
devices and strategies.
• We provide a tool support for the method that captures
General Terms multi-stroke gestures sketched by the users, transforms
Design, Experimentation, Human Factors, Verification. them into a model, and automatically generates (i) the
gesture catalogue and (ii) the source code of the gesture-
Keywords based IS interface.
Customised gesture, model-driven engineering, user interface,
gesture-based interaction. • We validate our approach by applying it to two cases in
different types of information systems for desktop-
1. INTRODUCTION computing.
Natural user interface (NUI) considers the use of many different This paper is organized as follows: Section 2 includes the
interaction modalities, including multi-touch, motion tracking, definitions of items contained in the proposal. Section 3 reviews
voice, and stylus [1]. Users interact with computers employing related work. Section 4 describes our proposal. Section 5
intuitive action such as touching, gesturing and speaking. introduces the tool prototype designed to apply this approach.
Several issues may hinder the wide adoption of gesture-based Section 6 describes its application in two cases and Section 70
interaction in complex information systems engineering. contains our conclusions and projected future work.
Currently, gesture-based interaction is limited to its specification
in the implementation stage in the software development lifecycle
2. BACKGROUND
(SDLC) using tools to write source code. Gesture-based interfaces 2.1 Stroke gestures
have been reported to be more difficult to implement and test than
Nacenta et al. [4] demonstrate that users prefer user-defined
traditional mouse and pointer interfaces [2], yet the development
gestures rather than stock and pre-defined gestures. Although
user-defined gestures offer better memorability, efficiency and

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 77 |
 Including multi-stroke gesture-based interaction in user interfaces using a model-driven method
O. Parra González | S. España | M. Bessa | O. Pastor
accessibility than pre-defined gestures, they have received little use of model-driven engineering in HCI to design user interfaces
attention in the literature [5]. According to the taxonomy of with this type of interaction.
gestures in [6], semaphoric gestures refer to strokes or marks
Aquino et al. [13] emphasize the importance of interaction
made with a mouse, pen or finger. This type of gesture is further
classified as single-stroke and multi-stroke, according to the modelling on the same level of expressiveness as any other model
number of strokes required to sketch them (see Fig. 1). involved in the development life cycle of an interactive
application. They define the presentation model of the OO-
Method as an abstract model from which the model compiler can
automatically generate a user interface for different interaction
modalities and platforms, although they do not include an explicit
reference to a type of interaction modality (e. g., graphical, vocal,
tactile, haptic, and multimodal).
Deshayes et al. [14] propose the use of MDE and model execution
in the context of human-computer interaction (HCI) by means of
Figure 1. Types of semaphoric gestures heterogeneous models obtained with the ModHel’X modelling
environment for developing a simple HCI application for gesture-
based interaction. Their application makes it possible to browse a
According to [7] a stroke gesture is commonly encoded as a time- virtual book using gestures (e.g., swiping, moving) in Microsoft
ordered sequence of two-dimensional points with coordinates (x, Kinect.
y). Optionally, stroke gestures can also have time stamps as the
third dimension so the sampling points are encoded as (x, y, t) if Vanderdonckt [15] describes a set of variables related to the
the temporal aspect of a gesture is to be preserved and used. In development of user interfaces, one of which considers interaction
this work, stroke gestures are used to issue commands, which are devices and styles. Interaction styles include the gesture
the names of the executable computing functions issued by the recognition. However, he point out that an abstract user interface
user. is independent of any interaction modality [16] so that an explicit
reference to a specific type of interaction style is not considered.
2.2 Model related definitions
A model is a description of a system or part of a system written in [17] includes a report regarding user interface plasticity and MDE,
a well-defined language. A well-defined language is a language in which three information spaces are defined: the user model,
with well-defined form (syntax), and meaning (semantics), which environment model, and platform model. The platform model
is suitable for automated interpretation by a computer [8]. A considers the possible interactions that can be included in a user
model, both source and target, is expressed in a language [9], for interface. This report also includes a description of models that
example, XML. have been defined with the aim of creating user interfaces. It also
mentions the variety of interaction modalities currently available
Model-Driven Architecture® (MDA®) is an architectural thanks to the diversity of technological spaces that can be
framework for software development. One of its fundamental included in the definition of concrete user interfaces.
aspects is its ability to address the complete development
lifecycle, covering analysis and design, programming, testing, Calvary et al. in [18] describe the relation between MDE and HCI
component assembly as well as deployment and maintenance. in implementing user interfaces. In this context, they introduce the
MDA specifies three default viewpoints on a system: computation models contained in a number of frameworks (e g., UsiXML,
independent, platform independent and platform specific. MDA CTTe), one being the interaction model considered in the process
specifies three default models of a system corresponding to these of defining user interfaces. However, the interaction modality is
three viewpoints: computation independent model (CIM), not considered.
platform independent model (PIM) and a platform specific model In [19], the authors propose the Abstract Interaction Model that is
(PSM) [10]. added to the Presentation Model in the OO-Method. Two sub-
Model-Driven Engineering (MDE) describes software models are considered to define the Interaction Model: the user
development approaches in which abstract models of software are model and abstract interface model. A set of interaction
created and systematically transformed into concrete components is defined in the abstract interface model that define
implementations [11]. Model-driven development (MDD) its interface with the software system. These components are
automates the transformation of models from one form to another. conceptual modelling elements that describe the interaction
behaviour expected by the user but not how it is implemented, so
Model-driven describes an approach to software development that this system does not include the interaction modality in the
whereby models are used as the primary source for documenting, process of user interface generation.
analysing, designing, constructing, deploying and maintaining a
system [10]. All the works cited above mention the importance of using MDE
and HCI to obtain user interfaces in a short time at a low cost.
Model-to-Text (M2T) transformation is a model management Although they also point out the need for a specification of an
operation that involves generating text (e.g., source code, interaction modality, they do not include gestures in their
documentation, configuration files) from models [12]. proposals. We considered gesture-based interaction in this
proposal in order to obtain a complete definition of user interfaces
3. RELATED WORK using MDE and HCI.
This section briefly reviews the role of model-driven engineering
(MDE) in HCI in which models are used to create a user interface
that includes user interaction. Several studies have reported on the
XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)
| 78 |
 Including multi-stroke gesture-based interaction in user interfaces using a model-driven method
O. Parra González | S. España | M. Bessa | O. Pastor

4. OUR PROPOSAL precedence order. Each posture in a gesture is related to a figure

This section describes gestUI [20], a user-driven iterative method (line, circle, etc.) with an orientation (up, down, left, right), and is
based on the MDD paradigm. The system defines multi-strokes qualified by a state (initial, executing, final).
gestures, creates a gesture catalogue model and using model In the platform-specific layer there are two activities: (i) Activity
transformations to obtain the interface source code, including A2, “Design gesture-based interaction” which consists of a
gesture-based interaction. gestUI is expected to be integrated into process to define the relationship between a gesture and an action
a full interface development method (shown in Fig. 2 by generic (command) to be executed in an information system. This activity
activities and artefacts in grey). The method can either be model- produces the gesture-based interaction model. If the stakeholders
driven or code-centric. gestUI is user-driven because the users consider that the definition of gestures is incomplete or inadequate
participate in all non-automated activities and is iterative because the process can return to Activity 1 to be redefined. (ii) Activity
it aims to discover the necessary gestures incrementally and A3, “Generate gesture specification”, employs an M2T
provides several loopbacks. transformation to generate a platform-specific gestures catalogue
Our proposal is based on Model-driven Architecture (MDA) specification.
considering three layers: the platform-independent layer, There are two activities in the code layer: (i) Activity A4,
platform-specific layer and code layer. gestUI consists of five “Generate gesture-based interface”, which employs the gesture-
activities and their products which are distributed in the three based interaction model and the platform-specific gestures
layers, as shown in Fig. 2. catalogue specification to obtain the source code of the gesture-
In the platform-independent layer, during the A1 activity, “Define based user interface by applying a model transformation. (ii)
gestures”, the stakeholders and developers have meetings to Activity 5, “Test gestures”, tests the gestures defined by the
discuss gesture definition for the user interface. The stakeholders stakeholders and included in the information system. This is an
try the gestures in the interface implemented in the tool to verify iterative process because it is possible to redefine the gestures if
that these gestures are suitable for the tasks in their information the users are not satisfied with their definition.
system. Finally, a gesture catalogue model is obtained conforms If the gesture catalogue is complete and the users agree with the
to the gesture catalogue metamodel (Fig. 3). In the gesture definition of the correspondence gesture-action in the information
catalogue each gesture is formed by one or more strokes defined system (IS), then the process is finished. The IS then contains
by postures, described by means of coordinates (X, Y). The gesture-based interaction as a complementary interaction modality
sequence of strokes in the gesture is specified by means of to the traditional interaction.

Figure 2. gestUI method overview

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 79 |
 Including multi-stroke gesture-based interaction in user interfaces using a model-driven method
O. Parra González | S. España | M. Bessa | O. Pastor

Figure 3. Gesture catalogue metamodel

Figure 4. gestUI tool overview

5. TOOL The third module “Model transformation” contains the definition

This section describes the tool that implements our proposal. This of the M2T transformations for some platforms as targets for
tool is structured in three modules as shown in Fig. 4. The gesture-based interaction in the user interface. Activity A3 is
number in brackets indicates the method activity each component included in this module.
supports. The method’s internal products are not shown. The The tool prototype is developed in Java and Eclipse and permits
relationship with the external gesture recogniser is represented. the proposal to be applied by means of its modules: (i) to define
The first module “Gesture catalogue definition” contains a process multi-strokes gestures, (ii) to obtain a gesture catalogue model
that obtains the gesture catalogue model from the users’ gesture using model transformations, (iii) to define gesture-action
definition made by the users. Activity A1 is included in this correspondence and, (iv) to generate source code to include
module. gesture-based interaction in user interfaces. At present, the tool
generates Java source code embedded into the IS interface source
The second module “Gesture-Action correspondence definition” code.
includes a process to specify the correspondence between gestures
and actions (commands) available in the IS. Activities A2 and A4 6. DEMONSTRATING THE PROPOSAL
are included in this module. This section describes an application of the method with the
implemented tool to include gesture-based interaction in existing

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 80 |
 Including multi-stroke gesture-based interaction in user interfaces using a model-driven method
O. Parra González | S. España | M. Bessa | O. Pastor
information systems (IS). gestUI is integrated into a code-centric read the information of a record. Using these gestures, we obtain
interface development method considering two legacy desktop- the gesture catalogue model by means of a model transformation.
computing IS: forms-based IS and case tools.
The process of obtaining a user interface with gesture-based
interaction involves the following steps:
a) Define the gestures according to the users’
requirements, and then obtain the gesture catalogue
model.
Figure 6. Gestures defined for form-based IS
b) Obtain the platform-specific gesture catalogue by means
of model transformation using the gesture catalogue Therefore, in order to define the gesture catalogue model the user
model and the target platform as source for the selects gestures from a gesture repository containing all the
transformation. gestures that the user has defined. This catalogue contains the
description of each gesture, but does not contain the action
c) Specify the user interface to which we will add the (command) definition to execute it (in Fig. 7 the action is
gesture-based interaction by definition of gesture-action assigned to “null”) because this feature (to assign an action) is
correspondence. The user interface source code which platform-specific must be specified in the next step.
includes the gesture-based interaction is generated.
d) Add the generated source code to the existing user
interface source code and recompile the system.

6.1 Form-based Information System

In the first case, we consider a form-based IS to manage
information for the computing department of a business company,
formed by departmental offices that have offices. In each
department there are employees who use computers with one or
more operating systems and printers.
The IS manages the information of: (i) the company, (ii)
departments, (iii) devices (computers, printers), (iv) the operating
systems installed in the computers, (v) the operation status of the
devices, (vi) the employee that uses the devices. For the sake of
brevity, we will only consider the option to manage information of
operating systems in the process of inclusion of gesture-based
interaction. Fig. 5 shows the domain class diagram of this IS.

Figure 7. Gesture catalogue model

Figure 5. Domain class diagram of the form-based IS

This software, written in Java using Swing in Eclipse, employs a
three-tier architecture: data access logic, business logic, and
presentation logic [20]. The set of operations that can be executed
in each component are called CRUD operations (create, read,
update, and delete). In this case we need to define gestures in Figure 8. Obtaining platform-specific gesture specification
order to execute these operations in each option (component) of
the IS.
The first step is the creation of new gestures using the interface The next step is the generation of platform-specific gesture
implemented for this task, (see in Fig. 6). The users define specification by choosing the gestures that are used in the IS (Fig.
gestures according to their preferences in order to execute actions 8). Using this specification and entering the target platform ($N)
in an information system. For instance, in a database (a) C, to and the target language (XML) as input data we apply a model
create a new record to enter in the IS, (b) U, to update the transformation to obtain the gestures to be used in the gesture
information of a record, (c) D, to delete a record, and (d) R, to recognition process using $N [21].

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 81 |
 Including multi-stroke gesture-based interaction in user interfaces using a model-driven method
O. Parra González | S. España | M. Bessa | O. Pastor
Fig. 9 shows an excerpt from a gesture definition generated in
XML to be used with $N. In this case, the gesture which
represents the letter “D” is formed by two strokes (shown in Fig.
6), each of which contains points with coordinates (X, Y) and a
timestamp (T).

Figure 11. Gesture-Action correspondence definition

When generating the user interface Java source code, many
references are included: (i) references to libraries to manage
gestures (e.g. listeners to manage mouse events such as move,
down, up, release, drag), (ii) references to the libraries of the
gesture-recognition technology (e.g. $N), (iii) references to the
method of executing the gesture-action correspondence (i.e.
executingActions in Fig. 12), (iv) references to the methods of
capturing gestures (i.e. mouseDragged, mouseReleased), and (v)
references to the method of loading the multi-stroke gesture
catalogue.
Figure. 9. An excerpt from a gesture written in XML
The next step concerns with the definition of the gesture-action
correspondence in order to complete the gesture definition. In this
case, the process consists of the selection of a user interface
source code to apply a parsing process and determine the actions
to be included in this source code. We consider code which
contains methods in Java containing “action perform” structures
(Fig. 10), which typically define the actions to be executed by
selecting buttons or options in the menu in a form-based IS.
Figure 12. An excerpt of source code to execute actions
Additionally, the definition of some classes have been changed in
order to implement mouseListener and mouseMotionListener to
detect mouse events.
Finally, when we execute the form-based IS (Fig. 13), we use the
gestures included in the source code to execute the actions
specified in the process. In this case, we can use traditional
interaction and gesture-based interaction.

Figure 10. An excerpt of source code of user interface

The developer assign the gesture-action correspondence in
collaboration with the user, supported by the Gesture-action
correspondence definition module. The process consist in select
one gesture (Fig. 11, left) for each action specified in the user Figure 13. Form-based IS with gesture-based interaction
interface (Fig. 11, centre). The correspondence gesture-action is
defined (Fig. 11, right). The next step is the source code
generation by selecting button “Generate” in the interface shown In the form-based IS shown in Fig. 13, the user draws the gesture
in Fig. 11. In this case, we obtain Java source code to include “C” with the aim of creating a new operating system record.
gesture-based interaction in the user interface source code When the gesture recognizer analyses the gesture sketched by the
user as correct, then a new window for the user to fill in the form
related to the operating system details.

6.2 Case tool

Now, we describe the application to the second type of IS: in this
case, we consider Graphical Editing Framework (GEF), an
Eclipse project which provides a development framework for
graphical representation of information. The GEF project is

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 82 |
 Including multi-stroke gesture-based interaction in user interfaces using a model-driven method
O. Parra González | S. España | M. Bessa | O. Pastor
developed in JFace and Standard Widget Toolkit (SWT) with different actions: (i) in the form-based IS to create a record, and
Eclipse Modelling Framework. GEF is designed with a Model- (ii) in the case tool to draw a rectangle.
View-Controller (MVC) architecture. Briefly, the function of each
Using the process described in this paper, we obtain the gesture
component is [22]:
catalogue model, then in the next step we get the platform-specific
• Model: holds the information (underlying objects being gesture specification and finally, the definition of gestures using
represented graphically) to be displayed and is persisted XML to use in the $N gesture recognizer.
across sessions.
The gesture-action correspondence definition is executed in the
• View: renders information on the screen and provides source code of Shapes. Fig. 14 includes an excerpt from the
basic user interaction, in this case, a canvas containing a source code that contains the reference to the aforementioned
collection of figures. actions: draw rectangles and ellipses.
• Controller: coordinates the activities of the model and
the view, passing information between them as
necessary. The controller has a collection of GEF edit
parts.
According to [22], the GEF project is formed by three
subsections:
Figure 14. An excerpt of source code of Shapes
• Draw2D, a lightweight framework layered on top of We apply the process of parsing in this source code to determine
SWT for rendering graphical information. the actions defined to draw the elements contained in the palette.
• GEF Framework, an MVC framework layered on top of The user selects the gestures in the catalogue (Fig. 15, left), and
Draw2D to facilitate user interaction with graphical defines the relation with the actions that were found in the parsing
information. process (Fig. 15, centre). Finally, the gesture-action
correspondence is obtained (Fig. 15, right).
• Zest, a graphing framework layered on top of Draw2D
for graphing.
In this work, we employ the example called Shapes, included in
GEF project, to apply the method. Considering that GEF has a
plug-in structure, then the Shapes source code is comprised by
two components: (i) source code of GEF which provides the
functionality of an editor (to display and draw elements in a
diagram), and, (ii) source code of the example, which define the
canvas and the palette of the tool and employs the methods Figure 15. Gesture-action correspondence definition
defined in the plug-in to use the editor. The palette defined in When generating the user interface Java source code, many
Shapes permits rectangles and ellipses to be drawn. Therefore, we references are included; for instance, in the GEF source code we
need to modify the View component both in the source code of add references to gesture management libraries (e.g. listeners to
GEF and the example Shapes, in order to define the finger manage mouse events such as mouseUp and mouseDrag), to the
gestures for drawing ellipses and rectangles. gesture-recognition technology libraries (e.g. $N) and the method
We describe below the process applied to the GEF framework in of executing the gesture-action correspondence (Fig. 16).
order to include gesture-based interaction in this case tool. Also, in the methods for managing mouse events we add source
Table 2. Gestures and elements related code to capture gestures (i.e. mouseDrag (Fig. 17), mouseUp),
and the method of loading the multi-stroke gesture catalogue is
Gestures Element related added.

Rectangle

Ellipse

Figure 16. An excerpt of source code to execute actions

Table 13 shows the gestures used to draw the elements defined in

the Shapes palette. In the process of defining gestures, in some
cases it is possible to “reuse” gestures that the users have sketched
previously, for instance, we use the gesture “C” to execute

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 83 |
 Including multi-stroke gesture-based interaction in user interfaces using a model-driven method
O. Parra González | S. España | M. Bessa | O. Pastor
We then automatically generated the final gesture-based interface
components and integrated them into the application interfaces.
The advantages of the gestUI proposal are: its platform
independence enabled by the model-driven development
paradigm, the convenience of including user-defined symbols and
its iterative and user driven approach.
Figure 17. An excerpt of the source code of GEF Future work will be developed along the following lines: (i) a user
We then consider the source code of the example in order to add study to determine user preferences in defining gestures according
the option of including gesture-based interaction in the palette. In to the task to be executed, (ii) applying this method to extending
the source code containing the definition of the palette we add the the Capability Design Tool (CDT) of the “Capability as a Service”
source code shown in Fig. 18. The option added is shown in a red Project (CaaS project) in order to incorporate gesture-based
circle in Fig. 19. The users thus have two styles of interaction in interaction into this framework.
GEF: (i) traditional interaction (using mouse and keyboard) and
(ii) gesture-based interaction (using fingers to sketch gestures). 8. ACKNOWLEDGMENTS
Otto Parra is grateful to his supervisors Sergio España and Óscar
Pastor for their invaluable support and advice. This work has been
supported by Secretaría Nacional de Educación, Ciencia y
Tecnología (SENESCYT) and Universidad de Cuenca of Ecuador,
and received financial support from Generalitat Valenciana under
Project IDEO (PROMETEOII/2014/039).

Figure 18. An excerpt from the source code to modify the 9. REFERENCES
palette [1] D. Wigdor y D. Wixon, Brave NUI world: designing natural
user interfaces for touch and gesture, USA: Morgan
Kaufmann Publishers Inc., 2011.
Obviously, then the source code should be compiled to obtain the [2] M. Hesenius, T. Griebe, S. Gries y V. Gruhn, «Automating
case tool with gesture-based interaction included. UI Tests for Mobile Applications with Formal Gesture
When the user employs the gestures defined in this case tool to Descriptions,» Proc. of 16th Conf. on Human-computer
draw an ellipse/rectangle on the canvas using his/her finger, the interaction with mobile devices & services, pp. 213-222,
gesture recognizer analyses the sketch. If it is correct, then the 2014.
system draws an ellipse/rectangle defined in the GEF palette. [3] K. Gerken, S. Frechenhäuser, R. Dörner y J. Luderschmidt,
«Authoring Support for Post-WIMP Applications,» Human-
Computer Interaction – INTERACT 2013. Lecture Notes in
Computer Science, vol. 8119, pp. 744-761, 2013.
[4] M. Nacenta, Y. Kamber, Y. Qiang y P. Kristensson,
«Memorability of Pre-designed & User-defined Gesture
Sets,» Proc. of the SIGCHI Conf. on Human Factors in
Computing Systems, pp. 1099-1108, 2013.
[5] U. Oh y L. FindLater, «The challenges and potential of end-
user gesture customization,» Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems -
CHI'13, pp. 1129-1138, 2013.
[6] M. Karam y M. C. Schraefel, «A taxonomy of Gestures in
Human-Computer Interaction,» de Retrieved from
http://eprints.soton.ac.uk/261149/, 2005.
[7] S. Zhai, P. O. Kristensson, C. Appert, T. H. Anderson y X.
Figure 19. Case tool with gesture interaction Cao, «Foundational Issues in Touch-Surface Stroke Gesture
Design — An Integrative Review,» Foundations and Trends
7. CONCLUSIONS AND FUTURE WORK in Human–Computer Interaction, vol. 5, nº 2, pp. 97-205,
gestUI is a model-driven method for developing multi-stroke 2012.
gesture-based user interfaces. We validated the method and [8] A. Kleppe, J. Warmer y W. Bast, MDA Explained: The
supporting tools by applying them to two types of applications: a Model Driven Architecture : Practice and Promise, USA:
form-based application and a CASE tool built using GEF (an Addison-Wesley Prof., 2003.
Eclipse framework with the structure of a plug-in) that can be
used to draw simple diagrams. We generated the ‘Platform- [9] S. J. Mellor, T. Clark y T. Futagami, «Model-driven
specific gesture specification’ for the two types of applications to development: guest editors' introduction,» IEEE Software,
illustrate the multiplatform capability of the approach. The vol. 20, nº 5, pp. 14-18, 2003.
gestures were successfully recognised by the corresponding tools.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 84 |
 Including multi-stroke gesture-based interaction in user interfaces using a model-driven method
O. Parra González | S. España | M. Bessa | O. Pastor
[10] F. Truyen, «The Fast Guide to Model-Driven Architecture. [17] J. Coutaz y G. Calvary, «HCI and Software Engineering for
The Basics of Model-Driven Architecture,» Cephas User Interface Plasticity,» de The Human-Computer
Consulting Corp., 2006. Handbook - Fundamentals, Evolving Technologies, and
[11] R. France y B. Rumpe, «Model-Driven Development of Emerging Applications, Julie, A. Jacko ed., CRC Press
Complex Software: A Research Roadmap,» Future of Taylor and Francis Group, 2012, pp. 1195-1220.
Software Engineering - ICSE, pp. 37-54, 2007. [18] G. Calvary, D.-P. A., A. Occello, R.-G. P. y M. Riveill, «At
[12] B. Ogunyomi, L. M. Rose y D. Kolovos, «On the Use of the Cross-Roads between Human-Computer Interaction and
Signatures for Source Incremental Model-to-text Model-Driven Engineering,» ARPN Journal of Systems and
Transformation,» Model-Driven Engineering Languages and Software, vol. 4, nº 3, pp. 64-76, 2014.
Systems. Lecture Notes in Computer Science, vol. 8767, pp. [19] F. Valverde, J. I. Panach y O. Pastor, «An Abstract
84-98, 2014. Interaction Model of a MDA Software Production Method,»
[13] N. Aquino, J. Vanderdonckt, J. I. Panach y O. Pastor, de Twenty-Sixth International Conference on Conceptual
«Conceptual Modelling of Interaction,» de Handbook of Modeling - ER 2007 - Tutorials, Posters, Panels and
Conceptual Modeling. Theory, Practice, and Research Industrial Contributions, Auckland, New Zeland, 2007.
Challenges, Berlin, Springer, 2011, pp. 335-358. [20] O. Parra, S. España y O. Pastor, «A Model-driven Method
[14] R. Deshayes, C. Jacquet, C. Hardebolle, F. Boulanger y T. and a Tool for Developing Gesture-based Information
Mens, «Heterogeneous modeling of gesture-based 3D Systems Interface,» de Proceedings of the CAiSE'15 Forum
applications,» de Proceedings of the 6th International at the 27th International Conference on Advanced
Workshop on Multi-Paradigm Modeling, 2012. Information Systems Engineering, Stockholm, Sweden, 2015.

[15] J. Vanderdonckt, «A MDA-Compliant Environment for [21] M. Fowler, Patterns of Enterprise Application Architecture,
Developing User Interfaces of Information Systems,» Boston, MA: Addison-Wesley Professional, 2003.
Advanced Information Systems Engineering LNCS in [22] L. Anthony y J. O. Wobbrock, «A Lightweight Multistroke
Computer Science, vol. 3520, pp. 16-31, 2005. Recognizer for User Interface Prototypes,» Proceedings of
[16] J. Vanderdonckt, «Model-Driven Engineering of User Graphics Interface (GI '10), pp. pp. 245-252, 2010.
Interfaces: Promises, Successes, Failures, and Challenges,» [23] E. Clayberg y D. Rubel, Eclipse Plug-ins, Massachusetts,
de ROCHI'08, Iasi, Romania, 2008. USA: Addison-Wesley, 2009.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 85 |
 Interfaces de Usuario Tangibles como Mecanismo de Interacción en Entornos Multi-dispositivos
E. de la Guía | M. D. Lozano | V. Penichet | L. Orozco | V. López

Interfaces de Usuario Tangibles como Mecanismo de

Interacción en Entornos Multi-dispositivos
Elena de la Guía, María D. Lozano, Víctor Penichet, Luis Orozco, Vicente López
Computer Science Research Institute
University of Castilla-La Mancha
Campus Universitario, Albacete - SPAIN
{mariaelena.guia, maria.lozano, victor.penichet, luis.orozco, vicente.lcamacho}@uclm.es

ABSTRACT Sheiderman[1] a la manipulación táctil que permite digitalizar

En los últimos años la tecnología ha crecido exponencialmente nos objetos y entornos reales, mezclando la parte digital con la real.
encontramos escenarios donde se trabaja con múltiples Actualmente se están insertado las interfaces de usuario tangibles
dispositivos, como son móviles, Tablet PC, portátiles, wereables, como mecanismo de interacción. Cuando hablamos de interfaces
etc. interactuando entre sí para ofrecer servicios fácilmente. Hoy de usuario tangibles (TUI o Tangible User Interface en inglés) [2]
en día encontramos mecanismos de interacción tangible y natural nos referimos a objetos físicos utilizados como representaciones y
que permiten fusionar el mundo real y digital simulando entornos controles de la información digital. De esta forma conseguimos
reales y familiares a los usuarios. Sin embargo, la diversidad de combinar dispositivos digitales con objetos reales creando
usuarios finales se ha incrementado, es decir, nos encontramos escenarios más familiares e intuitivos para el usuario.
usuarios con diferentes habilidades cognitivas, conocimientos, Ahora que tenemos entornos físicos reales surgen las siguientes
discapacidades, etc. Actualmente, el primer desafío es como preguntas. ¿Qué directrices son recomendables seguir para
diseñar y distribuir las interfaces/información alrededor del distribuir interfaces de usuario en los diferentes
entorno sin saturar al usuario. El siguiente desafío es como dispositivos/objetos/usuarios de la manera más eficaz posible?
motivar y guiar al usuario alrededor de entornos MDE sin La organización del artículo está definida de la siguiente manera:
distraerlo de su tarea u objetivo principal. Teniendo en cuenta que Primero se presenta un breve repaso sobre las interfaces de usuario
la interacción tangible es más familiar y fácil de aprender para los distribuidas y las técnicas de interacción tangible utilizadas en
usuarios que no tienen experiencia con la tecnología se han entornos multi-dispositivo. A continuación se describen los
desarrollado y evaluado cuatro prototipos. Son juegos MDE prototipos realizados basados en interfaces de usuario distribuidas
basados en interfaces de usuario distribuidas y tangibles. Después e interacción tangible. Por último, se presentan las
de evaluar los prototipos, analizar los datos y basándonos en las recomendaciones y directrices obtenidas gracias a las evaluaciones
lecciones aprendidas se han propuesto 8 directrices iniciales. El y lecciones aprendidas de los prototipos desarrollados.
objetivo principal es servir de guía para diseñar e investigar dichos
entornos. Las 8 directrices se han basado en los siguientes
2. TRABAJOS RELACIONADOS
elementos (espacio físico/ múltiples usuarios/ motivación). La rápida evolución de la tecnología ha cambiado la forma en que
interactuamos con sistemas interactivos. En los últimos años han
Categories and Subject Descriptors surgido entornos dotados de múltiples dispositivos, también
H5.2. Information interfaces and presentation: User Interfaces. – denominados MDE (Multi-Device Environment). Estos escenarios
Graphical user interfaces. soportan interfaces de usuario distribuidas. Esta se define como
una interfaz de usuario donde sus componentes pueden ser
General Terms distribuidos a través de una o más dimensiones como puede ser
Design, Experimentation, Human Factors, entrada, salida, plataforma, espacio y tiempo [3]. Según González
en [4], las interfaces de usuario Distribuidas (DUI) se pueden
Keywords clasificar en función de las características de la interfaz en un
Interfaces de Usuario Distribuidas, Interacción Tangible MDE. Las interfaces pueden ser divididas o distribuidas en el
entorno físico de acuerdo a sus propiedades. El objetivo principal
1. INTRODUCCIÓN de una DUI es facilitar las tareas a los usuarios en el sistema de
Hace años la interacción persona ordenador se basaba en software, poniendo a su disposición una configuración óptima de
interfaces de usuario controladas a través de un ratón y un teclado interfaz de usuario que están disponibles en el entorno de trabajo
conectados al ordenador. de los usuarios. Según Vandervelpen y Coninx en [6] los
Hoy en día nos encontramos con escenarios dotados de múltiples elementos que forman parte de un entorno de distribución de
dispositivos donde las interfaces de usuario tradiciones se quedan interfaces son los siguientes: Recursos de interacción (RIs) son
limitadas y surge la necesidad de estudiar e investigar nuevos canales de I/O disponibles para que el usuario ejecute una tarea.
mecanismos de interacción e interfaces que se adapten al entorno Incluye recursos de I/O como teclados, pantallas, micrófonos, etc.
y al usuario. Durante estos últimos años ha surgido un nuevo Hablamos de canales limitados a una dirección y una simple
concepto denominado DUIs (Distributed User Interfaces o modalidad. Dispositivos de interacción (DIs) son sistemas que
Interfaces de Usuario Distribuidas) su objetivo principal es integran computación y son capaces de controlar la entrada o
distribuir las interfaces en distintos dispositivos. Pero todavía envían la salida a los RIs individuales a los que se encuentran
existen aspectos a abordar, ¿Cómo interactuamos con las conectados. Un DI es una colección de RIs junto con la unidad de
interfaces de usuario?, los estudios en este campo han computo. Los DIs en este escenario serían ordenadores,
evolucionado desde la manipulación directa que proponía dispositivos móviles, etc. Sin embargo, la distribución de las

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 86 |
 Interfaces de Usuario Tangibles como Mecanismo de Interacción en Entornos Multi-dispositivos
E. de la Guía | M. D. Lozano | V. Penichet | L. Orozco | V. López
interfaces en el espacio es complejo, el desafío principal es:
¿Cómo configurar y distribuir las Interfaces de Usuario entre IR,
ID para conseguir un sistema usable y atractivo al usuario? Hay
estudios al respecto [5], sin embargo, no hacen hincapié sobre las
técnicas de interacción que sería más apropiada dependiendo del
sistema. Tampoco tienen en cuenta la psicología cognitiva humana
innata en los usuarios.
Con el fin de proporcionar diferentes técnicas de interacción
tangible nos hemos centrado en el gesto de acercar y alejar
también llamado “Approche&Remove” [7][8][9] en este caso
distinguimos del recurso interactivo que utilicemos como entrada,
este puede ser el dispositivo móvil, un objeto o los dedos. Estos
tipos de interacción permiten al usuario interactuar con interfaces
de usuario distribuidas fácilmente. De esta forma se ofrece un
estilo natural y fácil de interacción que resulta intuitiva y Figura 1. Distribución de una interfaz de usuario común en
motivadora para las personas que no tienen experiencia con la interfaces distribuidas y tangibles
tecnología. Han sido desarrollados con la tecnología NFC (Near
Field Communication).
3.2 Tipos de Interacción Tangible
Con el fin de interactuar con entornos multi-dispositivo donde la
3. CARACTERÍSTICAS DE LOS MDE interfaz de usuario principal está distribuida en un computador,
DESARROLLADOS tablet, smartphone, o proyección en la pared. Se han digitalizado
Se han desarrollado cuatro prototipos funcionales basados en objetos comunes como pueden monedas, tarjetas, juguetes,
juegos de aprendizaje y estimulación cognitiva. Teniendo en paneles, etc. ahora llamadas interfaces de usuario tangibles.
cuenta la diversidad de los usuarios, se han enfocado en usuarios Se describen tres tipos diferentes de técnicas de interacción, donde
con necesidades especiales. Los cuatro son colaborativos, se utilizan diferentes recursos de interacción (RI). La operación
interactivos y distribuyen interfaces de usuario en el entorno. Cada interna del sistema es la misma.
uno de ellos implementa una técnica de interacción tangible Input: Objeto inteligente
diferente. Esta técnica es llamada “Approach&Remove Object”, consiste en
interactuar con el sistema acercando y alejando la interfaz de
3.1 Distribución de interfaces de Usuario usuario tangible (objeto digitalizado) a un dispositivo de
Para diseñar la distribución de las interfaces de usuario se ha interacción (DI) en este caso cualquier dispositivo móvil
tenido en cuenta la cognición distribuida innata en los usuarios. (smartphone o tablet) que incorpore el lector NFC en su interior.
Como dice Hutchins [10] la cognición distribuida propone que la El dispositivo móvil (DI) es el encargado de conectarse con el
cognición y el conocimiento no se limitan a un individuo, sino que sistema y enviarle la información requerida, posteriormente el
se distribuye a través de objetos, personas y herramientas en el sistema mostrará la información de salida solicitad (Ver figura
entorno. Se ha hecho uso del modelo presentado en [15] donde se 2.a).
distribuye el contenido del juego teniendo en cuenta los modelos Input: Dispositivo móvil
mentales. Para el diseño de entornos MDE, hemos seguido los
Esta técnica de interacción también se denomina
siguientes criterios. La memoria a largo plazo se asocia con las
“Approach&Remove Device” debido a que el gesto que realiza el
interfaces de usuario tangibles, es decir, los objetos digitalizados
usuario para interactuar con el Sistema es el mismo, acercar y
que permiten al usuario interactuar con el sistema. De esta manera,
alejar el dispositivo móvil al panel digitalizado (Interfaz de
el proceso que seguiría la mente de los usuarios al utilizar el
usuario tangible) (Ver figura 2.b).
sistema es el siguiente: en primer lugar, la interfaz del juego es
Input: Dispositivo móvil & Objeto digitalizado
visualizada utilizando la memoria sensorial, a continuación,
mediante la memoria a corto plazo el usuario interioriza las Esta técnica de interacción es tangible pero combina dos tipos de
interfaces tangibles, finalmente, cuando se entiende la imagen interacción; tangible, es decir, el usuario interactúa con objetos
almacenada, se guarda en la memoria a largo plazo, resultando comunes y otra técnica basada en tocar una superficie digital con
intuitivo el uso del sistema. (Ver Figura 1). los dedos (touch). El usuario interactúa con la Tablet y puede
utilizar tantos los dedos como los objetos (Ver figura 2.c).
Output: Dispositivo móvil y Pantalla
Para mostrar los resultados, tenemos multi-dispositivos que nos
muestran mensajes e información multi-modal (audio, visual,
textual y gráfica).

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 87 |
 Interfaces de Usuario Tangibles como Mecanismo de Interacción en Entornos Multi-dispositivos
E. de la Guía | M. D. Lozano | V. Penichet | L. Orozco | V. López
usuarios. En los prototipos se incluyen tareas para que exploren el
entorno, recompensas, puntos, niveles para que mejoren las
capacidades cognitivas y sientan un desafío mientras utilizan el
sistema.
Control: Lepper y Malone [16] aconsejan que los juegos ofrezcan
A el control al usuario. En este caso los usuarios tienen sus propias
interfaces de usuarios tangibles y dispositivos móviles privados
con el fin de ofrecerles la confianza de que utilizan sus propios
recursos.
Feedback: La aplicación a través de audio, imágenes y texto
ofrece información relevante para guiar en todo momento al
B usuario.
Inmersión: La inmersión es la característica del juego que
provoca al jugador para que se vea envuelto en el mundo virtual,
volviéndose parte de éste e interactuando con él. En este caso el
mundo es real y combinamos el factor real (interacción y
búsqueda de objetos) con el factor virtual (tablet, Smartphone,
pantallas, etc. que nos ofrecen los juegos).
Interacción social: El factor real nos ofrece un espacio físico que
al mismo tiempo permite a los usuarios interactuar entre ellos con
para conseguir un objetivo común.

4. DESCRIPCIÓN DE LOS PROTOTIPOS

C A continuación se describen los entornos que soportan múltiples
Figura 2. Inputs en entornos MDE a) (IR) es el objeto dispositivos, interfaces de usuario distribuidas e interacción
inteligente; b) (IR) es el dispositivo móvil; c) (IR) puede ser un basadas en interfaces de usuario tangibles. Se han enfocados para
objeto inteligente o los dedos (touch) tareas/juegos colaborativos que soportan múltiples usuarios
interactuando simultáneamente.

3.3 Diseño del contenido

Para diseñar el contenido de los prototipos se ha buscado activar la
motivación por parte del usuario. La motivación es un factor
interno que incita a una acción, activa, dirige y mantiene la
conducta. Es decir, es el factor que va animar al usuario a utilizar e
interactuar con los sistemas propuestos. Existen dos tipos de
motivación: Intrínseca y Extrínseca: Las personas que 1.A 1.B
tienen motivación intrínseca realizan acciones porque encuentran
satisfacción en el mismo hecho de hacerlas, así como la auto
superación, la sensación de éxito, etc. Por otra parte, la motivación
extrínseca aparece cuando lo que atrae al individuo no es la acción
que se realiza en sí, sino lo que se recibe a cambio de la actividad
realizada (por ejemplo, premios, recompensa, puntos, etc.
Teniendo en cuenta que los prototipos son juegos se ha seguido
una serie de criterios basados en la teoría de flujo propuesta por 2.A 2.B
Sweetser y Wyeth [19].
Los puntos más importantes que se han tenido en cuenta para Figura 3.-Sistemas a-) Co-Brain Training , b) TraInAb
diseñar e implementar el contenido y distribución de las interfaces
de usuario han sido los siguientes.
Co-Brain Training (Collaborative Brain Training) [13] (Ver
Concentración: El objetivo de la aplicación es mantener la Figura 3 1.a) es un juego interactivo y colaborativo diseñado para
atención y concentración del usuario enfocada en la aplicación. estimular habilidades cognitivas como son la memoria, atención y
Para conseguir que la interfaz sea limpia y el usuario no se lenguaje en personas que tienen Alzheimer. Su funcionamiento es
distraiga con otro tipo de información se distribuye la interfaz de el siguiente: la interfaz principal del juego se proyecta en una
la aplicación siguiendo la cognición distribuida descrita pared permitiendo que puedan jugar múltiples usuarios. Los
anteriormente. Por un lado el espacio de trabajo principal y por usuarios interactúan con el sistema acercando el dispositivo móvil
otro lado los controles y dispositivos de navegación. que incorpora el lector NFC(como un apuntador) a un panel
Desafío o Reto: Según Malone [17] cualquier juego tiene que interactivo (Ver Figura 3 1.b) donde se muestran las opciones
proporcionar un reto, es decir, debe despertar la curiosidad de los posibles del juego.

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 88 |
 Interfaces de Usuario Tangibles como Mecanismo de Interacción en Entornos Multi-dispositivos
E. de la Guía | M. D. Lozano | V. Penichet | L. Orozco | V. López
TraInAb (Training Intellectual Abilities)[12] (Ver Figura 3 ser interfaces tangibles a través de las que se interactuará con el
2.a).es un juego interactivo y colaborativo diseñado para terapias juego.
de estimulación cognitiva en usuarios que tienen discapacidad Real Game es un juego ejecutado en el entorno real donde los
intelectual. El sistema integra una nueva forma de interacción a usuarios interactúan con el entorno y entre ellos a través de
través de objetos comunes de la vida cotidiana como pueden ser dispositivos móviles/objetos. Los usuarios exploran el entorno, el
cartas, juguetes, monedas. Su funcionamiento es el siguiente: la sistema detecta donde se encuentran y en función de esto les
interfaz principal del juego se proyecta en una pared con el fin de plantea retos a través de los dispositivos móviles que portan. Los
aumentar su visibilidad y permitir que puedan jugar múltiples retos se resuelven colaborativamente interactuando con los objetos
usuarios. Los usuarios interactúan con el sistema acercando los que se encuentran en una determinada zona del entorno, acercan el
objetos (interfaces tangibles), identificados mediante tecnología objeto a otro objeto/dispositivo móvil. Una pantalla principal se
NFC (Near Field Communication), al dispositivo móvil que muestra las puntaciones, recompensas, retos logrados, etc. (Ver
incorpora el lector NFC (Ver Figura 3 2.b). Figura 4.b)

5. LECCIONES APRENDIDAS
La obtención de datos nos permite definir directrices que ayuden a
futuros diseñadores e investigadores a tomar decisiones a la hora
de desarrollar entornos MDE que soporten interfaces de usuario
distribuidas y tangibles. Después de realizar iterativas
evaluaciones de los prototipos se han recogido los tiempos medios
dependiendo de la interacción, el número medio de errores, la
tangibilidad, affordance, número de interacciones verbales entre
los usuarios, el coste y la infraestructura de los sistemas. A
continuación se describen brevemente algunos de los resultados
obtenidos.
Teniendo en cuenta que los sistemas se diseñan para un un espacio
físico real, es necesario que sea usable, es decir, la interacción
debería ser fácil de aprender, sencilla, atractiva al usuario y
familiar, con el fin de romper la brecha digital que existen entre
varios grupos de usuarios. Por ese motivo, se ha decidido diseñar
entornos que simulen el modo real de trabajar que tienen los
usuarios en su vida cotidiana. La distribución de interfaces de
usuario se realiza siguiendo la psicología cognitiva distribuida, es
Figure 4. a) Sistema NFCBook b) Sistema Real Game decir, teniendo en cuenta los procesos mentales del usuario.
Usabilidad y fácil aprendizaje.
La tangibilidad es el atributo que define a los dispositivos/objetos
NFCBook [14] es un libro-juego, es decir, una obra literaria que son fácilmente detectables con los sentidos. En este caso los
donde se involucra al lector en el seno de la historia permitiéndole objetos son más comunes y familiares teniendo un grado de
decidir sobre el curso de la misma. Su contenido se ha basado en tangibillidad más alto. La técnica de interacción tangible basada
la colección de libros-juegos denominada "Elige tu propia en acercar los objetos a otros objetos o dispositivos comunes
aventura". El objetivo principal de la aplicación es motivar a los camuflados en objetos ofrecen una alta tangibilidad
usuarios para que adquieran el hábito de lectura ayudándoles de proporcionando entornos más familiares y fáciles de utilizar para
esta forma a desarrollar habilidades cognitivas tales como agilidad usuarios que no tengan experiencia previa utilizando las nuevas
mental, concentración, activación del sistema visual, etc. Para tecnologías Affordance es la cualidad de un objeto o ambiente
incentivar al usuario se ha tenido en cuenta factores lúdicos que que permite a un individuo realizar una acción, en este caso, los
pretenden convertir una actividad a priori aburrida en una objetos tienen un grado de affordance más alto. Se asemejan a los
actividad que motive al usuario a participar en ella (Figura 3.a). entornos reales.
El libro que antes era físico se ha convertido en formato digital Distribución de Interfaces de Usuario en RI/DI
para ser leído y consultado desde un dispositivo móvil (tablet, El tiempo medio que tardaban los usuarios en buscar la solución
smartphone, etc.). Para interactuar con el libro-juego se utilizan correcta en el juego dependía del orden en el que se habían
objetos físicos que integran NFC en su interior. De esta forma distribuido las interfaces de usuario. Un ejemplo se dio en el juego
antes de comenzar a jugar es necesaria una fase de búsqueda y de deletrear, teníamos 27 opciones correspondientes al abecedario,
selección de tarjetas comunes que integren NFC, después el buscar la letra correcta entre 27 objetos era muy tedioso y los
usuario tendrá el control para configurarlas y posteriormente usuarios tardaban más tiempo, sin embargo cuando las letras
disfrutar del libro-juego a través del dispositivo móvil y las estaban organizadas en el panel interactivo los usuarios accedían
interfaces tangibles o tarjetas. automáticamente a la opción correcta en cada momento,
La aplicación interactiva distribuye interfaces de usuario en necesitaban menos tiempo para buscar la opción, en tareas donde
dispositivos móviles y objetos comunes. El funcionamiento es el las opciones siguen un orden como son los números, letras, etc.. es
siguiente, el usuario a través de tarjetas físicas casuales que puede más intuitivo utilizar paneles interactivos con la técnica de
encontrar en cualquier lugar, como son tickets de transportes interacción consistente en acercar y alejar el dispositivo móvil a la
públicos, entradas a espacios públicos, museos, cines, etc. interfaz de usuario tangible. Es decir, los paneles interactivos son
personaliza sus objetos y su propio juego. La aplicación ha sido más aconsejables cuando podemos agrupar opciones. En el caso
implementada en Android e internamente utiliza tecnología NFC en el que tengamos que distribuir la información o necesitas
para dar un valor a las tarjetas físicas que posteriormente pasaran a tenerlo más organizado, el panel es preferible como interfaz de
XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)
| 89 |
 Interfaces de Usuario Tangibles como Mecanismo de Interacción en Entornos Multi-dispositivos
E. de la Guía | M. D. Lozano | V. Penichet | L. Orozco | V. López
usuario tangible. De esta manera, los paneles pueden distribuir Tabla 1. Comparación de las propiedades de las técnicas de
menús y atajos para utilizar el sistema. interacción tangible
El número de errores debido a la técnica de interacción se Alto, medio y Approach Approach& Approach&
ocasionaban por los siguientes motivos: bajo &Remove Remove el Remove el
el objeto objeto
1º. El usuario acercaba el dispositivo móvil al panel pero no dispositivo combinado
apuntaba de forma precisa la opción que deseaba seleccionar y el móvil con ‘touch’
lector recogía el valor de la opción más cercana.
% Errores Medio Bajo Medio
2º. El usuario acercaba la interfaz de usuario tangible al
dispositivo móvil, pero no en el área donde se encuentra el lector Coste de la Medio(un Bajo Bajo
NFC, por lo que tenían que buscar el área primero y acercar la Infraestructura dispositivo
tarjeta. Este error se dio cuando comenzaron el juego y más en el móvil cada
caso de la Tablet que el móvil, debido al tamaño del dispositivo. usuario)
Cuando el usuario acercaba la interfaz tangible al dispositivo Tangibilidad Bajo Alto Alto
móvil había menos errores, el dispositivo era más pequeño y la
Affordance Medio Alto Alto
lectura de la tarjeta NFC (que se encontraba en el interior del
objeto) era más precisa. Agrupamiento Alto Medio Medio
El número de errores fue mayor al acercar el dispositivo móvil a
la interfaz de usuario tangible. En cambio la técnica de interacción Interacción Medio Alto Medio
consistente en acercar la interfaz de usuario tangible al dispositivo Social
móvil eran más precisas por lo que el número de errores fue muy
reducido.
El coste es mayor cuando se utiliza la técnica de interacción 6. RECOMENDACIONES DE DISEÑO
basada en acercar el dispositivo móvil al panel interactivo, todos Después de analizar los datos se han definido ocho directrices
los usuarios deben tener su propio móvil. Sin embargo, acercar los iniciales para el diseño de sistemas MDE que soportan interfaces
objetos a un único dispositivo conlleva un coste muy bajo. de usuario distribuidas y tangibles.
Motivación- Múltiples Usuarios Directriz 1. Se debe organizar los objetos de una forma ordenada
La interacción social se facilitaba con las técnicas de interacción y consistente. Esta es una de las reglas más comunes que se siguen
basada en acercar los objetos a un dispositivo móvil. Se mejoraba diseñando GUI (Interfaces de usuario gráficas) y siguiendo las
la comunicación, comentaban entre ellos las posibles soluciones, y teorías de la escuela psicológica de la Gestalt [11], donde
se obtuvieron más interacciones verbales entre los usuarios. Les describen que las personas percibimos los objetos como patrones
resultaba más atractivo y novedoso. Cuando interactuaban bien organizados más que como partes discretas. La mente
acercando el dispositivo móvil al panel que compartían entre configura, a través de ciertas leyes, los elementos que llegan a ella
todos, cobraba importancia los turnos para que puedan interactuar a través de los canales sensoriales (percepción) o de la memoria
todos, el número de interacciones verbales entre ellos fue menor. (pensamiento, inteligencia y resolución de problemas).
Teniendo en cuenta la percepción visual del usuario y las
lecciones aprendidas. Se deben agrupar objetos de la misma
naturaleza en un panel, tabla, surface, etc. como por ejemplo
(Abecedario, números, opciones gestionar ficheros: abrir, guardar,
etc.). Los paneles se van a utilizar como atajos para realizar
acciones en el sistema que se llevan a cabo repetidas veces.
Directriz 2. Se deben dividir los espacios de trabajo en dos tipos
diferente. Por un lado el privado donde el usuario tiene el control
de su información, dispositivos, objetos, etc. Por otro lado el
espacio de trabajo colaborativo donde el usuario puede ver los
datos de los demás usuarios, enviar información a todos los demás
componentes del grupo, etc. De esta forma conseguimos que el
usuario tenga el control de su información dándole confianza [18].
Directriz 3. Añadir el componente “Espacio físico” donde el
entorno ofrece información al usuario permitiéndole explorar e
interactuar, reduce la carga cognitiva. En este caso, las interfaces
de usuario visuales deben ser sencillas e intuitivas con el fin de no
despistar al usuario. Además los mensajes auditivos, sensoriales,
etc. Son preferibles para no saturar al usuario con la información
visual.
Directriz 4. Añadir señales, pistas, etc. Cuando el usuario necesite
explorar a través del entorno. De esta forma se consigue que el

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 90 |
 Interfaces de Usuario Tangibles como Mecanismo de Interacción en Entornos Multi-dispositivos
E. de la Guía | M. D. Lozano | V. Penichet | L. Orozco | V. López
usuario no se sienta perdido y sepa en cada momento donde se in computing systems, pages 234–241, New York, NY, USA.
encuentra. ACM Press. 19, 20, 27, 36, 73, 96, 187
Directriz 5. Se debe reducir la distancia de los usuarios respecto a [3] Elmqvist, N. Distributed User Interfaces: State of the Art
los objetos que se utilizan repetidas veces mientras se usa el .Workshop on Distributed User Interfaces2011 (DUI) at the
sistema. Estos objetos deben estar cerca. 29th ACM CHI Conference on Human Factors in
Directriz 6. El Sistema debe avisar al usuario a través de Computing Systems 2011, ISBN: 978-84-693-9829-6,
alertas/notificaciones/pistas donde se está mostrando la Vancouver, Canadá, May 7-12, 2011.
información importante en cada momento de forma que el usuario [4] González,P.,Gallud,J.A.,Tesoriero,R. WallShare: A
preste atención a la información relevante. Collaborative Multi-pointer System for Portable Devices.
Directriz 7. Añadir componentes motivacionales es recomendable PPD10: Workshop on coupled display visual interfaces.
con el objetivo de que el usuario disfrute y se sienta inmerso en la May 25, 2010: Rome, Italy.
tarea. Se pueden añadir de dos tipos diferentes: Motivación [5] Vanderdonckt, J. Distributed User Interfaces: How to
intrínseca: El usuario debe sentir que tienen el control del juego, Distribute User Interface Elements across Users,
añadirle la posibilidad de comunicación con otros usuarios, la Platforms, and Environments. Proc. of XI Interacción, 20-
facilidad de explorar en el entorno y crearle desafíos. Extrínseca. 32, 2010.
Añadir mecanismos lúdicos, es decir puntos, ranking, premios,
refuerzo positivo, etc. [6] Vandervelpen,C. , Coninx, K. Towards model based design
support for distributed user interfaces, Proceedings of the
Directriz 8. Se debe tener en cuenta el awareness, es decir, el third Nordic conference on Human-computer interaction,
usuario debe ser capaz de percibir, sentir, o ser consciente de los p.61-70, October 23-27, 2004, Tampere,
acontecimientos, objetos, acciones, etc tanto del entorno como de Finland [doi>10.1145/1028014.1028023]
los demás usuarios. Es decir, cuando el usuario trabaje con grupos
de usuarios debe ser consciente de los demás usuarios y a la vez [7] Want, R., Fishkin, K.P., Gujar, A., Harrison, B.L.: Bridging
ser capaz de comunicarse con ellos y compartir información physical and virtual worlds with electronic tags. In:
fácilmente. La comunicación entre los usuarios debe evitar que sea Proceedings of the SIGCHI conference on Human factors
a través de mensajes escritos en dispositivos con reducida in computing systems: the CHI is the limit, ACM Press,
pantallas, es preferible que sea a través de botones, gestos, etc. Pittsburgh, Pennsylvania, United States, 1999.
Estas deberían ser mostradas a los demás a través de [8] Romero, S., Tesoriero, R., González, P. , Gallud, J. A.,
notificaciones, sonidos, mensajes sensoriales, imágenes, etc. Penichet, V. M. R.: Interactive System based on NFC to
manage Georeferenced Documents. Interacción 2009, X
7. CONCLUSIONES Congreso Internacional de Interacción Persona-Ordenador.
Actualmente nos encontramos con escenarios que distribuyen Barcelona. Septiembre 2009. ISBN-13:978-84-692-5005-1
interfaces en entornos con múltiples dispositivos. Interactuar con
diferentes interfaces no es una tarea trivial, por ese motivo hemos [9] Tesoriero, R., Tébar, R., Gallud, J. A., Penichet, V. M. R.,
propuestos la interacción tangible como mecanismo de Lozano, M.: Interactive EcoPanels: Ecological Interactive
interacción, esta resulta intuitiva y permite que la mayoría de los Panels based on NFC . Proceedings of the IX Congreso
usuarios se sientan familiarizados con objetos comunes, resultando Internacional de Interacción Persona-Ordenador Interacción
más agradable su uso. Las interfaces de usuario se han distribuido 2008. ISBN: 978-84-691-3871-7; pp 155-165.
siguiendo la psicología distribuida y se han añadido componentes [10] Hutchins, Edwin (1995). Cognition in the Wild.
para motivar a los usuarios a usar los sistemas. Después de Cambridge, MA: MIT Press.
analizar las evaluaciones realizadas de cuatro prototipos se han [11] Arnheim, R. (1985). Arte y Percepción Visual. Alianza
obtenido ocho directrices de diseño iniciales. Estos resultados son Editorial. Madrid
preliminares, como trabajo futuro se pretende realizar una
evaluación más exhaustiva con el fin de obtener criterios y [12] Elena de la Guía, María Lozano, and Victor R. Penichet.
patrones que ayuden a futuros diseñadores e investigadores a 2013. TrainAb: a solution based on tangible and distributed
construir sistemas de esta naturaleza. user interfaces to improve cognitive disabilities (CHI EA
'13). ACM, New York, NY, USA, 3039-3042.
8. AGRADECIMIENTOS DOI=10.1145/2468356.2479605
Esta investigación ha sido parcialmente subvencionada por el http://doi.acm.org/10.1145/2468356.2479605
proyecto regional con referencia POII-2014-008-A. Agradecemos [13] Elena de la Guía, María Dolores Lozano, Victor M. Ruiz
especialmente a Érica González, Yolanda Cotillas, Francisco Penichet: Tangible and Distributed User Interfaces to
Vizcaino, Ruben Nieto, Vicente López y Mª Dolores Monedero su Improve Cognitive Abilities within People Affected by
colaboración en este trabajo de investigación, igualmente Alzheimer's Disease. DUI@EICS 2013: 10-13
queremos agradecer a las asociaciones AFA, ACMIL y
ASPRONA de Albacete por su participación en las evaluaciones. [14] Elena de la Guía, María D. Lozano, Víctor M. R. Penichet,
and Rubén Nieto. 2014. NFCBOOK: GameBook Digital
9. REFERENCIAS Based on Tangible User Interfaces (Interacción '14). ACM,
[1] Schneiderman, B. Direct manipulation: a step beyond New York, NY, USA, Article 32 , 3 pages.
programming languages. IEEE Cornput. (Aug. 1983), 57-69. DOI=10.1145/2662253.2662285
http://doi.acm.org/10.1145/2662253.2662285
[2] Ishii, H. and Ullmer, B. (1997). Tangible bits: towards
seamless interfaces between people, bits and atoms. In CHI [15] Elena de la Guía, María D. Lozano, and Víctor M. R.
’97: Proceedings of the SIGCHI conference on Human factors Penichet. 2012. Interaction and collaboration supported by

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 91 |
 Interfaces de Usuario Tangibles como Mecanismo de Interacción en Entornos Multi-dispositivos
E. de la Guía | M. D. Lozano | V. Penichet | L. Orozco | V. López
distributed user interfaces: from GUIs to September 18-19, 1980, Palo Alto, California, USA
DUIs. (INTERACCION '12). ACM, New York, NY, USA,, [doi>10.1145/800088.802839]
Article 53 , 2 pages. DOI=10.1145/2379636.2379688 [18] Sara Streng , Karsten Stegmann , Sebastian Boring , Sonja
http://doi.acm.org/10.1145/2379636.2379688 Böhm , Frank Fischer , Heinrich Hussmann, Measuring
[16] Lepper, M. R., & Malone, T. W. (1987). Intrinsic effects of private and shared displays in small-group
motivation and instructional effectiveness in computer- knowledge sharing processes, Proceedings of the 6th
based education. In R. E. Snow, & M. J. Farr (Eds.), Nordic Conference on Human-Computer Interaction:
Conative and affective process analyses: Vol. 3. Aptitude, Extending Boundaries, October 16-20, 2010, Reykjavik,
learning and instruction (pp. 255--286). Hillsdale, NJ: Iceland [doi>10.1145/1868914.1869030]
Lawrence Erblaum [19] Penelope Sweetser , Peta Wyeth, GameFlow: a model for
[17] Thomas W. Malone, What makes things fun to learn? evaluating player enjoyment in games, Computers in
heuristics for designing instructional computer games, Entertainment (CIE), v.3 n.3, July 2005
Proceedings of the 3rd ACM SIGSMALL symposium and [doi>10.1145/1077246.1077253]
the first SIGPC symposium on Small systems, p.162-169,

XVI Congreso Internacional de Interacción Persona-Ordenador | Natural User Interfaces (II)

| 92 |
METHODOLOGIES AND
MODELS
 Limitaciones del Modelo de Tareas del W3C para aplica-ciones Post-WIMP
M. A. Teruel | A. C. Rodríguez | F. Montero | E. Navarro | V. López-Jaquero | P. González

Limitaciones del Modelo de Tareas del W3C para

aplicaciones Post-WIMP
Miguel A. Teruel, Arturo C. Rodríguez, Francisco Montero,
Elena Navarro, Víctor López-Jaquero and Pascual González
Grupo de Investigación LoUISE
Instituto de Investigación en Informática de Albacete
02071 – Albacete (España)
[email protected], [email protected], [email protected],
[email protected], [email protected], [email protected]

RESUMEN gran cantidad de datos o los juegos altamente interactivos.

La evolución tecnológica y la aparición de nuevos dispositivos También tienen deficiencias en aplicaciones que requieren una
que ésta conlleva está provocando grandes cambios en el modo en monitorización continua de señales de entrada, representación de
que las personas interaccionan con los sistemas software. Cada modelos 3D o simplemente en interacciones para las que no hay
vez son más los sistemas cuyas técnicas de interacción están widgets estándar. Por esta razón, las interfaces de usuario
basados en realidad virtual, reconocimientos de gestos, personalizadas se han convertido en la mejor alternativa a la hora
dispositivos corporales y muchos otros, que se alejan de realizar tareas complejas.
notablemente de la interacción clásica de los sistemas Post-WIMP es un término usado para describir interfaces que van
denominados WIMP (Windows, Icons, Menus, Pointers). Estos más allá de los paradigmas habituales (ventanas, iconos, menús y
nuevos sistemas, a los que se denomina sistemas Post-WIMP, punteros) [1]. Esta nueva generación de interfaces ha sido
necesitan, por lo tanto, que los lenguajes para la especificación de impulsada tanto por los avances en tecnología como por un mejor
sistemas evolucionen de igual forma para poder hacer frente al entendimiento de la psicología humana. Definidas por van Dam
incremento de la complejidad y a los nuevos conceptos inherentes [1] como interfaces “que contienen al menos una técnica de
a sus formas de interacción. En este artículo se presenta un interacción no dependiente de los clásicos widgets 2D, como
lenguaje de modelado de tareas de sistemas Post-WIMP, basado a menús o iconos”, algunos ejemplos de interacciones Post-WIMP
su vez en CSRML, como alternativa al actual meta-modelo de son: (i) realidad virtual, aumentada o mixta, (ii) interacción
tareas propuesto por el W3C. Este nuevo lenguaje tiene como tangible, (iii) computación ubicua, (iv) computación sensible al
finalidad especificar las tareas que serán realizadas por el usuario contexto, (v) interacción con dispositivos móviles, (vi)
a través de la interfaz de sistemas Post-WIMP con la expresividad computación perceptual, e (vii) interacción ligera, tácita o pasiva.
de la que carece el actual meta-modelo de tareas del W3C. Cabe Aunque algunas personas podrían clasificar estos estilos de
destacar principalmente dos ventajas de esta nueva propuesta interacción como innovaciones dispares, futuristas y heterogéneas,
respecto a la del W3C. Por un lado, se han incluido varios comparten importantes características como una considerable
elementos y relaciones para enriquecer el nuevo meta-modelo con demanda de información de contexto que ayude a entender,
la expresividad necesaria para llevar a cabo la especificación de la conectar y analizar el trabajo que se lleva a cabo.
colaboración de forma detallada. Por otro lado, nuestra propuesta
permite representar la información de awareness que los usuarios Esta evolución de las interfaces requiere que los lenguajes para el
Post-WIMP deberían tener durante la interacción con el sistema. análisis de requisitos y la especificación de sistemas sean
Para mostrar la utilización de este nuevo meta-modelo, adaptados de forma adecuada, pudiendo así recoger las
presentamos un ejemplo de modelado basado en un juego “first necesidades de información de estos nuevos paradigmas de
person shooter” Post-WIMP con una interfaz de realidad virtual. interacción. El análisis de tareas no es una excepción y, a pesar de
que los meta-modelos actuales han supuesto una herramienta
Categorías y descriptores de tema eficaz en el desarrollo de sistemas WIMP, las nuevas interfaces
H.5.2 [User Interfaces]: Interaction styles., I.3.6. [Methodology Post-WIMP introducen la necesidad de revisarlos. Además, el
and Techniques]. Interaction techniques. análisis de tareas está estrechamente relacionado con la
interacción, puesto que, como afirman Gea y Gutiérrez [2], cubre
Términos Generales diferentes técnicas orientadas a describir las interacciones entre
Modelado, Interacción, Experimentación, Lenguajes. las personas y los entornos de una manera sistemática y tiene
como objetivo obtener una descripción formal del conjunto de
Palabras clave acciones que debe realizar el usuario para lograr un fin; acciones
Post-WIMP; CSRML; W3C task model; collaboration; awareness. que pueden ser tanto cognitivas como interacciones con la
interfaz.
1. INTRODUCCIÓN
Durante las últimas dos décadas, investigadores en Interacción En este artículo, proponemos una alternativa al actual meta-
Persona-Ordenador (HCI) han desarrollado una amplia gama de modelo de tareas del W3C [3] considerando dos de los principales
nuevas interfaces que se alejan de las tradicionales WIMP aspectos clave de los sistemas Post-WIMP: awareness (habilidad
(Window, Icon, Menu, Pointing). Estas interfaces WIMP del usuario de ser consciente de lo que le rodea) y colaboración
presentan carencias para llevar a cabo tareas complejas como el entre usuarios. La necesidad de incorporar dichos conceptos
diseño asistido por ordenador, la manipulación simultanea de una surge, no sólo del estudio de la literatura relacionada, sino

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 94 |
 Limitaciones del Modelo de Tareas del W3C para aplica-ciones Post-WIMP
M. A. Teruel | A. C. Rodríguez | F. Montero | E. Navarro | V. López-Jaquero | P. González
también de la amplia experiencia acumulada en nuestro grupo de pueden definirse cuatro tipo de tareas: (i) User, o tareas
investigación a lo largo de los últimos quince años en el desarrollo de usuario: actividad cognitiva que no se realiza
de interfaces post-WIMP, que se inició con sistemas de directamente a través de la interfaz. (ii) System, o tarea
interacción 3D [4]. de sistema: acción realizada por la aplicación. (iii)
Interaction: acciones realizadas por los usuarios que dan
Como base para esta propuesta, se ha elegido el lenguaje CSRML lugar a respuestas instantáneas del sistema y (iv)
(Collaborative Systems Requirements Modelling Language, Abstract: tareas que se descomponen en subtareas de
[5][6]) ya que dispone de capacidad expresiva para especificar las distintas categorías.
necesidades de colaboración y awareness. Además, otra razón
para escoger este lenguaje es que ha sido evaluado para tratar con • Domain Object: un objeto del modelo de dominio
diferentes dominios, como la edición colaborativa de documentos relacionado con una tarea.
[5] o la inteligencia de negocio [7]. No obstante, el modelo de
tareas CSRML ha sido ampliado y modificado para conseguir dos • Condition Group: operación lógica que puede ser usada
objetivos. En primer lugar para hacerlo compatible con el modelo para especificar una precondición y/o una post-
W3C y en segundo lugar, para enriquecerlo con elementos de condición de una tarea.
awareness ligados a sistemas Post-WIMP, ya que CSRML estaba • Logic Operator: operador que conecta los operandos del
diseñado para atender básicamente a las necesidades de los Condition Group y especifica el tipo de operación
sistemas colaborativos . lógica.
Este artículo está organizado de la siguiente manera. La Sección 2 • Condition Literal: valor u objeto del modelo del
presenta el estado de la cuestión relacionada con el actual meta- dominio usado como operando en una operación lógica.
modelo de tareas propuesto por la W3C. En base a las deficiencias
detectadas en la propuesta de la W3C, en la Sección 3 se presenta • N-ary Operator: operador que establece una restricción
el meta-modelo de tareas para interfaces Post-WIMP. Para que temporal entre subtareas.
puedan apreciarse más claramente las diferencias de la propuesta
de la W3C y la nuestra, en la Sección 4 se presenta un ejemplo de • 1-ary Operator: operador que establece si una tarea es
modelado basado en un juego Post-WIMP que se ha modelado opcional y/o iterativa.
usando ambos meta-modelos de tareas. Finalmente, en la Sección
5 se presentarán algunas conclusiones.

2. META-MODELO DE TAREAS DEL W3C

Gea y Gutiérrez [2] definen el análisis de tareas como el estudio
de lo que un usuario tiene que realizar en términos de acciones
y/o procesos cognitivos para conseguir un objetivo. Este análisis
se traduce en modelos de tareas que tradicionalmente, han
proporcionado una descripción eficaz de sistemas interactivos
WIMP. Representan información estática de las tareas de
aplicación y de usuario a dos niveles: estructural (descomposición
de tareas) y temporal (relaciones temporales). Cada tarea
especificada en el modelo de tareas es una actividad o proceso
cuya realización es necesaria para conseguir los objetivos del
usuario. Además, los analistas pueden representar tareas en
diferentes niveles de abstracción, por lo que pueden considerar Figura 1. Meta-modelo actual del W3C [3]
una actividad como una tarea atómica o descomponerla en sub- El meta-modelo mostrado en la Fig. 1 permite al analista describir
tareas, con el objetivo de proporcionar una especificación con modelos de tareas de sistemas WIMP de forma eficaz. Este meta-
mayor detalle, y después, establecer relaciones temporales entre modelo usa las relaciones de composición para especificar
ellas. Sin embargo, esta descripción no tiene en cuenta detalles estructuras jerárquicas de tareas y asocia los operadores
sobre cómo se llevan a cabo los procesos e interacciones, ambos temporales a estas relaciones. Aunque estos operadores son
necesarios para el correcto diseño del diseño del sistema. suficientemente expresivos para describir dichas relaciones, sería
razonable incluir una nueva meta-clase específica para establecer
El modelo de tareas del W3C está basado en la notación
relaciones temporales entre tareas, asociando el operador a dicha
ConcurTaskTrees (CTT), ampliamente aceptada [8] y tiene como
clase, permitiendo así desligar las relaciones estructurales de las
objetivo definir un estándar para el modelado de tareas. El meta-
temporales y estableciendo un único operador para cada relación
modelo de tareas del W3C proporciona una estructura jerárquica
temporal.
entre tareas y ofrece varios operadores, usados también en CTT,
para definir las relaciones temporales entre tareas. Además, Las aplicaciones actuales tienden claramente a ser colaborativas
proporciona una taxonomía de tipos de tareas. Los elementos del (por ejemplo, juegos, editores de texto, redes sociales, IDEs, etc.)
meta-modelo de tareas (¡Error! No se encuentra el origen de la con un considerable número de usuarios interactuando
referencia.) son los siguientes: concurrentemente. Además, las interfaces de usuario de dichas
aplicaciones han evolucionado enormemente hacía el paradigma
• Task: acción que se realiza desde la perspectiva del Post-WIMP. Así, Jacob et al. [9] identificaron los siguientes
usuario. Desde el punto de vista estructural, hay dos cuatro aspectos del mundo real que una aplicación Post-WIMP
tipos de tareas: Tareas atómicas (acciones indivisibles) debería soportar:
y tareas compuestas (divisibles en subtareras). Además,

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 95 |
 Limitaciones del Modelo de Tareas del W3C para aplica-ciones Post-WIMP
M. A. Teruel | A. C. Rodríguez | F. Montero | E. Navarro | V. López-Jaquero | P. González
• Naïve Physics: las personas tienen conocimiento general Se ha escogido este lenguaje porque soporta la especificación de
del mundo físico. requisitos de sistemas colaborativos, uno de los elementos
principales de los sistemas Post-WIMP. CSRML es un lenguaje
• Body Awareness & Skills: las personas tienen de modelado de requisitos orientado a objetivos para la
conciencia/conocimiento de sus propios cuerpos y especificación de sistemas colaborativos respaldado por una
poseen habilidades para controlar y coordinar sus herramienta empíricamente validada [13]. Los elementos usados
cuerpos. para el modelado de tareas se describen a continuación (ilustrados
• Enviroment Awareness & Skills: las personas son en Fig. 3 y Fig. 4). Además, la representación gráfica de dichos
conscientes de lo que les rodea y poseen habilidades elementos se puede ver en la Fig. 2.
para negociar, manipular y moverse por el entorno. • Task: El concepto de tarea del W3C ha sido extendido
• Social Awareness & Skills: las personas son para incluir la importancia usando un código de colores
generalmente conscientes de otras personas en su (verde, amarillo, naranja y rojo) siendo el color verde el
entorno y tienen habilidades para interactuar con ellos. de menor importancia y el rojo el de mayor importancia.
Además, se han identificado cuatro tipos de tareas: (i)
Como se puede ver, todos estos aspectos están principalmente Abstract Task: Abstracción de un conjunto de tareas
relacionados con el concepto de awareness [10]. Por ejemplo, la concretas, (ii) User Task: Tarea realizada por un usuario
propiocepción del usuario en el entorno físico juega un papel sin necesidad de interactuar con la interfaz e (iii)
fundamental en entornos de realidad virtual en los que puede Interaction Task: Tarea que implica una interacción con
existir una interacción con cualquier parte del cuerpo. Una la interfaz o respuestas del sistema. Hay cuatro tipos de
representación virtual del cuerpo y otras informaciones, como la tareas de interacción: Individual Task es una tarea que
localización virtual pueden mejorar la experiencia de usuario al no conlleva colaboración (puede ser realizada por un
interactuar con sistemas Post-WIMP. El conocimiento de los participante). Estas tareas contemplan también las tareas
grupos o equipos a los que pertenecen los usuarios es un ejemplo de sistema, ya que éste es considerado como un actor
de Social Awareness. Otro ejemplo de awareness lo encontramos más dentro del modelo. Collaboration/
en sistemas en los que varios usuarios están editando un Communication/Coordination tasks son tareas que
documento al mismo tiempo, necesitan saber con quién están requieren al menos dos participantes para poder ser
colaborando y qué cambios están haciendo en el documento. Esta realizadas, con el objetivo de llevar a cabo algún tipo de
necesidad ha sido ampliamente analizada en Computer Supported colaboración / comunicación / coordinación entre ellos.
Collaborative Work (CSCW). Ellis et al. [11] identificaron las
fuertes relaciones entre awareness y colaboración en el modelo de • Legacy Elements: Con el fin de hacer este modelo de
colaboración 3C. Debido a que awareness y colaboración están tareas compatible con el del W3C, varios elementos han
muy extendidos en las aplicaciones contemporáneas, sido añadidos a su meta-modelo.
especialmente en entornos virtuales colaborativos (Collaborative • Role: Establece el comportamiento de un actor para un
Virtual Environments CVE) [12], un meta-modelo de tareas conjunto de tareas relacionadas. Un actor que
debería permitir especificar tales características. Por lo tanto, desempeña un rol puede participar en una tarea
puesto que la especificación actual del W3C fue originalmente individual o colectiva (mediante las relaciones de
diseñada para especificar entornos WIMP, en este artículo participación). Cabe señalar que la relación entre un
presentamos un meta-modelo de tareas extendido para entornos actor y un rol es especificada en el modelo de usuario,
Post-WIMP basado en CSRML con el fin de especificar estos no en el modelo de tareas.
innovadores sistemas.
• Goal: Contesta a la pregunta “¿por qué?”. Describe un
3. UN LENGUAJE DE MODELADO DE estado del mundo que el usuario querría alcanzar. Sin
TAREAS PARA ENTORNOS POST-WIMP embargo, un Goal (objetivo) no describe cómo debería
Una vez se han identificado los problemas y limitaciones del alcanzarse dicho estado.
meta-modelo de tareas del W3C para especificar sistemas Post-
WIMP, presentamos en esta sección el meta-modelo de tareas
CSRML [5] adaptado para la especificación de tareas Post-WIMP.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 96 |
 Limitaciones del Modelo de Tareas del W3C para aplica-ciones Post-WIMP
M. A. Teruel | A. C. Rodríguez | F. Montero | E. Navarro | V. López-Jaquero | P. González

D
Dependency

Means-end link

Task decomposition link

+/-/?
Contribution link
[Guard]
Playing link

Responsibility link
1..*
Participation link

Figura 2. Representación gráfica de los elementos del meta-modelo de tareas para sistemas Post-WIMP

Figura 3. Meta-modelo de tareas para sistemas Post-WIMP (Elementos)

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 97 |
 Limitaciones del Modelo de Tareas del W3C para aplica-ciones Post-WIMP
M. A. Teruel | A. C. Rodríguez | F. Montero | E. Navarro | V. López-Jaquero | P. González

Figura 4. Meta-modelo de tareas para sistemas Post-WIMP (Relaciones)

• Softgoal: Es una condición del mundo que el usuario What – Group goal es un ejemplo de Social Awareness.
quiere alcanzar, pero a diferencia del concepto Goal, Además, estos elementos de awareness están
esta condición no está claramente definida. Un Softgoal clasificados en cuatro categorías relacionadas con el
es normalmente un atributo cualitativo que restringe presente, el pasado, el futuro y aspectos sociales. Por
otros elementos, como un objetivo, una tarea o un último, su importancia puede ser establecida como nice
recurso. to have (N), desirable (D), highly desirable (HD) o
• Resource: Es una entidad que el usuario necesita para mandatory (M).
alcanzar un objetivo o realizar una tarea. El principal • Participation Link: denota quién está involucrado en una
interés de un recurso es saber si está disponible y de tarea. Esta relación tiene un atributo para especificar su
quién. cardinalidad, es decir, el número de usuarios que pueden
• Awareness Resource: Este elemento representa involucrarse en la tarea. Opcionalmente puede tener un
información de awareness que un usuario necesita para Awareness Resource ligado que representa que el Role
completar una tarea. Este elemento es representado tiene una necesidad especial de percepción para
como un conjunto de elementos de Awareness participar en la tarea. Sin esta percepción, la realización
(Awareness Elements) ligados a una relación de de la tarea se vería afectada negativamente o incuso el
participación entre un rol y una tarea. Role podría ver impedida su participación en la tarea.
• Awareness Element: Un Awareness Resource muestra • Task Decomposition Link: Describe elementos
todos los Awareness Elements disponibles (si son esenciales de una tarea. Un Task Decomposition Link
necesarios) e indica su importancia en función de la relaciona una tarea con subtareas a realizar, subgoals a
contribución a la realización de la tarea. Estos conseguir, Resources necesarios y softgoals que
elementos de awareness son una extensión de los normalmente definen la calidad de los objetivos para la
identificados por Gutwin y Greenberg [14] para tarea.
entornos Post-WIMP con el propósito de ser • Means-end Link: documenta qué softgoal, task o
compatibles con los diferentes elementos de awareness resource contribuyen a alcanzar un objetivo.
identificados por Jacob et al. [9]. Por ejemplo, el • Dependency: Es una relación ente un depender y un
elemento Presence – Where – Reach está relacionado dependee para un dependum. El depender y el dependee
con el elemento de Jacob Body awareness; el elemento son actores y el dependum puede ser un goal, una tarea
Future – Where – Next Position está relacionado con el (task), un resource o un softgoal. El depender depende
conocimiento del entorno; y Social & group dynamics – del dependee para alcanzar un objetivo, realizar una

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 98 |
 Limitaciones del Modelo de Tareas del W3C para aplica-ciones Post-WIMP
M. A. Teruel | A. C. Rodríguez | F. Montero | E. Navarro | V. López-Jaquero | P. González
tarea o usar un recurso. Si el dependee no puede equipo), Shoot, Aim, Pull The Trigger, Get Power-ups, Deploy
proporcionar un dependum al depender la realización de Power-up (tarea de sistema), Show End (tarea de sistema).
la tarea, alcanzar el objetivo o usar el recurso se vuelve Además, puesto que el juego a implementar es tanto colaborativo
difícil o imposible. como Post-WIMP, puede proporcionarse a los usuarios
información de awareness sobre el entorno del juego (por
4. MODELADO DE UN SISTEMA POST- ejemplo, estado del jugador, posición de otros personajes, terreno
WIMP de juego, etc.). Concretamente, la información de awareness que
Para analizar el meta-modelo propuesto, presentamos un ejemplo será proporcionada por el sistema se describe en la Tabla 1.
de modelado que consiste en un juego First Person Shooter (FPS) Una vez explicadas las tareas del juego y las necesidades de
multi-dispositivo de realidad virtual. Este sistema hará uso de awareness, todo ello será modelado usando tanto el actual meta-
diferentes dispositivos para facilitar la interacción entre el usuario modelo de tareas de la W3C (en adelante W3CTM) como nuestra
y el sistema. En primer lugar utilizará un dispositivo de propuesta PWTM (Post-WIMP Task Meta-model).
visualización inmersivo para proporcionar al usuario los gráficos
3D así como para capturar los movimientos de cabeza del usuario Tabla 1. Información de awareness del FPS
para controlar la dirección de la visión de la representación virtual Tipo de Awareness Descripción
del usuario (personaje). Para proporcionar al usuario sonido Own status Información sobre el estado de salud del
posicional, permitiéndole saber la dirección de los disparos y los personaje y los power-ups activos
pasos de otros personajes, se dispone de auriculares de sonido Connected players Lista que mostrará tanto los jugadores
envolvente, los cuales incorporan un micrófono para comunicarse conectados como sus equipos.
con otros usuarios. Por otra parte para incorporar sensaciones Allies location Localización de los usuarios miembros del
hápticas se situarán varios actuadores vibrotáctites sobre el mismo equipo es mostrada en un mini-mapa
cuerpo del usuario para permitir que éste sienta en qué lugar del Power-ups position Las nuevas mejoras del arma y del personaje
cuerpo y cuándo el personaje es alcanzado. Se situarán 7 que se han desplegado en el juego se
muestran en el mini-mapa.
dispositivos colocados en cabeza, brazos, pecho y espalda. Para
Received Damage Cuando el personaje de un usuario es
controlar la posición de la mano dominante del usuario que simula alcanzado por una bala, el usuario sentirá una
el arma, se utilizará un tracker que recoja la posición de la mano. vibración en su propio cuerpo, obteniendo así
Por lo tanto, la dirección y el disparador/gatillo del arma del información de awareness de la dirección del
personaje serán controlados con este dispositivo. Finalmente, en disparo (por ejemplo si el personaje es
su mano no dominante, el usuario dispondrá de un mando para alcanzado desde atrás, el usuario sentirá una
controlar los movimientos del personaje (adelante, atrás y a los vibración en su espalda).
lados). Caused Damage Cuando un personaje es alcanzado por un
disparo se proporciona información auditiva.
Por otra parte, las tareas que un usuario realizará durante el juego Steps and Shooting Las armas y los pasos de los personajes
son: DefeatEnemy (tarea principal), Move, Communicate sound producirán sonidos que pueden dar
(dependerá de la distancia entre los jugadores), Revive Player (dos información de awareness sobre su
jugadores pueden devolver la vida a otro jugador del mismo localización.

Figura 5. Especificación de tareas usando el meta-modelo del W3C

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 99 |
 Limitaciones del Modelo de Tareas del W3C para aplica-ciones Post-WIMP
M. A. Teruel | A. C. Rodríguez | F. Montero | E. Navarro | V. López-Jaquero | P. González

Figura 6. Especificación de tareas usando el meta-modelo Post-WIMP

4.1 USANDO W3CTM PARA relacionada con la propiocepción del usuario y de las dinámicas
sociales y de grupo.
ESPECIFICAR LAS TAREAS DEL JUEGO
La Fig. 5 muestra una instancia del modelo obtenido usando el 4.2 Post-WIMP Task Meta-model PARA
meta-modelo de la W3C para nuestro ejemplo de modelado.
Como puede observarse, la tarea DefeatEnemy es la raíz del árbol
ESPECIFICAR LAS TAREAS DEL JUEGO
de tareas. Esta tarea se descompone en otras subtareas. Move, Tras especificar las tareas del ejemplo de modelado usando el
LookAt, RevivePlayer, Communicate, Listen y DeployPowerUp meta-modelo de la W3C, se procederá al modelado del mismo
que pueden ser realizadas de forma concurrente. DeployPowerUp sistema usando nuestra propuesta. Como puede observarse en la
habilita la tarea GetPowerUp puesto que ésta no puede ser Fig. 6, esta especificación enriquece la mostrada anteriormente
(Fig. 5) añadiendo nuevos elementos y relaciones que aportan
realizada si no hay power-ups en el escenario. GetPowerUp
habilita la tarea Shoot y ésta, ShowEnd. La tarea Move puede ser características de colaboración y awareness al modelado. Puesto
realizada de dos formas diferentes, por lo que se descompone que este nuevo modelo está basado en un lenguaje de requisitos
Walk y Run y el operador N-Ary entre estas tareas será el operador orientado a objetivos [15], se ha añadido el objetivo Achieve the
Choice. Además, el sistema muestra el movimiento sobre el mapa defeat of enemies al modelo de tareas, ya que es el principal
de forma concurrente. Para conseguir power-ups, el usuario debe objetivo de los usuarios que realizan la tarea Play Game. Además,
moverse a través del escenario virtual. Cuando el personaje pasa a tres recursos llamados Speed, Reach y Power, correspondientes a
través de un ítem virtual que representa el power-up, el sistema tres diferentes power-ups han sido incluidos en el modelo de
tareas. Otra diferencia es que las tareas de interacción han sido
informa al usuario mediante la tarea ShowPowerUp. La tarea Shot
se descompone en dos subtareas: Aim y PullTheTrigger. Ambas detalladas definiendo su tipo y prioridad. Como ejemplo, la tarea
tareas pueden ser realizadas de forma concurrente. Cuando un Communicate es considerada una Communication Task, ya que
equipo gana un juego, el sistema informa a los usuarios del final. permite a los usuarios de un mismo equipo comunicarse entre
El modelo, mostrado en la Fig. 5, presenta varias deficiencias de ellos. Las tareas Listen y Pull Trigger son consideradas
cara a posteriores fases del proceso de desarrollo. En primer lugar, Coordination Tasks. Estas tareas de coordinación son un tipo
no aparecen los participantes de las tareas ni el concepto de rol, lo especial de tareas colaborativas en las que distintas informaciones
que impide describir quiénes intervienen en tareas como (ruido y salud en nuestro caso) permiten coordinarse a varios
jugadores. Además, la tarea Revive Player, es una Collaborative
Communicate o RevivePlayer y el papel que juegan en las
mismas. Por otro lado, este modelo no nos permite definir la Task ya que dos diferentes personajes tienen que aproximarse a un
información de awareness necesaria para la realización de algunas tercero que ha muerto para poder revivirlo. Finalmente, otras
tareas. Por ejemplo, la tarea Communicate, según la propia tareas como Look At Game Scenario o Walk son tareas
especificación del ejemplo, sólo puede llevarse a cabo cuando la individuales que pueden ser realizadas por un solo usuario. Estas
posición de los participantes hace que estén lo suficientemente dos tareas han sido asignadas con la prioridad más alta (color
cerca, por lo que esta información es relevante para el desarrollo rojo) ya que deben ser implementadas en primer lugar para
del sistema. Por último, en un entorno virtual de inmersión obtener un prototipo de la aplicación lo antes posible. Por esta
completa, la tarea de dirigir la mirada hacia distintas zonas del misma razón, Aim también ha sido considerada una tarea de alta
prioridad, mientras que Listen y Deploy Power-Up han sido
escenario requiere de información de awareness que está
consideradas con una prioridad normal porque en nuestro primer

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 100 |
 Limitaciones del Modelo de Tareas del W3C para aplica-ciones Post-WIMP
M. A. Teruel | A. C. Rodríguez | F. Montero | E. Navarro | V. López-Jaquero | P. González
prototipo no es absolutamente necesario tener el sonido o el identificadas por Jacob [9], el modelo de awareness de CSRML,
sistema de power-ups totalmente desarrollado. llamado Gutwin’s Workspace Awareness [14] ha sido enriquecido
Como se muestra en la Fig. 6, la especificación Post-WIMP no con características especiales como Context Awareness o Social
tiene tareas de sistema. Cuando una tarea es realizada por el Awareness. Además, se ha presentado un ejemplo de modelado en
sistema, se indicará con la participación del Role “System” en el que se ha modelado un juego Post-WIMP tanto con la actual
dicha tarea (por ejemplo la tarea Deploy Power-Up es realizada propuesta del W3C como con el lenguaje presentado en este
por el sistema). Además, otros cuatro roles toman parte en esta trabajo. Esto nos ha permitido ilustrar que las características Post-
especificación: Player, Allies, Enemies y Human, que es el WIMP relacionadas con el awareness y la colaboración sólo
“super-role” de los tres primeros. Sin embargo, hay que señalar pueden ser especificadas con éste último.
que la asociación del usuario con los roles que puede jugar se
especifica en el modelo de usuario, no en el modelo de tareas, que A pesar de que este modelo ha mostrado ser más adecuado para
no se presenta aquí por problemas de espacio. sistemas Post-WIMP, es necesario profundizar más en este
Los Participation Links son usados para especificar qué usuarios sentido. Con el objetivo de mejorar la especificación de requisitos
están involucrados en tareas de interacción. Por ejemplo, dos de sistemas Post-WIMP, nuestros siguientes trabajos se centrarán
diferentes Participation Links están relacionados con la tarea en un modelo de contexto que complemente el modelo de tareas.
Communicate, porque cuando se realiza esta tarea un jugador (1) Así, mediante el uso de dicho modelo, será posible especificar
habla con uno o más (1..*) de sus aliados. cómo las aplicaciones Post-WIMP pueden adaptarse al contexto
Finalmente, cinco diferentes recursos de awareness se han de uso (es decir, hardware disponible, capacidades del usuario,
adjuntado a los Participation Links para representar la características del entorno físico como luz y ruido, etc.). Al
información de awarenes requerida para realizar las tareas. mismo tiempo, la implementación del caso de uso presentado en
Primero, el jugador debe ([M]) tener el recurso de Awareness este trabajo ya ha comenzado. Ya está disponible un prototipo
Allies location para proporcionarle la localización de sus aliados a jugable basado en la especificación de tareas creada mediante la
la hora de realizar la tarea Communicate, ya que la comunicación propuesta de meta-modelo de tareas post-WIMP recogida en este
sólo puede llevarse a cabo si los miembros del equipo están cerca. artículo.
Adicionalmente, sería deseable ([N]) tener información sobre 6. AGRADECIMIENTOS
dónde miran otros jugadores para coordinar ataques conjuntos.
Este trabajo está financiado por el Ministerio de Economía y
Además, cuando se realiza la tarea Look At Game Scenario, los
Competencia de España y por la fundación FEDER de la UE bajo
jugadores necesitan información de Awareness sobre su propio
el proyecto insPIre (TIN2012-34003). También ha sido financiado
estatus, algo que es altamente deseable ([HD]), para ser
por el Ministerio de Educación, Cultura y Deporte de España
conscientes de sus propias habilidades (power-ups). Finalmente,
gracias a la becas FPU (AP2010-0259).
es necesario que cada usuario sepa a qué grupo pertenece así
como los miembros que posee ese grupo. Por último, es deseable 7. REFERENCIAS
([D]) saber quiénes son los miembros de los equipos. Como se ha [1] A. van Dam, “Post-WIMP User Interfaces,” Commun. ACM,
mostrado, la especificación Post-WIMP permite especificar la vol. 40, no. 2, pp. 63–67, 1997.
colaboración y el awareness de forma adecuada, siendo estos dos
aspectos primordiales para el desarrollo de sistemas Post-WIMP [2] M. Gea and F. L. Gutiérrez, El Diseño. Capítulo 5 del libro
contemporáneos. Esta especificación de la información de electrónico de la Interacción Persona-Ordenador. Asociación
awareness solventa los problemas especificados en la sub-sección AIPO, 2002.
anterior. [3] “W3C: MBUI - Task Models.” [Online]. Available:
http://www.w3.org/TR/task-models/.
5. CONCLUSIONES Y TRABAJOS
[4] P. González, F. Montero, V. López, A. Fernández-Caballero,
FUTUROS J. Montañés, and T. Sánchez, “A Virtual Learning
A lo largo de los últimos años, no sólo la apariencia visual de las Environment for Short Age Children,” in IEEE International
interfaces de usuario ha cambiado, sino también la forma en la Conference on Advanced Learning Technologies
que las personas interactúan con los ordenadores. De hecho, las (ICALT’01), 2001, pp. 283–284.
clásicas interfaces WIMP están dando paso a nuevos sistemas
Post-WIMP que van más allá de ventanas, iconos, ratón y [5] M. A. Teruel, E. Navarro, V. López-Jaquero, F. Montero,
punteros. La interacción con estos nuevos sistemas está basada en and P. González, “CSRML: A goal-oriented approach to
reconocimiento de gestos, realidad virtual, dispositivos model requirements for collaborative systems,” in Lecture
corporales, etc. Debido a la evolución de los sistemas de Notes in Computer Science (including subseries Lecture
interacción, los mecanismos necesarios para especificarlos deben Notes in Artificial Intelligence and Lecture Notes in
hacerlo también. Este hecho supone la principal motivación para Bioinformatics), 2011, vol. 6998 LNCS, pp. 33–46.
la realización de este trabajo, que propone el desarrollo de un [6] M. A. Teruel, E. Navarro, V. López-Jaquero, F. Montero, J.
modelo de tareas alternativo a la actual revisión del modelo del Jaen, and P. González, “Analyzing the understandability of
W3C, para adecuar el modelado de tareas a la especificación de Requirements Engineering languages for CSCW systems: A
requisitos de sistemas Post-WIMP. La colaboración es uno de los family of experiments,” Inf. Softw. Technol., vol. 54, no. 11,
aspectos clave en dichos sistemas, por lo que nuestra propuesta se pp. 1215–1228, 2012.
basa en el modelo de tareas Collaborative Systems Requirements
[7] M. A. Teruel, R. Tardío, E. Navarro, A. Maté, P. González,
Modelling Language (CSRML) [5], que implementa el
J. Trujillo, and R. Terol Muñoz, “CSRML4BI: A Goal-
ampliamente aceptado modelo de colaboración 3C [10]. Con el
Oriented Requirements Approach for Collaborative Business
objetivo de hacer que este modelo tenga la capacidad de
Intelligence,” in 33rd International Conference on
especificar las necesidades de awareness de sistemas Post-WIMP
Conceptual Modeling (ER’14), 2014, pp. 423–430.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 101 |
 Limitaciones del Modelo de Tareas del W3C para aplica-ciones Post-WIMP
M. A. Teruel | A. C. Rodríguez | F. Montero | E. Navarro | V. López-Jaquero | P. González
[8] F. Paterno, “Model-based design of interactive applications,” [12] A. García and J. Molina, “Enhancing collaborative
Intelligence, vol. 11, no. 4, pp. 26–38, 2000. manipulation through the use of feedback and awareness in
[9] R. Jacob, R. Jacob, A. Girouard, A. Girouard, L. Hirshfield, CVEs,” Proc. 7th ACM SIGGRAPH Int. Conf. Virtual-
L. Hirshfield, MS, MS, 2008, and 2008, “Reality-based Reality Contin. Its Appl. Ind. ACM., p. 32, 2008.
interaction: a framework for post-WIMP interfaces,” in [13] M. A. Teruel, E. Navarro, V. López-Jaquero, F. Montero,
portal.acm.org, 2008, pp. 201–210. and P. González, “A CSCW Requirements Engineering
[10] I. Steinmacher, A. P. Chaves, and M. A. Gerosa, “Awareness CASE Tool: Development and usability evaluation,” Inf.
support in global software development: A systematic review Softw. Technol., vol. 56, no. 8, pp. 922–949, 2014.
based on the 3C collaboration model,” in Lecture Notes in [14] C. Gutwin and S. Greenberg, “A descriptive framework of
Computer Science (including subseries Lecture Notes in workspace awareness for real-time groupware,” Comput.
Artificial Intelligence and Lecture Notes in Bioinformatics), Support. Coop. Work, vol. 11, no. 3–4, pp. 411–446, 2002.
2010, vol. 6257 LNCS, pp. 185–201. [15] M. A. Teruel, E. Navarro, V. López-Jaquero, F. Montero,
[11] C. A. Ellis, S. J. Gibbs, and G. Rein, “Groupware: some and P. González, “Comparing Goal-Oriented Approaches to
issues and experiences,” Communications of the ACM, vol. Model Requirements for CSCW,” in Evaluation of Novel
34, no. 1. pp. 39–58, 1991. Approaches to Software Engineering, L. A. Maciaszek and
K. Zhang, Eds. Springer-Verlag, 2012, pp. 169–184.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 102 |
 An Ontology-Driven Approach to Model & Support Mobility and GeoLocation Based Campus Interactions
M. Khemaja | F. Buendía

An Ontology-Driven Approach to Model & Support Mobility

and GeoLocation Based Campus Interactions
Maha Khemaja Félix Buendía
PRINCE, ISITC, H Sousse Computer Engineering Dept.
University of Sousse, Tunisia Universitat Politècnica Valencia
+216 23501625 +34 963877000 (ext. 75734)
[email protected] [email protected]

ABSTRACT Therefore, there is a need to incorporate this kind of context

information based on the user's profile and preferences, the user's
University campuses are increasingly becoming places where location, the status or the scheduling of an activity that is
different interactions take place including those related to occurring at the present time. This idea was advanced by authors
instructional and learning tasks. Most of these interactions require such as Weiser [[1]] or Crook & Barrowcliff [[2]] who believed in
gathering contextual information from the campus facilities, the potential of ubiquitous computing to offer new tools and
platforms, services or people that are part of this environment. The patterns of student’s interactions. Such potential was analyzed in
current work aims at providing an ontology-based proposal, which 13 institutions that provided "ubiquitous computing" in very
can enable to model most of these context features that can be different ways [[3]]. DeGagne and Wolk [[4]] examined how
useful to support multiple kinds of interactions in Campus college students used technology in their everyday life on a
settings. This proposal permits to reason and make inferences on campus with ubiquitous computing and communication.
contextual information and features in such a way to provide users Currently, many universities have platforms and apps like
with mobile geolocated services such as path finding, small group iStanford 1 or UCC2 that provide generic coarse-grain contextual
meetings planning or personalized information about campus information useful for their students’ lifetime. Nevertheless, the
facilities. The proposed ontology has been applied in a University support of specific context-aware interactions usually requires a
campus with the purpose of guiding users to improve their more fine-grain approach that represents context concepts and
personal experiences. their relationships in a more detailed way. Several initiatives have
been promoted for applying ubiquitous computing technologies
Author Keywords for campus environments in specific cases.
Ontology proposal; Campus interactions; context modeling;
mobile apps; geolocation services. For example, Aware Campus [[5]] or Active Campus [[6]] explore
location-aware applications addressed to guide users and allow
them to annotate or share location information with others.
ACM Classification Keywords MyCampus [[7]] is another application that was conceived to
enhance everyday campus life at Carnegie Mellon University
H.5. Human computer interaction (HCI). Ubiquitous and mobile
introducing context-aware agents, recommender services and case-
computing, mobile devices.
based reasoning. Moreover, Scott & Benlamri [[8]] describe
1. INTRODUCTION several initiatives to move traditional learning spaces into smart
University campuses are today organized as small or medium-size ones by using ubiquitous context-aware services.
towns where different types of interactions take place. They The current work focuses on the modeling of contextual
usually involve common services such as resident information that can be useful to support Campus interactions. It is
accommodation, catering, or social meetings but, obviously, the practically impossible to cope with every potential interaction in
most demanded services deal with learning activities or student this type of settings but a possible subset or interaction patterns
life in the campus. Even in this type of learning services, there is a can be defined. For example, outdoor activities can be performed
wide range of possibilities that are based on indoor/outdoor in several disciplines from architecture; agricultural studies to arts
scenarios, lecture vs. lab or seminar sessions, presential or online either for collaborative or individual tasks. These activities may
activities, individual or collaborative tasks as well as the diversity require a certain support beyond the academic works to be
of disciplines and domains that are taught in higher education. developed within them. Therefore, models have to be provided to
both represent information items about the context surrounding the
Moreover, campuses increasingly provide spaces for exhibiting
target learning activity and reason and infer related knowledge and
artistic masterpieces or historical work and seek to attract and
information. From these models, context aware mobile and
inform new comers about them. Most of these campus facilities
geolocation apps addressed to campus settings can be further
are generally supported by information systems (internal data
developed.
bases, web portals, intranet sites) allowing users to query for
specific information as lecture schedules, labs availability, tutorial
sessions or specific artistic show location. However, these
information systems do not take into account one’s user context 1
https://itservices.stanford.edu/service/mobile
and act only on a reactive manner to user’s explicit requests.
2
http://www.usca.edu/ucc/index2.html

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 103 |
 An Ontology-Driven Approach to Model & Support Mobility and GeoLocation Based Campus Interactions
M. Khemaja | F. Buendía
The rest of the paper is organized as follows. Section II presents axioms and rules and finally, instances are built and described. (3)
some works related to the proposed approach. Section III Ontology Formalization (in a formal or semi-formal) model (4)
describes the main features of the overall approach with the Ontology Implementation and validation using an Ontology
ontology proposal and the development process fundamentals. development framework or a programming environment. In order
Section IV illustrates the application of the presented approach, to define the domain and scope of the ontology, a list of
and, finally, Section V gives some concluding remarks. competency questions that a knowledge base should be able to
answer are gathered [[17]]:
2. RELATED WORKS • What are the elements that compose the user's context in
Along last years, several ways to model context issues have been a campus setting?
proposed. Strang [[9]] established that a well designed model is a • What kind of contextual information could enable a
key element in any context-aware system and identified several proactive decision making about user's interactions as,
approaches from the more simple based on key-value data for example, indoor or outdoor activities?
structures to mark-up scheme, graphical, object oriented, logic- • What kind of activities could engage a given user?
based or ontology models. Ontology notations represent a • What kind of scenarios and activities could be carried
powerful semantic mechanism that enable not only to reason or out by campus actors accordingly to their preferences
infer new information but also to share it in different ways. In this and profiles?
sense, we can differentiate generic ontology proposals that have • Which kind of information will allow users to plan
been applied to campus samples from other specifically addressed meetings and through which campus facilities?
to this environment. CoBrA-ONT [[10]] is an example of ontology How could new campus users locate buildings, sculptures, other
for context-aware pervasive computing environments that was persons or different objects in the campus?
applied in campus context. The CoCA (Collaborative Context-
Aware) service platform [[11]] is based on a hybrid context To answer these competency questions, several taxonomies are
management model and it was used to build a Pervasive Campus- identified. For instance, a context element taxonomy illustrated in
Aware Smart Onlooker (PiCASO) scenario. The SeCOM model Figure 1 constitutes as explained in [[18]] a first layer of elements
[[12]] was used to develop a Geomobile system at the FUV or primary context types for characterizing the situation of a
(Federal University of Viçosa) that contained information about particular entity. These context types act as access points into
the structure and activities carried out in the campus. Its context other sources of contextual information called secondary context
model was based on ontological concepts such as Actor, Activity, (e.g. the email address of a given user) for the entity as well as the
SpatialEvent and Temporal Event. This Geomobile system was primary context for other related entities (e.g. other people in the
developed using the C#NET language and addressed to PocketPC same location).
emulators though the conceptual ideas are similar to those
proposed in the current paper. Moreover, the SeCOM model was
used to develop the iCas system [[13]] based on a Java+XHMTL
framework to support students and teachers in their campus life
using OWL services. Otherwise, Geo-Wiki [[14]] provided a
semantic geographical knowledge based on Google Maps
technologies but it is addressed to specific wiki services.

3. OVERALL APPROACH
The main idea in the current work aims at providing an ontology-
based proposal which can enable, first, to model most of those
context features that can be useful to represent Campus
interactions, and, second, to support a development process, which
helps with the elaboration of mobile geolocated applications in
this context.
3.1 Context ontology Figure 1. Context Ontology model.
A methodology addressed to model the ontological context has
been applied. First, the category of the ontology to develop is Figure 2 shows part of the Context Ontology that models different
identified. Secondly, a development process inspired from several types of activities and their related interactions. The Activity
ontology construction methodologies is applied [[15]], [[16]]. taxonomy illustrates possible activities that could engage users in
Ontology construction methodologies promote the following basic Campus settings where the Learning Activity concept is
phases: (1) Ontology Specification which specifies the ontology distinguished from Recreational Activity Concept while the
purpose, domain and scope and it requires to decide whether or Facility taxonomy represents items and elements provided by the
not to reuse existing ontologies, (2) Ontology Conceptualization in campus infrastructure that link them to the corresponding
which the important terms of the ontology are enumerated, activities. Figure 3 shows part of the ontology model that
taxonomies i.e. classes and classes hierarchies are defined, binary represents the Geographic area entity where a specific Activity is
relationships, class and instance attributes or properties, formal developed.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 104 |
An Ontology-Driven Approach to Model & Support Mobility and GeoLocation Based Campus Interactions
M. Khemaja | F. Buendía

Figure 2. Campus Activity Ontology model

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 105 |
 An Ontology-Driven Approach to Model & Support Mobility and GeoLocation Based Campus Interactions
M. Khemaja | F. Buendía
The Geographic Area considered in Campus settings deals with Area and may be part of a Path that could be followed by a
physical elements that can be found in a Campus together with Campus user.
their organizational aspects. The concept of POI (Point of
Interest), for example, is described as a GeoLocation point having
a certain textual description. It is located in a certain Location

Figure 3. Campus GeoLocation Ontology Model.

All concepts, their properties and types are defined clearly in a Environment The surroundings or conditions in which a
Glossary as well as in a dictionary. Table 1 contains main person lives or operates.
ontology terms representing classes or entities such as User, The setting or conditions in which
Activity, and POI concepts. This latter is specifically interesting a particular activity is carried on
because it represents a specific location in a campus that could Campus The grounds and buildings of a university
attract new comers or students and which could be useful for them
either for recreational, learning or cultural related activities. Scenario The way in which a situation may develop.

Table 3. Ontology elements. Geographical An area, especially part of a country or

area the world having definable characteristics an
Name Description d boundaries
Location A particular place or position
Context The set of circumstances or facts that surroun
d a particular event, situation, etc Point of A specific point location that someone may
Context A kind of circumstance or fact composing Interest find useful or interesting
element context Path A road, way, or track made for a particular pu
User/End The person or the organization that is part of rpose. The route or course along which a
User the campus environment and to which the person moves
software/applications are intended. Campus Something (such as a building or large piece
Activity A thing that a person (user) or a group does Facility of equipment) that is built for a specific
or has done purpose. It makes an action, operation, or

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 106 |
 An Ontology-Driven Approach to Model & Support Mobility and GeoLocation Based Campus Interactions
M. Khemaja | F. Buendía
activity easier. logic should also be taken into account as illustrated by Figure 4.
This architecture model based on MVC (Model View Controller)
Moreover, constraints on concepts and relations between them are enables a development process which could be carried out by a
identified and expressed as axioms or domain/range constraints. development team with different roles and skills.
The ontology expressivity power is enhanced by a set of logical
rules. These rules are defined and used to enable inferences and
reasoning in the context of the proposed ontology. Table 2 shows
an excerpt of such axioms and logical rules.
Table 2. Ontology Constraints and Rules.
Constraint Example
tDisjoint An object or instance of student for example
classes could not be an instance of faculty. An
individual activity cannot be at the same time
a collaborative activity.
If Student(?S) and Faculty(?F)
→isDifferent(?S,?F)
If Activity(?A) and isCollaborative(?A) →
Existential A Path with POI is a path that fulfils the
restriction constraint at least one POI exists and pertains
to the path
isPath_With_POI(?P) → ∃ POI(?po) and
isLocatedIn(?po,?P)

Rule Example

Add_GUI_ A user U who's GUI interface is a Map m and

Component who have visited a POI p1 at Time t1 and
who's next planned Activity A at Time t2 is to
visit another POI p2 is informed about the
next POI location by adding a Marker to his Figure 4. Application architectural model.
interface:
In order to grant a best user experience, the interaction model as
If IsLocated (U, p1,t1) and well as the user interface should be intuitive, ergonomic and
Next_InPath(p1,p2) and Activity(A,t2)="Visit compliant to the user's habits and preferences. It should also avoid
POI" → her/him having an overloaded view. User inputs are requested
set_Up_GUI_Component(Marker(Map only when they are strictly necessary. The user's interactions are,
m,p2)) therefore, driven by a set of components that are able to detect
her/his context and provide her/him consequently and seamlessly
Draw_Directi A user U who's current Location l is far from with the best fitted services.
on_Path form the POI p2 to be visited is informed
about the next POI location by drawing Taking into account all these considerations, the core application
Direction path on his interface: process structure addresses these actions (1) initialize the first
application's State, (2) sense and analyze Context info, (3) update
If IsLocated(U,l,t1) and IsFarFrom(l,p2) → the application State, (4) display the next State. The process
Draw_Direction_Path(p1,p2) continues at point (2) until the user explicitly stops the application.

Apply_Zoom When the distance between two POIs is small The development process targeting this kind of application is
it is better to ZoomUp the Map: incremental. It also follows a "bottom-up" approach through the
next steps:
If Distance (p1, p2) < 2 Km
→Apply_Zoom(m, 13) • Identify possible users’ needs and interaction rules that
drive the application behavior.
• Identify and develop information sources and data
structures (Ontology sources and Databases).
3.2 Development process • Identify existent components that could be reused and/or
Modern software development is increasingly requiring separation reassembled as for instance, third party services such as
of concerns. Therefore, each component to be developed should those provided by Google Play services, Weather
individually address a different purpose and brings itself Forecast services or also transportation and traffic
contribution to achieve the global application goal. Separation services.
between the user's interface, the interaction mode and the business

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 107 |
 An Ontology-Driven Approach to Model & Support Mobility and GeoLocation Based Campus Interactions
M. Khemaja | F. Buendía
• Identify issues such as User Interface structure, on the action to be realized as, for example, to add one GUI
components, GUI services and attributes. component or to update one another.
• Develop context sensing and Events Listeners
components.
• Develop Events handlers and implement their
interactions with business services.
• Develop business services that firstly, reason and
request ontology sources and databases and, secondly,
act on the user's interface components accordingly to
interaction rules by triggering GUI services.
In the context of the present work, point (1) and (2) are
represented by developed ontologies as well as their associated
rules.
The point (4) considers Widgets and Graphical User Interface
components provided by mobile devices.
The point (5) relies on sensor modules (e.g. GPS) present on the
user's devices, on remote services (e.g. Weather services) as well Figure 5. Services taxonomy.
as on the user's interaction with the app GUI components. Each 4. SAMPLE APPLICATION
detected event (e.g. Location changed) will trigger the relevant As a proof of concept, a sample application is provided that deals
Event handler. Mappings between Event Handlers and capabilities with specific interactions within a University Campus. These
provided by services developed in point (6) are inferred interactions can be addressed to educational purposes for Campus
automatically which constitute the core of the interaction logic students but also to allow other users to take advantage of its
provided to the user. As illustrated by Figures 4 and 5, service services or visit its patrimony facilities. First, the geographical
categories vary from Decision Making to GUI Services. Moreover, campus environment is described. Thereafter, technical details of
these services collaborate to detect the user's context and decide the mobile application used within this environment are provided.

Figure 6. University Campus map.

Figure 7 shows a detailed view of one of the sectors represented in
4.1 Environment description the map. It is located close to the main street and there are two
The context of the sample application is based on an existing
campus entries labeled as L and M. There are also two near bus
University Campus. Figure 6 shows a map of the target campus,
stops and a tram station. The selected sector is composed by
which represents the geographical areas that integrate it. Each area
several buildings (6A-G), sports courts (green areas), two Meeting
is identified by a sector label (e.g. sector 2C) and it includes a set
Points (P6A and P6B), and a set of numbered red spots.
of buildings, objects and other campus facilities such as sport
arenas or yards. Additionally, this map displays some numbered
blue little squares that represent “Meeting points” for each campus
sector.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 108 |
 An Ontology-Driven Approach to Model & Support Mobility and GeoLocation Based Campus Interactions
M. Khemaja | F. Buendía

Figure 7. Campus sector map.

These spots are signaling special artistic points (e.g. sculptures
that are part of the Patrimony campus) which can be interesting for
Art students or an occasional visitor for cultural exhibitions at the
campus. Figure 8 shows a group of students observing a sculpture
called “Cactus” and listening to the tutor explanations. These
students are in a course of Sculpture Conservation and they want
to know those technical issues that can be harmful for sculpture
materials. This situation could be a sample scenario where
interactions can take advantage of geolocation services.

Figure 9. Campus path

The application is structured according to the architecture layers
illustrated in Figure 4, with self contained components and
services within each layer.
The resource layer constitutes information sources as databases
and ontologies for managing campus organizational data as well as
contextual information to be stored and processed. The context
ontology is implemented using the Protégé ontology editor and it
is expressed by means of the Ontology Web Language (OWL).
The business or application model layer contains services for
reasoning and querying the ontology using the SPARQL language
and the Andro Jena API. These services have been implemented as
an external Android service exposing its interface via the Android
Interface Definition Language (AIDL).
The controller layer implements several categories of listeners
Figure 8. Campus Sculpture exhibition. related specifically to context change events as, for example,
location changes or user's interactions with the map. These
4.2 Technical solution listeners are associated to event handlers which capture event
The sample application called OntoMapAndro has been developed information and delegate it to be processed by services in the
through an Android framework and deployed using a mobile application model layer.
device. It tracks the user's location and allows her/him to interact
Decisions made by this layer after considering new contextual
with a main GUI component which is a Map representation of the
information may specifically trigger GUI services to update the
campus space, interesting POIs and paths (see Figure 9) within the
user's Interface. For example, Figure 10 shows an OntoMapAndro
previously introduced campus environment.
screenshot displaying information (text & image) about the object
that is currently visited. This information is displayed over the
MapView (Google Map based) as a consequence of triggering a
NextStep event.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 109 |
 An Ontology-Driven Approach to Model & Support Mobility and GeoLocation Based Campus Interactions
M. Khemaja | F. Buendía

Figure 11. Zoom level adaptation.

5. CONCLUSION
The current work has presented an approach to support the
modelling of contextual information in campus settings using a
systematic development process. The proposed approach has been
Figure 10. Object information display based on an ontology proposal focused on representing concepts
and relationships either for indoor or outdoor campus scenarios.
Finally, the view layer represents mainly the client components or This ontology includes the definition of axioms and rules that are
services that interact with the Google Map API. Client's services useful to reason or infer new information promoting learning
implement specifically functionalities to display maps, to change purposes or sharing knowledge. An implementation of this
map attributes as the zoom level and camera position, to also ontology has been developed using well known semantic
display the user's positions, add specific markers, and draw paths technologies such as Protégé, JENA and SPARQL tools
among two or more POIs. These information items are adapted demonstrating the feasibility of the approach.
according to the ontology rules defined previously. For example,
Figure 11 shows how the zoom level has changed because the Moreover, the developed ontology has been applied in the context
distance to the next object to visit is under a specific threshold. of a university campus showing its potential to model context
information that allows campus users to perform multiple kinds of
queries focused on different types of interactions. It is important to
highlight the possibilities provided by using ontology sources in
order to drive the application execution and improve its
adaptability in a changing context. Further works plan to address
development issues such as the generation of mobile apps in
multiple platforms and the improvement of accessing ontology
sources via web services. Another research line consists in
analyzing the users’ interactions and their behaviour when they are
using geolocated applications based on the current proposal. These
findings could lead to recognize mobile behaviour patterns and
provide better user adaptations.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 110 |
 An Ontology-Driven Approach to Model & Support Mobility and GeoLocation Based Campus Interactions
M. Khemaja | F. Buendía
Ontologies for Distributed Systems, Knowledge Review,
6. REFERENCES 2003, Acapulco MX.
[1] Weiser M. The future of ubiquitous computing on campus.
Communications of the ACM, 1996, 41(1):41–42. [11] Ejigu, D., Scuturici, M., Brunie, L. Hybrid Approach to
Collaborative Context-Aware Service Platform for Pervasive
[2] Crook, C., & Barrowcliff, D. Ubiquitous computer on
Computing. Journal of Computers, 2008,Vol 3, No 1, 40-50.
campus: Patterns of engagement by university students.
International Journal of Human-Computer Interaction,2001, [12] Lamas, A. R., Filho, J. Oliveira, A. P,. Júnior, R. M. A
13,245−256. mobile geographic information system managing
contextaware information based on ontologies. Ubiquitous
[3] Brown D.G. Ubiquitous Computing: The Universal Use of
Computing and Communication Journal, 2009, vol. 4, pp.
Computers on College Campuses. David G. Brown (Ed.)
71–727.
2003. Anker Publishing Company, Incorporated.
[13] Sousa, J.-P., Carrpatoso, E., Fonseca, B., Pimentel, M., Neto,
[4] DeGagne and Wolk. Unwired: Student Use of Technology in
R. A Service-Oriented Middleware for Composing Context
the Ubiquitous Computing World. Information Systems
Aware Mobile Services. Composition of context aware
Education Journal, 2007, 5 (13)..
mobile services using a semantic context model. International
[5] Burrell, J., Guy, G., Kubo, K., Farina, N. Context-aware Journal on Advances in Software, 2009, 2:2-3, p 275-287.
computing: A test case. In Proceedings of UbiComp Springer
[14] Gao, Y., Gao,S., Liu, R. Liu, Y. A semantic geographical
, 2002, 1–15.
knowledge wiki system mashed up with Google Maps.
[6] Griswold, W. G., Shanahan, P., Brown, S. W., Boyer, R., Science China Technological Sciences, 2010, Volume 53,
Ratto, M. , Shapiro, R. B.,. Truong T. M. Activecampus – Issue 1 pp 52-60.
experiments in community-oriented ubiquitous computing.
[15] Fernández-López, M., Gómez-Pérez, A., & Juristo, N.
Paper CS2003-0750, Computer Science and Engineering, UC
(1997). Methontology: from ontological art towards
San Diego (2003).
ontological engineering. Proceedings of the Ontological
[7] Sadeh, N. Gandon, B. and Kwon, O.B. Ambient Intelligence: Engineering AAAI-97 Spring Symposium, 1997.
The MyCampus Experience. Chapter 3 in Ambient
[16] Gómez-Pérez, A., & Suárez-Figueroa, M. C. Scenarios for
Intelligence and Pervasive Computing, Eds. T. Vasilakos and
building ontology networks within the NeOn methodology.
W. Pedrycz, ArTech House.
In ACM Proceedings of the fifth international conference on
[8] Scott, K., Benlamri, R. R. Context-Aware Services for Smart Knowledge Capture, 2009 pp. 183-184.
Learning Spaces, IEEE Transactions on Learning
[17] Grüninger, M., & Fox, M. S. Methodology for the Design
Technologies, 2010, vol.3, no. 3, pp. 214-227.
and Evaluation of Ontologies. Workshop on Basic
[9] Strang, T., Linnhoff-Popien, C. A Context Modeling Survey, Ontological Issues in Knowledge Sharing, 1995.
in: J. Indulska, D.D. Roure (Eds.), Proceedings of the First
[18] Abowd, G. D., Dey, A. K., Brown, P. J., Davies, N., Smith,
International Workshop on Advanced Context Modelling,
M., & Steggles, P. Towards a better understanding of context
Reasoning and Management, in conjunction with UbiComp
and context-awareness. In Handheld and ubiquitous
2004, University of Southampton.
computing. International Symposium on Handheld and
[10] Chen H., Finin T., Joshi A. An ontology for context-aware Ubiquitous Computing, 1999. pp. 304-307.
pervasive computing environments. Special Issue on

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 111 |
 SEGA-ARM: A Metamodel for the Design of Serious Games to Support Auditory Rehabilitation
D. Céspedes-Hernández | J. L. Pérez-Medina | J. M. González-Calleros | F. J. Álvarez Rodríguez | J. Muñoz-Arteaga

SEGA-ARM: A Metamodel for the Design of Serious

Games to Support Auditory Rehabilitation
David Céspedes-Hernández1, Juan Manuel González- Jaime Muñoz-Arteaga1
3
Jorge Luis Pérez-Medina2, Calleros3, Francisco J. Álvarez Benemérita Universidad Autónoma
1 de Puebla
Universidad Autónoma de Rodríguez1,
Aguascalientes 2
Université catholique de Louvain Av. 14 Sur y San Claudio, Puebla,
Av. Universidad 940, Ags., Mexico Place des Doyens 1 Mexico
{dacespedes, fjalvar, Louvain-la-Neuve, Belgium [email protected]
jmunozar}@correo.uaa.mx [email protected]
ABSTRACT to provide an application that may be used to help on the
Hearing impairments are widely recognized as one of the most performance of assisted therapy sessions, but to propose a
common disability that affects many people. There are several definition of modeling elements along with their relationships and
works addressing the design and development of systems for the rules in order to enable the creation of semantic models, that may
rehabilitation of users with disabilities, as well as for the later result on the implementation of a family of serious games
definition of models to express their capabilities and disabilities. with the purpose of supporting the process of auditory
Unfortunately, when we want to develop interactive systems to rehabilitation, meaning by this, a metamodel [12] for the design of
assist hearing impairment therapy, knowledge of analysis and serious games to support auditory rehabilitation.
design, typical of the methodologies is absent. In this paper, we The rest of the paper is structured into sections. Section two
introduce SEGA-ARM: a metamodel to support the design of presents a state of the art with relevant concepts to auditory
serious games for auditory rehabilitation, considering concepts rehabilitation and serious games, as well as other related works,
related to speech therapy sessions, user capabilities, the context of and the description of useful techniques and tools for the
use, and concepts related to serious games design. The proposed development of this project. For the description of the elaborated
solution is provided using a semi-formal UML notation. Two model, section three firstly presents the packages in which it is
game prototypes are presented to validate the viability of the divided. Later, in order to facilitate the comprehension of the
application of the metamodel for serious games development and model’s classes, a case study is introduced. Finally, instancing the
to point out its reutilization and extension. case study, each one of the packages’ element is described. In the
fourth section, a discussion on the characteristics of the full model
Categories and Subject Descriptors and the relationships that exist between classes of its different
H5.2 [Information interfaces and presentation]: User packages is carried out, introducing a second case study. Last,
Interfaces–Prototyping; user-centered design; K8.0 [Personal conclusions and future work are mentioned in section number
computing]: Games. five.
General Terms 2. STATE OF THE ART
Design, Theory
2.1 Concepts on Auditory Rehabilitation
Keywords In order to better understand the aim of the present work, a set of
Model-driven development, speech therapy, serious games, concepts relative to auditory rehabilitation and serious games has
Human-computer interaction, hearing impairment, cochlear to be defined.
implant, user-centered design.
Rehabilitation is the dynamic process by which physical, sensory,
1. INTRODUCTION and mental capacities are restored or developed in people with
In several contexts such as healthcare, military and educational, disabling conditions [13]. When it comes to rehabilitation for
games have been applied as a way to improve skills and for auditory disabilities, it is covered mainly by speech therapy and
training professionals [1] [2]. This type of games is called serious reinforced by occupational and physical therapy. The success of a
games [3] [4]. Despite the fact that the concept of serious games rehabilitation program depends on various factors: timing, patient
itself, might be considered as an oxymoron since “Games are diagnosis and treatment planning [14]. For the purposes of this
inherently fun and not serious” [5], it is important to clarify that particular work, only speech therapy is addressed. Speech therapy,
many works had been conducted in the field addressing the also known as speech-language pathology, is the rehabilitative
development of these games, always considering fun as a main treatment of physical and/or cognitive disorders resulting in
component of them [6]. Several works addressing the design and difficulty to perform verbal communication. Speech therapists
development of systems for the rehabilitation of users with asses, diagnose, treat and help to prevent disorders related to
disabilities [7] [8], as well as for the definition of models to speech, language and other elements of communication [15]. The
express people’s capabilities and disabilities [9] [10] had been ultimate goal of these specialists is to help patients develop or
carried out. However, even though hearing impairment is widely recover reliable communication and other skills. Most speech-
recognized as one of the most common disabilities [11], there is a language pathologists work with determined age groups, such as
lack of work focusing on providing tools to ease the design and children or elderly and focus on the treatment of certain
development of applications for the domain of auditory communication problems, such as those resulting from strokes or
rehabilitation. In this sense, the objective of this work is not only hearing loss [16].

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 112 |
 SEGA-ARM: A Metamodel for the Design of Serious Games to Support Auditory Rehabilitation
D. Céspedes-Hernández | J. L. Pérez-Medina | J. M. González-Calleros | F. J. Álvarez Rodríguez | J. Muñoz-Arteaga
Hearing impairment may be caused by different factors, such as, 2.3 Related Work
accidents, viral diseases, noise, and genetic causes [17]. Software designers may achieve the goal of expressing the
Depending on the background of the patients with hearing capabilities and disabilities, along with other valuable insights of a
impairment, among other classifications, they can be categorized user to developers via “Personas” models [9]. “Personas” are
into two groups: prelocutive (loss hearing toke place before the hypothetical archetypes of actual users and their description may
acquisition of spoken language) and postlocutive (patients whose consist on different aspects according to what is intended to make
loss of hearing has taken place after the development of the basic the development team aware of. Alternately, in [10] a metamodel
skills of spoken language) [18]. Once a hearing impairment has was proposed by Kaklanis to allow the expression of the whole
been diagnosed, the specialists involved on the patient’s user capabilities and disabilities with the intention of virtual user
treatment, evaluate the best option in order to improve his/her modeling. The main advantage that was encountered while
conditions. In this sense, some of the patients may be candidates comparing the “Personas” representation to this second proposal,
to receive a cochlear implant as the one mentioned in the case is that “Personas” provide only a natural language description of
study (Subsection 3.1) of the present work. the characteristics of the user, while Kaklanis’ disability and
A cochlear implant [19] is an electronic device that stimulates the capability models’ structure allows their representation on a
auditory nerve to allow sound perception. It is composed by an machine-readable format.
external component that receives incoming sound, processes it For the present work, the disability model by Kaklanis, shown in
and transfers the resultant signal across the skin. Another device Figure 1, is considered as it was defined, since it provides a
receives and decodes the transmitted signal and stimulates the generic description of any kind of disabilities with the affected
auditory nerve directly, bypassing the hair cells that implement tasks on it involved, while the capability model was in turn
the first stage of auditory neural processing in normal ears. simplified to extract from it only those parts related to the hearing
According to the National Institute on Deafness and Other and speech skills, and enriched with characteristics that authors
Communication Disorders (NIDCD) on the United States of mention as relevant for the domain of speech therapy in children,
America, by December 2012, approximately 324,200 people such as loudness, speech rate and phrasing [22][23].
worldwide have received cochlear implants [20].
Concerning serious games design, an effort has been realized by
several authors, in order to abstract components that lead to the
definition of a guide on what to be aware of when developing a
game with these characteristics [24][25]. However there is not yet
a consensus on the components or elements that should be
considered when performing design and implementation, Table 1
presents some of the design elements that are usually taken into
account for the development of games [24-28].
Moreover, Longstreet & Cooper [29] proposed a conceptual guide
for development of serious games to support higher education,
their conceptual definition includes most of the design elements
reported on Table 1, but a structural notation was not provided
and the relationship between the involved elements is only
explained in a narrative manner, making its reutilization
Figure 1. The disability model proposed by Kaklanis [10]. complicated.
In this section, a general panorama of the domain concepts was
2.2 Concepts on Serious Games provided in order to facilitate the comprehension of further
To define a serious game, [21] begins by defining a game as a
sections. Also, related works are considered and described,
physical or mental challenge, that is played according to specific
pointing out their usefulness for accomplishing the objectives
rules, with the objective of amusing or entertaining the
presented on section one, and also reporting their limitations.
participant. Then, continues to build the definition of a videogame
as a mental challenge that is played through a computer according Finally, taking into account the concepts addressed in the last two
to certain rules for entertainment, fun or to achieve a goal. And subsections, a decision was made to consider the proposition of a
finally, provides the definition of a serious game as a mental metamodel composed by three main packages: context, theoretical
challenge, played through a computer according to specific rules, therapy, and performed therapy, and to add it the definition of a
which uses entertainment mechanisms to achieve training or serious games design elements extension.
capacitation goals in areas such as education, health and military.
Notice that in the rest of the paper, this is taken as the definition
of a serious game, just pointing out that it may also consist of
physical challenges (e.g., tongue movement, phoneme
pronunciation, and lips movement). The interest on the usage of
serious games on rehabilitation lies on the fact that is has been
demonstrated that they increase motivation towards therapy
sessions, which represents a major problem due to the repetitive
nature of exercises [14].

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 113 |
 SEGA-ARM: A Metamodel for the Design of Serious Games to Support Auditory Rehabilitation
D. Céspedes-Hernández | J. L. Pérez-Medina | J. M. González-Calleros | F. J. Álvarez Rodríguez | J. Muñoz-Arteaga
Table 4. Game design elements in the literature [24-28]. 3.1 Case Study
Game The case study consists on a scenario in which the design of a
Design Description serious game is required in order to support the therapy sessions
Element conducted by a speech therapist for a six year old male patient
The definition of characters allows user diagnosed with deep bilateral hypoacusia who has recently
engagement to the game, and provides the received a cochlear implant. The therapist has already performed
Characters possibility to define actions. Helps improving the tests over the patient to determine his exact condition and counts
player experience. Considers both, player and on a full expedient describing his capabilities towards hearing and
non-player characters speech skills. The intention with the required game is to extend
The rational of the game described in a narrative and complement the therapy sessions that the patient already
Narrative way, promotes the immersion of the player, helps receives with sessions conducted by himself in his home as those
description to define the needed resources, characters, game suggested in [30]. The language therapist wants the game to
mechanics and challenges. reinforce one of the therapy activities that the patient finds to be
The definition of challenges to be faced during repetitive since it has to be performed in almost every single
the gameplay, gives the player the opportunity to session.
Challenges
try his/her skills and to compete either with a
partner or with the game itself.
Determines the level of immersion that is
decided to use in the game. The most used point-
Type of view of-view perspectives for gameplay are: first-
person, third-person, third-person trailing,
overhead, and three-fourths isometric.
Constraints determine characteristics of the
Constraints challenges (e.g., limited time or resources,
competitive and sequential activities).
Establishing a reward system (e.g., points,
badges, levels, and leaderboards) allows
Rewards
improving the player experience and motivating
him/her to achieve a goal.
As with rewards, the establishment of
punishments, such as a decrement of points, and
Punishments
losing a life in the game, motivates the user to Figure 15. The exercise conducted in the therapist’s office.
achieve a determined goal.
The definition of interactive elements supports The exercise consists on that the therapist chooses a series of
Interactive phonemes sharing a consonant (e.g., ma, me, mi, mo, and mu),
the immersion of the player on a virtual world.
elements write them down on a notebook as shown in Figure 2, putting the
Their definition helps to design player tasks.
For each action performed by the player, the consonant on one column of the page and the vowels in another
game provides a reaction. In this way, the user is next to it, then asks the patient to pronounce each one properly
Feedback aware of the consequences of his/her acts on the while connecting the consonant with the vowels with lines, and
game and an appearance of continuous dialog is then repeat this task several times and with different phonemes.
given to him/her. This activity is designed to be performed in approximately ten
minutes and also considers phoneme visual recognition by the
3. SEGA-ARM: A METAMODEL FOR THE patient. It is also known by the therapist that the patient counts on
a 10.1” tablet with Android 4.2 OS. The information that the
DESIGN OF SERIOUS GAMES TO specialist possesses has been structured in terms of the here
SUPPORT AUDITORY REHABILITATION proposed metamodel as a way to facilitate the communication
For achieving the objectives mentioned on the introduction, a between the speech therapist and the game development team.
metamodel (SEGA-ARM) is proposed, considering for its The following section reports as an instantiation of the
definition concepts related to context, auditory rehabilitation, metamodel’s classes and how data was arranged and organized.
planned therapy, performed therapy, and serious games design
elements. The elaborated model is divided for its best 3.2 Description of the Classes Involved on the
comprehension and usage into three packages, presented in Figure Model
3, and an extension shown in Figure 4, each one of them identified The following subsections are for used for providing a detailed
by a different color and name, and designed to contain elements description of the packages and classes that take part on SEGA-
that support the design of serious games for auditory ARM. For every single package that is described, there is an
rehabilitation. instantiation in terms of our case study introduced on section 3.1.
In the rest of this section, SEGA-ARM is described in detail and 3.2.1 The Context Package
explained through a case study of a serious game design for A package designed to understand the Context [31] in which the
supporting the process of rehabilitation of a patient with a deep user interacts with the system, and to provide the developers with
hypoacusia condition, recipient of a cochlear implant. a brief description of relevant characteristics to make decisions
about what interactive modality to implement, and resources

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 114 |
 SEGA-ARM: A Metamodel for the Design of Serious Games to Support Auditory Rehabilitation
D. Céspedes-Hernández | J. L. Pérez-Medina | J. M. González-Calleros | F. J. Álvarez Rodríguez | J. Muñoz-Arteaga
selection among others. The Context is identified by an id and a and a user (Patient).
descriptive name, and conformed by an Environment, a Platform

Figure 3. The Context, Theoretical part of the Therapy, and Dynamical part of the Therapy packages with their relationships to
the serious game extension of the SEGA-ARM metamodel proposed.
The Environment describes the physical place in which the non-noisy interior environment, a 10.1” tablet with Android Jelly
interaction occurs. For the specific domain of this work, it is Bean 4.2 operating system, audio output, voice input and camera
necessary to point out if it is noisy or not, and to give it a name, an usage enabled, with android speech recognition/synthesis software
id and a type (i.e., exterior or interior). as platform, and the patient as user. Note that even though the
The Platform definition is carried out to determine the characteristics of the context must be taken into account by the
characteristics of the device that is going to host the application, serious game’s designer; the implemented solution may work
and is formed by a SoftwarePlatform and a HardwarePlatform. under other similar contexts as well.
The SoftwarePlatform represents those characteristics relative to 3.2.2 The Static Part of the Therapy Package
the software in the hosting platform i.e. operating system name The purpose that is pursued with the design of this package is to
and version, as well as the speech recognition/synthesis software provide the speech therapist with a semiformal notation to express
on it installed, while the HardwarePlatform class defines the characteristics of a therapy plan, keeping that information
characteristics regarding the hardware of the hosting platform and structured in a way that results understandable for serious games
relevant to the domain, such as the possibility of using sound designers. The definition of this part of the model was inspired
outputs, voice inputs or a camera. mainly by the work done by Kaklanis for modeling a Virtual User
In our example, the context in which the therapy sessions take [10], extended to allow the expression of a full auditory therapy
place is defined by a therapist’s office as a non-noisy, interior session program, and enriched with concepts found out to be
environment, has not platform and the user role on it is taken by relevant for patients of auditory rehabilitation on the literature
the patient, meanwhile, the context of use of the application that is [22] [23] and through the attendance to speech therapy sessions.
intended to be developed, is composed by the patient’s home as Abstracted from the full capability model [10] that describes

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 115 |
 SEGA-ARM: A Metamodel for the Design of Serious Games to Support Auditory Rehabilitation
D. Céspedes-Hernández | J. L. Pérez-Medina | J. M. González-Calleros | F. J. Álvarez Rodríguez | J. Muñoz-Arteaga
physical, cognitive and behavioral user characteristics, for the part of this paper, only considers the elements related to speech
specific domain of this work, the CapabilityModel presented as and hearing.

Figure 4. The serious games design elements extension provided for SEGA-ARM
The only parameter in the Hearing container is the Ear that in Corresponding to the identified disability, a rehabilitation
turn acts as one for the values related to each ear of the user: program was designed comprising approximately 300 hours, to
ResonanceFrequency and HearingThreshold. The Speech be provided in half an hour, two times a week sessions. One of
container includes Voice and Prosody elements. Voice elements the activities to be performed by the patient consists on
include Pitch and Loudness, each one with its measure units and repeating a series of phonemes in a determined order. The
values; while Prosody is formed by SpeechRate, VoiceStress therapist mentions a phoneme and expects the patient to repeat
and Phrasing. it. If a phoneme is not correctly pronounced, the therapist
A DisabilityModel [10], allows the description of all the reinforces its pronunciation and motivates the user to try again.
disabilities of the user as well as the affected by them tasks. For The objective of the task is to improve the patient’s
each disability, a name and a type (e.g. motor, auditory, and pronunciation and to get him/her used to repeat the sounds that
vocal) must be provided, and it is related to affected tasks. For a he/she listens to. For this particular activity, only a notebook and
determined DisabilityModel, a TheoreticalTherapy may be a color pencils are used.
proposed. A TheoreticalTherapy is a full rehabilitation plan and 3.2.3 The Dynamic Part of the Therapy Package
consists of a certain number of sessions and a series of In order to represent the current progress and performance of the
theoretical activities. A TheoreticalActivity is in turn, a series of Patient towards the therapy program, and to provide the
actions with a common objective to be performed by the patient development team with a structure to understand how to
either using or not PhysicalResources (e.g., a mirror, sticks or evaluate patient’s activities, the dynamic part of the therapy
cards). The planned duration, objective and description are package was created. This package contains then the Patient
attributes of a TheoreticalActivity. class, with an age attribute, related to a PerformedTherapy
For our case study, the disability and capability models which reports the number of completed sessions and groups
correspond to the diagnosis that was realized by the specialist in objects of type PerformedActivity.
speech therapy. The disability that was encountered is deep A PerformedActivity is based on a TheoreticalActivity and its
hypoacusia, and the affected tasks by it, are hearing and main goal is to keep a record of the advance presented by the
speaking. For the capability model, in the hearing branch, the patient while performing a given activity. In order to allow the
patient presents hearing thresholds of more than 90 dB nHL for evaluation of a PerformedActivity, a Metric class was also
frequencies of 1000, 2,000 and 4,000Hz in both ears; and a defined. Metrics allow the use of a formula in order to compute
resonance frequency between 800 and 1,200Hz. For the speech a value and to express it on a determined measure unit (e.g.,
branch, the patient’s voice was measured, getting as values for centimeters, seconds and decibels). For the evaluation of a
the pitch 250-300Hz, and 73dB for the maximum loudness. The formula, a series of Parameters i.e., criteria to be used while
prosody was also analyzed, obtaining for it typical results for a evaluating, may be defined. There could be nested metrics since
patient of the age and condition as the previously described. some formulas may need some others to be previously computed

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 116 |
 SEGA-ARM: A Metamodel for the Design of Serious Games to Support Auditory Rehabilitation
D. Céspedes-Hernández | J. L. Pérez-Medina | J. M. González-Calleros | F. J. Álvarez Rodríguez | J. Muñoz-Arteaga
in order to take values from them. Metrics in this domain may for this package, a series of challenges has to be designed to
be useful for measuring among others, tongue movement (i.e., represent the goals that are pursued while playing the game. A
tongue tip position, tongue vibration, and tongue position), lips Challenge consists of a description and is composed by at least
movement (upper lip movement and lower lip movement), one Constraint. Constrains may include one or several
pronounced phonemes and blow action. conditions (viz. time, limited resources, rewards, punishments,
In the context of the case study, the performed therapy and and mandatory collaboration).
performed activity classes keep track on how the patient is For the given case study, taking into account the description
evolving and advancing through the therapy program and their provided by the speech therapist for the context, theoretical and
involved values are constantly updated. The metric to be used dynamic packages, the serious game designer proposes to
for the specific theoretical activity described in section 3.2.2, elaborate a game with the characteristics that are presented in
consists on evaluating how a phoneme is pronounced by the Table 2 and which resultant user interface is shown in Figure 6.
patient, recognize it and compare it to an expected phoneme It is important to point out that the definition of some of the
passed as parameter according to a specific tolerance, using for game design elements was achieved through the establishment
this purpose the formula represented on Figure 5 where the of relationships between the therapy packages and the serious
pronounced by the patient phoneme (pp) is compared to the games extension (e.g. TheoreticalActivity and the Action). Next
expected phoneme (ep) and then, if the distance between them is section emphasizes this characteristic of SEGA-ARM along
smaller than the tolerance value (t), the utterance from the with others that had not been yet addressed.
patient is accepted as correct.
Table 2. Characteristics of the designed serious game.
Game name Froneme the frog.
Froneme the frog is in the edge of a river
looking for food to eat. In order to go find it, he
Figure 5. Formula for recognition considering expected has to jump between leaves avoiding to fall on
phoneme (ep), utterance (pp), and a tolerance value (t). the water. To perform a jump, the player has to
Narrative
identify the phoneme associated to the
3.2.4 The Serious Games Extension description
destination leaf and pronounce it. If the player
In attention to the main objective of this paper, an extension to is unable to provide a correct utterance,
the therapy metamodel for considering serious games design Froneme falls down into the water and
elements was elaborated. For the proposal of this extension, therefore the game is over.
mainly Longstreet´s approach is considered [29], enriched with
concepts obtained through a deeper survey on serious games Player The patient represented by a frog.
design elements [26-28]. The intention on the definition of this Mosquitoes representing Froneme’s food;
extension is to provide developers with a tool for passing from a Interactive leaves to allow Froneme jump from one to
narrative description of a game to a semiformal structure that objects another identified by a phoneme; water that has
may be related to the one of the core metamodel. A to be avoided by Froneme.
SeriousGame is composed by Mechanics to record the player Eat mosquitoes by landing on the leaves that
progress and to manage the punctuations, one or many instances they are on, jump from one leaf to another by
Player
of the Character class to define players as well as non-player pronouncing phonemes, and fall down on the
actions
characters (NPC), and SimulationGameElements such water if not correct utterance is provided on the
challenges, rules, scenarios and interactive objects. given time interval.
A Character object represents an entity that realizes actions. The game is designed with an isometric type of
These instances can be either a Player or a Non- view, and the following game situations:
PlayerCharacter, and need Resources in order to be properly Simulation Froneme in the border of the river (initial state
displayed. For the definition of NPCs, it is necessary to give environment of the game), Froneme on a free leaf, Froneme
them a name, decide the relationship that they maintain towards on a leaf occupied by a mosquito (eating), and
the Player, and when possible, provide their gender and cultural Froneme falling on the water.
information e.g. language, nationality, degree among others The challenges involved in the game are the
[29]. In order to handle events on the game and describe their following: jump from a leaf to another
consequences, it was necessary to define the Action class. An pronouncing a correct phoneme, jump from a
Action is represented by an event (e.g. key press, click, tap, leaf to another within a determined time
recognized word or movement, or scripts) that triggers it, along interval (time limit) to earn points (reward), or
Challenges
with a description, duration, and a script to execute as let Froneme to fall into the water (punishment);
consequence. Actions are performed by Players or executed by choose a leaf to jump on from a series of
NPCs and may whether use or not an InteractiveObject that are provided options (limited resources) and if
in turn, objects that belong to the SimulationEnvironment and there is a mosquito on the selected one, earn a
are created to support the game interaction, for instance, the bigger amount of points (reward).
scenario, the floor, and colliding objects. The For each time the player takes Froneme to a
SimulationEnvironment description contemplates the definition valid leaf, 10 points are going to be added to
of a type of view (e.g., first person, third person, or isometric) Mechanics the budget, If there is a mosquito on the
[26], and helps to hold the game situation i.e. level, state, and occupied leaf, ten extra points should be added.
maximum amount of errors [27] in a narrative manner. Finally The time available for producing the phoneme

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 117 |
 SEGA-ARM: A Metamodel for the Design of Serious Games to Support Auditory Rehabilitation
D. Céspedes-Hernández | J. L. Pérez-Medina | J. M. González-Calleros | F. J. Álvarez Rodríguez | J. Muñoz-Arteaga
is five seconds. Time and punctuations are
recorded and reported.

4. DISCUSSION
As it may be seen in Figure 3, there are relationships between
the therapy packages and some of the serious games design
elements extension classes. These connections allow the
developers to know where to extract information that is
supposed to be represented in the game, and how to present it.
Punctually, there are relationships between: the Patient
(Dynamic part of the therapy) and the Player (Serious games) to
remark the fact that the patient has to be considered as the player
and to get some information from his/her profile; the
PerformedTherapy (Dynamic part of the therapy) and the
Mechanics (Serious games), to track the advance of the player
through the game and give a feedback for the therapist; and the
TheoreticalActivity (Theoretical part of the therapy) and the
Action (Serious games) in order to understand the tasks that the
patient is supposed to perform while playing the serious game. It
is also important to point out that three levels were considered
while modeling the therapy: (1) the patient (capabilities and
disabilities), (2) the therapy program (theoretical therapy), and
(3) the involved activities to meet the plan (theoretical activity).
Each one of those three levels has equivalence on the dynamic
part of the therapy in order to keep track of the advance and
performance of the patient through the therapy. Figure 6. Screenshot of the serious game Froneme the frog.
When compared to the proposal in [29], our metamodel has the One of the aimed advantages of our proposal is reutilization, and
advantage of providing a semiformal notation which may be in order to demonstrate it, a second instantiation of serious
used in a methodological process as a conceptual guide for game, called Roberto the robot, was realized in which the very
achieving the goal of expressing and therefore implementing same definitions of the case study described in section 3 for the
serious games for a specific set of therapy activities. Along with context and for the static and dynamic parts of the therapy
its description, it also allows identifying the actor that is packages are taken into account, varying only the serious game
supposed to fulfill the different classes and attributes, and extension elements and therefore creating a different serious
provides a definition of the relationships that exist among them. game but for the same rehabilitation purpose. For this second
However, in this paper the use of knowledge bases and game, a character representing the player called Roberto the
taxonomies is not considered resulting on the lack of a robot is set on a scenario representing the moon surface looking
repository of activities to provide to the developers. for pieces to assemble a rocket to return to the Earth. In order to
find them, he has to run and jump over rocks avoiding hitting
them. To perform a jump, the player has to pronounce a required
phoneme. If the player is unable to provide a correct utterance,
Roberto hits a rock and finally after 5 impacts falls down. The
interactive objects are: small rockets representing rocket pieces,
and rocks that have to be avoided by the player. The player
actions are: to take rocket pieces by reaching them, jump over
rocks, hit rocks if no correct utterance is provided on a given
interval. The game was designed with a third person type of
view, and the following game situations: Roberto on the surface
of the moon, Roberto jumping over a rock, and Roberto reaching
a rocket piece. The challenges involved in the game are the
following: jump over rocks pronouncing a correct phoneme
(limited resources) within a determined time interval (time limit)
to earn points (reward), or hit rocks and fall (punishment); and if
a rocket piece is reached earn a bigger amount of points
(reward). As mechanics of the game, for each time Roberto
jumps a rock, 10 points are added to the budget, if a rocket piece
is reached, ten extra points are added. The time available for
producing the phoneme is five seconds. On the development of
this game, shown in Figure 7, also interface elements from the
Froneme the Frog game were reused. There are also works on
the definition of rules for defining different user interfaces as
output [32] but those objectives are out of the aim of this paper.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 118 |
 SEGA-ARM: A Metamodel for the Design of Serious Games to Support Auditory Rehabilitation
D. Céspedes-Hernández | J. L. Pérez-Medina | J. M. González-Calleros | F. J. Álvarez Rodríguez | J. Muñoz-Arteaga

7. REFERENCES
[1] Arnab, S., Dunwell, I., & Debattista, K. (2013). Serious
games for healthcare: Applications and implications.
Medical Information Science Reference.
[2] McDowell, P., Darken, R., Sullivan, J., & Johnson, E.
(2006). Delta3D: a complete open source game and
simulation engine for building military training systems.
The Journal of Defense Modeling and Simulation:
Applications, Methodology, Technology, 3(3), 143-154.
[3] Michael, D. R., & Chen, S. L. (2005). Serious games:
Games that educate, train, and inform. Muska &
Lipman/Premier-Trade.
[4] Breuer, J. S., & Bente, G. (2010). Why so serious? On the
relation of serious games and learning. Eludamos. Journal
for Computer Game Culture, 4(1), 7-24.
[5] Kuipers, D. A., et al. (2013). Design for Transfer. In
Serious Games Development and Applications (pp. 239-
Figure 7. Screenshot of the serious game Roberto the robot. 246). Springer Berlin Heidelberg.
It is important to note as another advantage of the here proposed [6] Ritterfeld, U., Cody, M., & Vorderer, P. (Eds.). (2010).
metamodel that since the packages included on it keep a certain Serious games: Mechanisms and effects. Routledge.
level of independence towards the serious games design [7] Kikuchi, T., et al. (2007, June). Quasi-3-DOF rehabilitation
extension, it is possible to use them, and to define new system for upper limbs: its force-feedback mechanism and
extensions in order to design and develop other types of software for rehabilitation. In Rehabilitation Robotics,
software solutions for auditory rehabilitation (e.g., diagnosis, 2007. ICORR 2007. IEEE 10th International Conference on
tracking, and advising systems). Likewise, by developing (pp. 24-27). IEEE.
models for other domains than auditory rehabilitation such as [8] Burdea, G., Popescu, V., Hentz, V., & Colbert, K. (2000).
educational and military, it may be possible to relate them to the Virtual reality-based orthopedic telerehabilitation.
extension here provided to enable the development of serious Rehabilitation Engineering, IEEE Transactions on, 8(3),
games for those ambits. 430-432.
[9] Casas, Roberto, et al. User modelling in ambient
5. CONCLUSIONS AND FUTURE WORK intelligence for elderly and disabled people. Springer Berlin
In this paper we proposed SEGA-ARM, a metamodel for the Heidelberg, 2008.
design of serious games to support auditory rehabilitation. The [10] Kaklanis, N., A Coupled User and Task Modelling
mentioned model consists of three packages, regarding the Methodology for Accessible Product Design. PhD
context of use, the theoretical part of the therapy, and the dissertation, University of Surrey. March 2013.
dynamic part of the therapy; and an extension for its [11] Ventry, I. M., & Weinstein, B. E. (1982). The hearing
implementation through serious games. As the description of handicap inventory for the elderly: a new tool. Ear and
every single package and class was provided, the actors on it hearing, 3(3), 128-134.
involved (i.e., designer, development team, and speech therapist) [12] Koch, N., & Kraus, A. (2003). Towards a common
were mentioned along with tasks to be performed by them, and a metamodel for the development of web applications. In
case study was in it instanced to allow a better understanding. Web Engineering (pp. 497-506). Springer Berlin
Two prototypes of serious game to support a therapy activity Heidelberg.
were developed. In the near future the developed serious games [13] Brandt Jr, E. N., & Pope, A. M. (Eds.). (1997). Enabling
are going to be evaluated by a speech therapist in order to America: Assessing the Role of Rehabilitation Science and
determine if it is possible to include them at first as support tools Engineering. National Academies Press.
for in-consulting-room therapy sessions, and then as a way to [14] Rego, P., Moreira, P. M., & Reis, L. P. (2010, June).
complement and extend the received therapy sessions with Serious games for rehabilitation: A survey and a
sessions conducted by the patient himself. Also, more extensions classification towards a taxonomy. In Information Systems
are going to be developed in order to verify the completeness of and Technologies (CISTI), 2010 (pp. 1-6). IEEE.
the metamodel for different scopes. [15] McDonnell, A., & Akruvala, S. D. (2012). Rehabilitative
Therapists in Geriatric Health Services. Managing Geriatric
6. ACKNOWLEDGEMENTS Health Services, 45.
We acknowledge the support that was given by CONACYT, [16] Bureau of Labor Statistics, U.S. Department of Labor,
The Université catholique de Louvain and the Autonomous Occupational Outlook Handbook, 2014-15 Edition,
University of Aguascalientes for the development of this work Speech-Language Pathologists, on the Internet at
through a research stay program. We also acknowledge http://www.bls.gov/ooh/healthcare/speech-language-
Professor Jean Vanderdonckt from Université catholique de pathologists.htm (visited March, 2015).
Louvain for his valuable support on the development of this [17] World Health Organization, Fact sheet No.300 Updated
project. March 2015, on the Internet at
http://www.who.int/mediacentre/factsheets/fs300/en/
(visited March, 2015).

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 119 |
 SEGA-ARM: A Metamodel for the Design of Serious Games to Support Auditory Rehabilitation
D. Céspedes-Hernández | J. L. Pérez-Medina | J. M. González-Calleros | F. J. Álvarez Rodríguez | J. Muñoz-Arteaga
[18] Pereira, A. (2010). Criteria for elaborating subtitles for deaf Proceedings of the 15th International Academic MindTrek
and hard of hearing adults in Spain: Description of a case Conference: Envisioning Future Media Environments (pp.
study. Listening to Subtitles. Subtitles for the Deaf and 9-15). ACM.
Hard of Hearing, 87-102. [26] Taylor, L. N. (2002). Video games: Perspective, point-of-
[19] Svirsky, M. A., et al. (2000). Language development in view, and immersion (Doctoral dissertation, University of
profoundly deaf children with cochlear implants. Florida).
Psychological science, 11(2), 153-158. [27] Salen, K., & Zimmerman, E. (2004). Rules of play: Game
[20] National Institute on Deafness and Other Communication design fundamentals. MIT press.
Disorders, U.S. Department of Health & Human Services. [28] Sánchez, J. L. G., Zea, N. P., & Gutiérrez, F. L. (2009).
Quick Statistics. On the Internet From usability to playability: Introduction to player-centred
http://www.nidcd.nih.gov/health/statistics/pages/quick.aspx video game development process. In Human Centered
(visited February, 2015). Design (pp. 65-74). Springer Berlin Heidelberg.
[21] Zyda, M. (2005). From visual simulation to virtual reality [29] Longstreet, C. S., & Cooper, K. (2012, July). A meta-
to games. Computer, 38(9), 25-32. model for developing simulation games in higher education
[22] Chin, S. B., Bergeson, T. R., & Phan, J. (2012). Speech and professional development training. In Computer Games
intelligibility and prosody production in children with (CGAMES), 2012 17th International Conference on (pp.
cochlear implants. Journal of communication disorders, 39-44). IEEE.
45(5), 355-366. [30] Ittleman, M. A. (2012). The Teaching of Talking: Learn to
[23] Lenden, J. M., & Flipsen, P. (2007). Prosody and voice Do Expert Speech Therapy at Home with Children and
characteristics of children with cochlear implants. Journal Adults. Morgan James Publishing.
of Communication Disorders, 40(1), 66-81. [31] Calvary, G., et al. (2003). A unifying reference framework
[24] Aldrich, C. (2009). The complete guide to simulations and for multi-target user interfaces. Interacting with Computers,
serious games: how the most valuable content will be 15(3), 289-308.
created in the age beyond Gutenberg to Google. John Wiley [32] Florins, M., et al. (2006). Splitting rules for graceful
& Sons. degradation of user interfaces. In Proceedings of the
[25] Deterding, S., et al. (2011, September). From game design working conference on Advanced visual interfaces (pp. 59-
elements to gamefulness: defining gamification. In 66). ACM.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 120 |
 Towards the Definition of a Framework for the Manage-ment of Interactive Collaborative Learning Applications for Preschoolers
L. Rodríguez-Vizzuett | J. L. Pérez-Medina| J. Muñoz-Arteaga | J. Guerrero-García | F. J. Álvarez-Rodríguez

Towards the Definition of a Framework for the

Management of Interactive Collaborative Learning
Applications for Preschoolers
Liliana Rodríguez-Vizzuett1, Jaime Muñoz-Arteaga1, Francisco J. Álvarez-
Jorge Luis Pérez-Medina2, Josefina Guerrero-García3, Rodríguez1
1 2 3
Universidad Autónoma de Université catholique de Louvain Benemérita Universidad Autónoma
Aguascalientes Place des Doyens 1 de Puebla
Av. Universidad 940, Ags., Mexico Louvain-la-Neuve, Belgium Av. 14 Sur y San Claudio, Puebla,
[email protected], {fjalvar, jorge.perezmedina Mexico
jmunozar}@correo.uaa.mx @uclouvain.be [email protected]
ABSTRACT acquiring knowledge. In most countries, preschool education
The adoption of technologies in education has been changing the begins at the age of three years, but there are cases in which
traditional teaching system considerably. New educational children between 2 and 6 years are enrolled into it. However,
paradigms make use of technological tools to enhance the our proposal focuses on children aged between 4 and 6 years.
knowledge given in a classroom. In the literature review there Preschool educational institutions of several countries
aren't reports of Frameworks that are based on models and established official documents which are provided to educators,
methods for developing Interactive Collaborative Learning with skills that must be met by the children, so that at the end of
Applications for Preschoolers (ICLAP). This paper proposes a this stage students have begun a process of formal contact with
meta-model for a Framework supporting the development of the language writing, through the exploration of texts with
ICLAP taking into account the technology, the educational different characteristics (e.g., books, newspapers and
aspect and collaborative techniques that were observed during instructions) [2-6].
the attendance to sessions in schools. The ICLAP meta-model Augmented Reality (AR) can be used in many application
includes useful concepts for the use of Augmented Reality since domains. In recent years, a number of prototypes and
we found that it emerges as a technology of interest in diverse applications have been created in areas such as Computer
areas and especially in education. Our contribution could be graphics, Human Computer Interaction (HCI), vision, and
used for teachers, designers and developers as a conceptual collaborative work including the following sectors: architecture,
guide. In addition, a study case is presented to illustrate the main design, engineering and production planning, games, education
concepts included in the meta-model. and learning [7-9]. AR can be designed not only to support
Categories and Subject Descriptors learning related disciplinary contents but also to provide other
skills such as critical thinking, collaboration and information
D.3.3 [Programming Languages]: Language Constructs and
sharing [9]. Applications that make use of AR achieve to
Features – frameworks; H.5.2 [User Interfaces]: Prototyping,
generate the basic principles of Collaborative Learning: (1)
User- centered design; K.3.1 [Computer Uses in Education]:
Positive Interdependence; (2) Promotion of Interaction; (3)
Collaborative Learning
Individual Accountability; (4) Interpersonal and Small Group
General Terms Skills; and (5) Group Processing. Positive Interdependence
Design, Human Factors, Collaboration. ensures that team members perceive that they are linked with
each other so that one cannot succeed unless everyone succeeds.
Keywords In Promotion of Interaction the students promote each other's
Collaborative Learning, Augmented Reality, Framework, success by assisting, supporting, encouraging, and praising each
Collaborative Application. other's efforts to learn. The Individual Accountability means that
each individual student's performance is assessed and the results
1. INTRODUCTION are given back to the group and the individual. The Interpersonal
The inclusion of technology in education should be considered and Small Groups Skills suggests that the student is required to
only a tool to support since it is not to coming to replace the work as part of a team (teamwork), and finally for the Group
professor, but to help the student to have many auditory and Processing the team members discuss how well they are
visual elements to enrich the process at teaching–learning. An achieving their goals and maintaining effective working
important factor to be considered is the help that relationships.
Communication and Information Technologies (TIC) give Our proposal, for the specific case study that is here presented,
because it provides the students with many kinds of tools to be makes use of Augmented Reality. The role of the AR is to
used within in the classroom and for enhancing the way in facilitate interaction among preschoolers using QR codes. These
which knowledge is acquired [1]. In traditional education, the codes are the markers that will be recognized by the application.
teacher is responsible for student learning by defining learning
objectives or content area learning tasks, and designing and There are many interactive applications to support the learning
making an assessment of what has been learned by the students. of reading and writing, however, these do not report to use a
In the traditional model of learning, the concept of collaboration methodology of development involving educational content,
is included to allow individuals to enrich the experience of making the applications not only to entertain children but also to

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 121 |
 Towards the Definition of a Framework for the Manage-ment of Interactive Collaborative Learning Applications for Preschoolers
L. Rodríguez-Vizzuett | J. L. Pérez-Medina| J. Muñoz-Arteaga | J. Guerrero-García | F. J. Álvarez-Rodríguez
help them to obtain knowledge [10]. Moreover, the literature communication [16]. Collaborative learning has as goals to
survey does not report models for Interactive Applications promote the interaction among students, valuing the individual
supporting Collaborative Learning in Children, despite the fact contribution of each student that assumes tasks and
that there are collaborative applications, those do not report to responsibilities, development of group and personal skills such
be developed under a user-centered paradigm, besides, even as listening, participating, coordinating activities and leading.
though they count with collaborative characteristics, the absence For last the self-assessment is promoted since the students need
of a model describing their design makes their components to continuously know from their performance and their
reutilization limited. In order to develop literacy skills in equipment. To carry out collaborative learning, four important
children and considering the benefits that TICs offer, in this components are needed: students, the professor or facilitator,
paper a set of rules and constructs for the ellaboration of technological resources, and the environment for the
semantic models for ICLAPs is proposed, meaning by this, a collaboration. In collaborative learning there are different roles
meta-model for development ICLAP involving several actors to carry out a task, however, each member of team is responsible
(i.e., student, professor, software engineer, designer and for a task and when all the team members finish their activities,
programmer) within each development phase, this meta-model they present to the rest of the team what they did, so it is
pursues the purpose of including a collaborative component that important that there is good communication between all team
allows its reuse in any type of ICLAP, and is based on the members to do a better work. Collaborative activities enhance
available literature, the observation of the techniques used by learning by allowing individuals to exercise, verify, solidify, and
preschool professors, as well as technological and educational improve their mental models through discussions and
aspects. information sharing during the problem-solving process [17].
The rest of this paper is structured as follows: Section 2 Collaborative learning helps to generate a common language, by
introduces our approach by making an analysis of the literature establishing rules for the group functioning and in order to
related to the concepts of framework, collaborative learning, reduce the fear of criticism and to promote feedback. By taking
learning technologies, augmented reality, and a subsection with advantage of the knowledge and experience of the team,
related work. Section 3 introduces an ICLAP Meta-model. The according to their area of expertise and different viewpoints, the
meta-model is later discussed using a case study in section 4. quality of the decisions and of the obtained products is
Finally, section 5 presents our conclusions and future work. improved.

2. REVIEW 2.3 Learning Technologies

Technology is a tool that can change the nature of learning [19].
2.1 Framework The multimedia resources have the potential to create high
A framework provides an abstraction of a solution to different quality learning environments because they provide to the
kinds of problems that have some similarities. It also provides students a better way to learn. In the literature, some authors say
the steps or phases that must be followed to implement a that for any design to be successful, it must be based on the
solution without getting into the details of what activities are needs and interests of users as this ensures that it will fulfill the
done in each phase [11]. The using of a framework for functions for which it was elaborated [20]. In a classroom
developing ICLAP aims to generate a conceptual and supported by technology, students don't "learn" technology
technological support structure, in which other software because it only provides the tools to be used for authentic
development can make reuse of required components. For this learning, so the technology is a mean, not an end. It is well
purpose, firstly the construction of a meta-model that possesses known how difficult it is to engage children in learning,
basic elements such as collaborative learning, learning contents however, it has been reported that children enjoy the process
and the technology is required. when their tutor uses a touch screen and other devices in order to
make classes more interactive and interesting [21]. This leads to
2.2 Collaborative Learning the assumption that by involving technology, it’s easier to attract
A skill is defined as the ability to coordinate knowledge and the kid’s attention and to make the educational process more
know-how to respond to a given situation or to solve problems enjoyable for both, the tutor and the pupils. As classroom
in everyday life [12]. The skills in preschool that children should computer technology is being used for different types of
develop are: (1) the ability to assume different roles, both in the communication i.e. for presentation, for class interaction, and for
game and in various activities, (2) rely on their peers to achieve collaboration, students are required to assume different roles and
goals, (3) resolve conflicts through dialogue, and (4) respect and must be willing to collaborate with others. Another advantage of
recognize the rules of coexistence in the classroom, school and using technology in the classroom is its flexibility and
beyond [2][13][14]. Within the strategies that have to favor the adaptability to different type of learning [22]. A collaborative
learning, collaborative learning is the most important activity application is a software application that allows multiple users
that takes place in small groups mainly in a classroom in order interaction to achieve a common goal, receives input from
to share different viewpoints to form a more complete multiple users and displays outputs to multiple users. Also
knowledge [15]. The collaborative learning supported by couple those users, ie, allows the entrance of one of them to
technology consists on that two or more persons share the influence the output shown to the other [23].
responsibility of building knowledge, based on the interaction
and decision making using the technological resources as 2.4 Augmented Reality
mediators for this process. The collaborative learning process Augmented Reality (AR) is a paradigm of interaction that stems
consists of several tasks that must be performed by a facilitator from the desire to merge the processing capabilities of a
or teacher and students, also, has as key elements, the computer in the user's real environment. The goal of this
cooperation between group members, active learning, positive technology is overcoming the boundary between the real world,
interdependence, responsibility, coordination and including the user, his/her environment and the computing

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 122 |
 Towards the Definition of a Framework for the Manage-ment of Interactive Collaborative Learning Applications for Preschoolers
L. Rodríguez-Vizzuett | J. L. Pérez-Medina| J. Muñoz-Arteaga | J. Guerrero-García | F. J. Álvarez-Rodríguez
world. A key feature of AR systems is that real-world elements model (apprentice and facilitator), the content model which
are involved in the interaction. The existence of real and virtual mentions that the contents are characterized as learning objects,
worlds implies that the purpose of the task may match to one of the communication model where the process is defined by which
the two. The term Mixed Reality corresponds to the interval is conducted the interaction between users and fosters
between the real and the virtual. Likewise, Dubois [7] proposes collaboration and finally the assessment model which takes into
a taxonomy based on the objective of where the crucial point in account the evaluation of service quality. The points that are
their approach was identifying the task supported by the system evaluated are: number of subscriptions that are the recognized
to be analyzed and designed. The taxonomy considers two users by the application, the number of connections of each
continuous: (1) Augmented Reality and (2) Virtual Reality subscription, the statistics of users who sent a message, the
(VR). In AR systems the user’s interaction with the physical response time of the facilitator to respond a message sent by an
world is enriched by the aggregation of data and services offered apprentice user, the evaluation of the contributions made on
by the numerical world (computer), in the natural environment collaboration and lastly the relevance of the message to both the
of the user. Comercial applications such as Dokéo+ learner and facilitator.
(http://www.dokeo-realiteaugmentee.com/), and L’Aphablet
(http://www.clementoni.com/) make use of AR and help to 3. AN INTERACTIVE COLLABORATIVE
reinforce learning in children, however, they do not report being LEARNING APPLICATIONS FOR
designed to support collaborative learning. Dokéo+ allows
children to learn different topics (science, animals, history,
PRESCHOOLERS META-MODEL
mathematics), but does not allow the interaction of more than This section shows the meta-model for ICLAP. The purpose of
this meta-model is to present a conceptual guide that allows the
one user at a time. L'Alphabet enables learning letters as well as
objects which names contain them. This tool uses audio and development of ICLAP considering collaborative techniques
animation resources, and has a bilingual mode, however, it that were observed while attending sessions in schools in
doesn't report either to enable collaboration among children. It is Mexico. The ICLAP meta-model follows a User-Centered
important to point out that both applications use AR markers. approach. The authors have had the opportunity of proposing
Figure 1 allows distinguishing the two streams corresponding to activities for the children, carry them out and thus obtain the
the both extremes of a continuum between real and virtual definition of collaborative strategies that are used by teachers to
accomplish the acquisition of collaboration skills.
worlds.
3.1 General Description
The main concepts of the ICLAP meta-model are presented on
Figure 2 that is divided into six packages. The first one refers
the Evaluation package of the whole application and its usage.
The second package concerns the Technology and is formed by
Interface and GameElement. The third is for description of
Collaborative Learning, conformed by the class
InteractiveCollaborativeLearning and by the one referent to
InteractiveCollaborativeApplication. The fourth package
concerning the LearningContent and formed by
LearningContent and LearningStyle, the fifth package
concerning the Context model formed by the User, the Platform
and the Environment. Finally, the Interaction Resources package
consisting of the elements that will be used to develop an
application.

Figure 1. Classification of Mixed Systems [7][24]

3.2 The Evaluation Package
An important part to be considered, is the evaluation of
2.5 Related Work interactive applications concerning usability, skills and
In [25] the theory of the activity is defined as a strategy for the collaborative learning using instruments and also using quality
development of collaborative learning systems in which takes metrics [2] [29-33]. The evaluation is the measurement of
into account the activity, goal, result, community, rules, works’ knowledge, skills and performance of students when using a
division and tools. Likewise, the modeling methodology collaborative interactive application. It uses instrument that are
AMENITIES [26] define four views for cooperative systems. documents which purpose is to justify that interactive
They’re: group view (groups, roles and actors), cognitive view collaborative applications working for learning and
(tasks, sub-activities and actions), interaction view (protocols, collaboration. An Instrument class is composed of an
devices, media) and information view (resources, documents, IntegratedMeasure element that compute the Measure of each
messages). Penichet [27] highlights the importance of user Metric. Each instrument has one or several sections that relates
interaction with collaborative systems, and makes a proposal to to the topic to be evaluated (usability, learning skills,
collaborative environments from obtaining requirements to final collaborative skills, perception of technology, collaborative
system maintenance, through the phases of analysis, design, work, and quality assessment). In turn, the section has a name
implementation, and evaluation. Finally [28] makes a literature and one order. Moreover the dimension refers to each of the
review which mentions that it is important to consider the user characteristics to be evaluated in each instrument (e.g., Easy to
interaction with the system to be developed, with special remember, language and communication, cordination). The
emphasis on user interfaces, without leaving aside the user dimension is composed of a description and order.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 123 |
 Towards the Definition of a Framework for the Manage-ment of Interactive Collaborative Learning Applications for Preschoolers
L. Rodríguez-Vizzuett | J. L. Pérez-Medina| J. Muñoz-Arteaga | J. Guerrero-García | F. J. Álvarez-Rodríguez

Figure 2. An Interactive Collaborative Learning Applications Meta-Model

Every dimension must carry at least one item; the attributes of an and the errorSolution; GameWorld, considering the scene in
item are the question to be answered, and the order representing which the game takes place, the view (e.g. first person, third
the value that was given to it. A Metric is a mean to understand, person, isometric and cabin view), and the interactiveElements,
monitoring, controlling, predicting and testing the software which are those elements that the User may use to interact with
development and maintenance projects [34]. Metrics has a set of the system (e.g., images, audio and animations); and
attributes: name, description, and also a lowValue and upperValue InteractiveSystem, to describe the modality of interaction. Table 1
that are the measured values for each metric. Metrics produce shows an OCL Rule defined to ensure that instances of
measures that throw values that are the results for each InteractiveCollaborativeApplication class have three different
instrument. elements of type GameElements at the same time
3.3 The Technology Package
For the part of the model concerning the Interface, the model
provided by [35] is taken. Its definition is achieved by using the
theoretical framework CAMELEON [36] that comprises four
levels of abstraction: (1) Tasks & Concepts: description of the
users’ activities. (2) Abstract User Interfaces (AUI): Definition of
abstract containers and individual interaction components. (3)
Concrete User Interfaces (CUI): Definition of the interaction
modality and usage of concrete interaction objects. (4) Final User
Interfaces (FUI): Operational elements, this is, the implemented
UI for a determined platform. Elements for the design of
videogames according to [37] are considered. Those elements
being: ErrorManagement, for the presentation of errors to the
User, including an icon, a message, a trigger called detectedError,

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 124 |
 Towards the Definition of a Framework for the Manage-ment of Interactive Collaborative Learning Applications for Preschoolers
L. Rodríguez-Vizzuett | J. L. Pérez-Medina| J. Muñoz-Arteaga | J. Guerrero-García | F. J. Álvarez-Rodríguez
Table 1. OCL rule for InteractiveCollaborationApplication the acquisition of theoretical knowledge, but comprehension, and
instances discovery of the environment.
Table 2. The Structure of Learning Content
Name
Description
Objective
3.4 The Collaborative Learning Package Learn to know Learn to be Learn to do
Evaluation of learn to Evaluation of learn to Evaluation of learn
For the part of the model concerning the CollaborativeLearning,
know be to do
an InteractiveCollaborativeActivity [38] is an activity that Learning Type
involves the Feedback between two entities (viz. tutor-pupil and Learning Style
pupil-pupil) [39], and a series of attributes: rules to follow during The LearningStyle describes the way in which Users learn as well
the activity, collaborativeSkills, rewards as incentives for the as the actions to be performed in order to promote a more
User (e.g., badges and levels), executionTime in seconds, effective learning(e.g., Felder and Silverman models, Neuro
monitoring performed by the Tutor, teamSize that is the number of linguistic programming, Kolb model, and Multiple Intelligences
members in each group between 2 and 6 [40], and the model). However, due to the generality of the model proposed,
numberOfTeams participating on the collaborative activity. The more styles can be adapted to this class. The attributes that are
collaborativeSkills are those to be achieved during the considered for the definition of the LearningStyle are the
collaborative learning process, and are determined by a set of voicePitch, the posture of the User while conducting activities,
strategies to allow the development of mixed skills (concerning the preferred perceptionChannel for the User to get knowledge
learning and personal/social development), in which every team (e.g. images, audio and physical activities), and voiceSpeed.
member is responsible for both, his/her and the other members’
learning. The InteractiveCollaborativeActivity is related to a Stakeholders may invoke Table 2 to elaborate the Learning
CollaborativeComponent. The CollaborativeComponent has as Contents. The table is formed based on the elements of the
attributes a name, a description, an objective and a package. The elements are: Name, Description of the learning
collaborativePattern. Papageorgiu [41] defined a collaboration content, Objective, Learnings, EvaluationOfLearning,
pattern as a means to capture best practices about recurring LearningType and finally the LearningStyle.
collaborative problems and solutions. As part of this component, 3.6 The Context Package
there are Roles defined for each team member. For the Roles
The Context [45] represents the conditions in which interaction is
definition, it is necessary to specify a roleObjective and roleTasks
carried out and is formed by the User, this class contains as
that are the purpose of the role and the actions to be performed by
attributes an id, that is a key for the identification of each User;
each one. For their relevance to this work, the Roles that are
the preferred language for communication; a stereotypeName, the
considered are: Resource Manager, Time Keeper, and Leader. The
role assigned for an activity (viz. Tutor or pupil); the
Leader is in charge of asking the teacher for assistance whenever
systemExperience, representing how experienced is the User on
it is needed. This role also must be able to organize other team
the usage of interactive applications; the taskExperience,
members in order to properly carry out the activity. The Resource
containing the explanation of how familiar the activity is o the
Manager is supposed to ensure that the given application is used
User; the taskMotivation, motivation or stimuli that cause interest
in a proper manner, and finally the Time Keeper must notify each
on the User for completing the task in a correct manner; the
member of the group about the available time for successfully
deviceExperience, the experience of the User on the usage of
performing the assignment [42].
electronic devices such as tablets, smartphones, computers and
3.5 The Learning Content Package consoles; and the age of the User.
The LearningContent is the information referent to a subject of
study and is formed by a LearningObject [43] that is a basic
reusable structure that contains detailed information about certain
topics and allows the relation to other similar objects. A
LearningObject has as attributes an id, a name, a date of creation,
a description, and an objective. And it’s contained into a category
(e.g., Reading and Writing), subset of subjects that are related to
an area (e.g., Language). The LearningObject contains in turn,
Content, which is the information to be presented and has as
attributes an id, a name, and an associated LearningType (e.g.
casesBased, problemBased, projectBased and gameBased).
Content may be presented in several formats such as videos, texts,
images, audios and animations. Related to the LearningObject,
there are the Author class that represents its creator, the
EvaluationOfLearning and the Learnings class that conform it.
Learnings [44] are that knowledge acquired during the education
process and are specified as learn to be (Being), referring to the
skills developed to be practiced on the daily life; learn to do Figure 3. The Interaction Resource package
(Doing), that is the integration of knowledge to be useful on the
professional life; and learn to know (Knowing), meaning not only

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 125 |
 Towards the Definition of a Framework for the Manage-ment of Interactive Collaborative Learning Applications for Preschoolers
L. Rodríguez-Vizzuett | J. L. Pérez-Medina| J. Muñoz-Arteaga | J. Guerrero-García | F. J. Álvarez-Rodríguez
The User can be either a Tutor (professor, parent or supervisor) or
a Pupil (student or trainee). The Platform entity represents a series
of elements that allows the execution of interactive applications.
The attributes are id, name and description and is composed by
softwarePlatform and hardwarePlatform. The only attribute for
the softwarePlatform is the operatingSystem, while for the
description of the hardwarePlatform, it is necessary to specify a
deviceType (e.g. tablet, smartphone and computer), the screenSize
in inches, the availability of a touch screen on it (isTouchScreen)
and the availability of an internetConnection. The Environment
class describes properties of interest about the place in which the
users perform a collaborative activity. Their attributes are an id, a
name, the statement of whether the place is noisy or not (isNoisy),
and the statement of whether the place is stressing or not
(isStressing), this means, if there are or not characteristics that Figure 5. The Task tree model of the case study application
propitiate a change on the Users’ behavior during the activity. The The task model shows the activities that must be carried out in the
definition and consideration of the last two attributes of this class application. The application is aimed at learning the letters and
is important since they may help to determine characteristics that objects that children see in their daily lives and must do so
directly impact on the learning process. collaboratively. The first task that user need to do is select the
language in which they will play, for this, the users have French
3.7 The Interaction Resources Package and Spanish as options. Next, the user must select the number of
The interaction resources package presented in Figure 3 considers players, the application allows to play for 2-5 players, followed by
all interaction elements. Its elements are important within any
this, the user must choose the difficulty level, these levels are:
application that is implemented because they are the basis with
easy, medium and hard. After this selection the game begins. To
which the user interacts. These elements may be images, audio,
play, each child will have a pack of cards with QR codes and must
numbers, words or letters regardless of the amount used of these. be sequentially showing to webcam the letters to complete the
AR applications should require markers associated to these name of the object that appears on the screen. The game consists
resources. For example, QR codes as those shown in Figure 4. on presenting pictures (animals, household items, food, human
Moreover in Figure 4, we show the interaction resources that can body, etc.) and boxes where users must place the letters of their
be used in any type of interactive application.
names. For the first level of difficulty the application will show
the boxes with the letters in watermark to point out where the user
should place each one. For the second level, the system only will
show the first letter of the word and the users should be able to
complete it, and finally at the third level there will be only boxes,
without any aid, in which users should be able to complete the
word. Figure 6 presents screenshots for each level of difficulty.
When the letter shown by the user is correct the application will
fill the corresponding box with it and also, the application will
present an animation with balloons, a trophy and audio. When a
user makes a mistake, the application shows an image to the user
asking him to try again.

Figure 4. The interaction resources

4. CASE STUDY A COLLABORATIVE

LEARNING ALPHABET APPLICATION
In this section we consider a demonstrative case study. The aim is
to carry out benefits of the proposed meta-model. In this case
study the following packages are being instanced: Learning
Content, Collaborative Learning and Technology since they are
most important for the design phase. In order to take decisions for
the development phase, the Context package should be considered
and on the evaluation stage, the Evaluation package has to be
taken into account. The case study consists of a set of sequential
tasks accomplished for children in order to indicate the letters of
the word that is presented by the application. The interaction
scenario is defined in a task tree model depicted by Figure 5.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 126 |
 Towards the Definition of a Framework for the Manage-ment of Interactive Collaborative Learning Applications for Preschoolers
L. Rodríguez-Vizzuett | J. L. Pérez-Medina| J. Muñoz-Arteaga | J. Guerrero-García | F. J. Álvarez-Rodríguez
correspond to each one. To identify each child within the system,
distinct markers must be used.

Figure 8. Cards with QR codes given to each child

The application displays the word to be completed and among all
children should complete the word based on each turn generated
by the application. When the letters are insufficient for every child
to participate, turn assignation in the next word should start with
those who did not. Table 3 shows the design of Learning Content
Figure 6. The levels of difficulty on the developed application required by the interactive application. The table is constructed by
Collaborative skills to be achieved in children using this the professor based on skills for learning and collaboration that
interactive application are: (1) Show facility to interacting with he/she wants to develop in children by using the application. Once
others; (2) Accept to perform different roles and assume their this table is constructed, with help of a designer, prototypes of
responsibility with the team; (3) Respect the turns when carrying interfaces are created and the type of elements to be used for
out an activity; and (4) Respect their peers. To achieve these displaying learning content is defined. The most important
collaborative skills, it is required to have a collaborative strategy. elements that the developer is supposed to know in order to
One is the use of turns. A turn consists of assigning a task to a implement the interactive application are: a name, a description,
particular child. The collaborative component implemented in this an objective, the learnings and their associated evaluations, as
application will be the Turn Assignation pattern as shown in well as a learning type and a learning style.
Figure 7. Table 3. The Learning Content for “Learning the letters”
Learning the letters
Use the knowledge that they have of their names and other words to write
something they want to express
Recognizes characteristics of the writing system use resources (marks,
graphs, letters) to express in writing their ideas
The letters and its Show some images The words in
representation are shown with daily objects that watermark are
to the student. Also, contain letters shown to the
their utility in daily life students and they
is explained are required to put
letters in place
Students should Questions are asked The words without
Figure 7. Collaboration Pattern in Spanish recognize letters and about the letters in watermark are
their order in the objects’ names. shown to
The objective of this pattern is to allow each team member to alphabet Students are asked to the students and
fulfill a role and carry out assigned activities. The pattern begins complete those words they are required
with an activity to do, then the number of participants who will to check if they to put letters in
perform the activity should be select, later it is possible make a correctly identify place
turn assignation in a sequential or random manner as requested by them
the teacher. To ensure the participation of all team members in our Based on Games
interactive application turn assignation must be performed Visual, Auditive and Kinesthetic
sequentially. We must also ensure that all children participate.
This will be done as follows: Each child will be given a pack of 5. CONCLUSIONS AND FUTURE WORK
letters with QR codes as those shown in Figure 8; each package This paper presents a meta-model for ICLAP to support the
will have different colors to allow children to identify those that education process. It will serve as base to our framework and
permits a conceptual guide for teachers, designers and developers
for the implementation of this type of applications. The proposed
XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models
| 127 |
 Towards the Definition of a Framework for the Manage-ment of Interactive Collaborative Learning Applications for Preschoolers
L. Rodríguez-Vizzuett | J. L. Pérez-Medina| J. Muñoz-Arteaga | J. Guerrero-García | F. J. Álvarez-Rodríguez
model considers technological and educational aspects, as well as [9] Klopfer, E., Perry, J., Squire, K., & Jan, M. F. (2005, May).
collaborative techniques that were observed during the attendence Collaborative learning through augmented reality role
to sessions in schools in Mexico through the performance of playing. In Proceedings of the 2005 conference on Computer
activities that teachers carried out with children using support for collaborative learning: learning 2005: the next 10
collaborative strategies for teaching letters and their use on daily years! (pp. 311-315). International Society of the Learning
life. One of the advantages that the meta-model provides is the Sciences.
capability of its packages to be reused in other type of [10] Proyecto H@z TIC. Guía práctica de aprendizaje digital de
collaborative applications, another advantage is that it can be used lectoescritura mediante tablet para alumnos con síndrome de
in other areas where there is collaborative work. The Technology Down, Ministerio de Educación, Cultura y Deportes, 2012.
Package allows to add other game design elements according to [11] Johnson, R., Foote, B. « Designing Reusable Classes”.
the requirements of the application to be developed. The learning Journal of Object-Oriented Programming, vol. 1, no. 2, pp
content package allows adaptation to learning styles and types as 22-35 (1988)
required. The proposed meta-model presents concepts that can be [12] Nova, A. P. (2011). Formación docente: hacia una definición
easily understood by the different actors involved the del concepto de competencia profesional docente. Revista
development process of interactive collaborative learning electrónica interuniversitaria de formación del profesorado,
applications i.e., professor, software engineer, designer, 14(1), 67-80.
programmer and tester. Actually, the presented case study is in [13] Delors, J. (2013). Los cuatro pilares de la educación. Galileo,
development phase and once a first prototype is produced, we will (23).
be able to easily evaluate the interest and the feasibility of our [14] Brophy, J. (1998). Un ambiente propicio para el aprendizaje
approach by conducing user experiments supported by the en el aula. “Oportunidades para aprender”,“Construcción de
instruments that were mentioned as part of the evaluation un soporte para lograr el trabajo comprometido de los
package. Besides, in parallel a methodology is being designed in alumnos”,“Aprendizaje colaborativo” y “Expectativas de
order to guide the development process, by establishing different logros”, en La enseñanza, México, UNESCO/SEP (Serie
actors and the tasks that each one must fulfill for producing Cuadernos. Biblioteca para la actualización del maestro).
collaborative interactive applications, taking into account all [15] Ordóñez, C. A. C., & Mendoza, J. (2006). Cómo aprovechar
concepts of ICLAP meta-model. As well, we will perform el "aprendizaje colaborativo" en el aula. Educación y
different case studies with children. educadores, 9(2), 61-76.
[16] Panitz, T. (2008). 44 Benefits of Reflective Learning.
6. ACKNOWLEDGMENTS Retrieved May 30, 2008, from Global Development
We acknowledge the support that was given by CONACYT, the Research Centre: http://www.gdrc.org/kmgmt /c-
Université catholique de Louvain and the Autonomous University learn/44.html
of Aguascalientes for the development of this work through a [17] Alavi, M. (1994). Computer-mediated collaborative learning:
research stay program. We also acknowledge Professor Jean An empirical evaluation. MIS quarterly, 159-174.
Vanderdonckt for his valuable support for the development of this [18] Glassett, K., & Schrum, L. (2009). Teacher beliefs and
project. student achievement in technology-rich classroom
environments. International Journal of Technology in
7. REFERENCES Teaching and Learning, 5(2), 138-153.
[1] Graells, P. M. (2000). Impacto de las TIC en educación: [19] Spiliotopoulos, T. (Ed.). (2010). Integrating Usability
funciones y limitaciones. Departamento de Pedagogía Engineering for Designing the Web Experience:
Aplicada, Facultad de Educación, UAB. Methodologies and Principles: Methodologies and Principles.
[2] Secretaría de Educación Pública de México SEP. Enfoque IGI Global.
Centrado en Competencias. Available online at http://www. [20] Kosakowski, J. (1998). The Benefits of Information
dgespe.sep.gob.mx/reforma_curricular/planes/lepri/plan_de_ Technology. ERIC Digest.
estudios/enfoque_centrado_competencias. Consulted on [21] Schmid, E. C. (2008). Potential pedagogical benefits and
February 2015. drawbacks of multimedia use in the English language
[3] Szinger, V. Pre-school literacy in the light of European classroom equipped with interactive whiteboard technology.
tradition. Computers & Education, 51(4), 1553-1568.
[4] Sharp, C. (2002, November). School starting age: European [22] Dewan, P., Choudhary, R., & Shen, H. (1994). An editing
policy and recent research. In LGA Seminar 'When Should based characterization of the design space of collaborative
Our Children Start School’. applications. Journal of Organizational Computing and
[5] Lisle, J. (1998). Proyecto principal de educación en América Electronic Commerce, 4(3), 219-239
Latina y el Caribe. El informe Delors dentro del contexto [23] Milgram P., Kishino, F. A Taxonomy of Mixed Reality
Americano. Visual Displays, Transactions on Information Systems, pages
[6] Rose, P., et al. (2014). Informe de Seguimiento de la EPT en 1321-1329, 1994.
el mundo. Enseñanza y Aprendizaje: Lograr la calidad para [24] Barros, B., Vélez, J., & Verdejo, F. (2004). Aplicaciones de
todos. Informe 2014. la Teoría de la Actividad en el desarrollo de Sistemas
[7] Dubois, E.: Chirurgie Augmentée: un Cas de Réalité Colaborativos de Enseñanza y Aprendizaje. Experiencias y
Augmentée; Conception et Réalisation Centrées sur Resultados. Inteligencia artificial, 8(24), 67-76.
l'Utilisateur. PhD Thesis University of Grenoble I, France [25] Gea, M., Gutiérrez, F., Garrido, J., & Cañas, J. (2002).
pages 275, 2001. AMENITIES: Metodología de Modelado de Sistemas
[8] Hannes Kaufmann, Dieter Schmalstieg. Mathematics and Cooperativos. In Workshop de Investigación sobre nuevos
geometry education with collaborative augmented reality.
Computers and Graphics, 2003.

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 128 |
 Towards the Definition of a Framework for the Manage-ment of Interactive Collaborative Learning Applications for Preschoolers
L. Rodríguez-Vizzuett | J. L. Pérez-Medina| J. Muñoz-Arteaga | J. Guerrero-García | F. J. Álvarez-Rodríguez
paradigmas de interacción en entornos colaborativos [35] Calvary, G., Coutaz, J., et al., A Unifying Reference
aplicados a la gestión y difusión del Patrimonio cultural. Framework for Multi-Target User Interfaces, Interacting with
[26] Penichet, V. M., Lozano, M. D., & Gallud, J. A. (2005). Computers, 15(3), June 2003, pp. 289–308.
Hacia una propuesta metodológica para entornos [36] Aldrich, C., The complete guide to simulations & serious
colaborativos. III Taller de Sistemas Hipermedia games, Pfeiffer, 2009
Colaborativos y Adaptativos. [37] Álvarez, I., & Guasch, T. (2006). Diseño de estrategias
[27] Aular, Y. J. M., & Pereira, R. T. (2007). Metodologías para interactivas para la construcción de conocimiento profesional
el desarrollo de ambientes de aprendizaje en entornos en entornos virtuales de enseñanza y aprendizaje. Revista de
colaborativos: Una reflexión teórica. Multiciencias, 7(1). Educación a Distancia, (14).
[28] Jung, H. W., Kim, S. G., & Chung, C. S. (2004). Measuring [38] Calzadilla, M. E. (2002). Aprendizaje colaborativo y
software product quality: A survey of ISO/IEC 9126. IEEE tecnologías de la información y la comunicación. Revista
software, 21(5), 88-92. Iberoamericana de Educación, 1(10).
[29] Nielsen, J. (1994). Usability engineering. Elsevier. [39] Adams, D., & Hamm, M., Cooperative Learning, Critical
[30] Oviedo, A., Instrumento de Evaluación del Grado de Thinking and Collaboration Across The Curriculum. 2nd ed.
Apropiación Asistido por TIC de Competencias en Springfield III: C.C. Thomas, 1996
Preescolar. 2014 [40] Papageorgiou, N., et al. (2009). A collaboration pattern
[31] Quezada, P., Método de evaluación para TIC's aplicadas en model for virtual organisations. In Leveraging Knowledge
la educación preescolar, caso: Sistema del Estado de for Innovation in Collaborative Networks (pp. 61-68).
Aguascalientes. Tecnologías y Aprendizaje: Avances en Springer Berlin Heidelberg.
Iberoamérica Vol2. 2013 [41] Olguín, S. C., et al. (2006). Ciencias. Antología. Primer
[32] García, M., Santana, P. (2015). Introducción a la usabilidad Taller de Actualización sobre los Programas de Estudio
de los videojuegos. La Interacción Humano-Computadora en 2006. Reforma de la Educación Secundaria. Coordinador
México. Muñoz, J., González, J., Sánchez, A., Eds. Pearson, editorial. SEP
México. [42] Rodríguez-Vizzuett, L., et al. Metodología para el Desarrollo
[33] Basili, V. R., Briand, L. C., & Melo, W. L. (1996). A de Sistemas Interactivos para Gestión y Visualización de
validation of object-oriented design metrics as quality Objetos de Aprendizaje.
indicators. Software Engineering, IEEE Transactions on, [43] Zhou, N. Z. (1996). Four ‘Pillars of Learning’ for the
22(10). Reorientation and Reorganization of Curriculum: Reflections
[34] Stanciulescu, A. (2008). A methodology for developing and Discussions.
multimodal user interfaces of information systems (Vol. [44] Limbourg, Q. (2004). Multi-path development of user
556). Presses univ. de Louvain. interfaces (Doctoral dissertation, UCL).

XVI Congreso Internacional de Interacción Persona-Ordenador | Methodologies and Models

| 129 |
 DOCTORAL
COLLOQUIUM
 Intelligent Playful Environments for Animals
P. Pons | J. Jaen | A. Catala

Intelligent Playful Environments for Animals

Patricia Pons, Javier Jaen, Alejandro Catala
Grupo ISSI, Departamento de Sistemas Informáticos y Computación
Universitat Politècnica de València
Camí de Vera s/n, 46022 Valencia (Spain)
[email protected], [email protected], [email protected]

ABSTRACT [12] initiative are studies where the animals interact with
We are evolving towards an interconnected and ubiquitous world, specialized keyboards and/or touch screens to communicate with
where digital devices and interfaces progressively adapt the trainers. Technology has also been used to improve critical
themselves to fit our needs and ease our daily activities. Although tasks of some animals: ACI principles have been applied to
we coexist with plenty of animal species, such as our pets, we are develop suitable dog-oriented interfaces for Diabetes Alert Dogs
approaching the evolution of technology in a strictly human- (DAD), allowing them to alert emergency services when their
centric manner. A new field in Computer Science, called Animal- owners suffer a hypoglycemic attack [8].
Computer Interaction (ACI), aims at filling this technological gap In recent years, several studies have focused on enhancing animal
by developing systems and interfaces specifically designed for play using technology for different purposes. LonelyDog@Home
animals. Supporting animals’ natural behavior and habits with [1] proposes a web interface where the human can connect to a web
suitable technology could improve both humans and animals’ camera located at her home and check how their pets are, feed them or
wellbeing. As a consequence, this doctoral research aims to play with them remotely. Canine Amusement and Training [11] is a
explore, design and develop animal-centered intelligent systems training system where, using a projector and cameras, the training
that focus on enhancing one of the most natural animal behaviors: activity becomes a playful experience. Felino [10] is a tablet-
play. Therefore, the main goal of this research is to expand ACI based game where a human and a cat can play together capturing
with the ability of automatically manage and adapt animals play different creatures on the screen. The human controls several
activity in order to improve their wellbeing. aspects of the game, adapting it to the cat’s reactions and
preferences.
Categories and Subject Descriptors
Although there are already games for animals which make use of
H.5.3 [Information Interfaces and Presentation]: User
technology to improve their experience, these games have some
interfaces – interaction styles, user-centered design, input devices
limitations which will be outlined in the next section.
and strategies.

General Terms 3. INTELLIGENT PLAYFUL

Design, Experimentation. ENVIRONMENTS FOR ANIMALS
3.1 Motivation
Keywords The focus of this research activity on animal play is motivated by
Animal Computer Interaction; Playful Interaction; Intelligent two reasons. On the one hand, play is one of the most natural
Environment; Animals; Interfaces. behaviors among all animal species. Animals do not need to be
taught on how to play [2], as it stands as a natural and voluntary
1. INTRODUCTION activity, which has also its repercussion in elder stages of their
Human Computer Interaction (HCI) has brought countless lives. As an example, felines’ play is always based on hunting or
benefits to human wellbeing. By easing the interaction with chasing behaviors, as in the wild, these animals traditionally used
computer systems, technology has become more user-friendly and to hunt for food. On the other hand, verbal or written
it is being applied to our everyday activities in a seamless manner. communication is unfeasible when considering animals as target
However, other species are not taking advantage of all these users, and thus, other methods should emerge in order to
progresses. Due to different physical features or mental determine the perceived usability and usefulness of the system.
perceptions of the world, animals are not able to understand our Therefore, playful interactions could be an effective way of
technology as we do. Therefore, a new research field called communicating with animals and gathering information about the
Animal Computer Interaction (ACI) [3,4] has emerged. It focuses usefulness of a system. As summarized in Section 2, animal play
on providing animals with similar technological improvements to has already started to be the object of a digital revolution.
the ones humans already have. ACI considers animals as the However, existing digital games for animals are focused on a
target users of digital systems, and proposes the development of single activity or purpose and only use one digital device.
interfaces specifically designed for them. The doctoral research Moreover, they require human mediation for the animal to play
presented in this paper proposes expanding ACI with the ability of and with time the animal might lose interest when it gets used to
automatically managing and adapting animals play activity in play the same game.
order to improve their wellbeing, focusing on animal pets and
specifically in cats and dogs. The main contribution of this doctoral research is two-fold.
Firstly, we will consider digital games for animals as playful
2. RELATED WORKS environments composed of several interconnected digital devices.
There have been several research studies involving animals which Those devices have to be appealing and suitable for the animal
use technology as a bridge to communicate or interact with them. and for the purpose of the game. Secondly, a playful environment
The LANA Project [9], the Ai Project [5] or the SpeakDolphin of this characteristics will be an intelligent system which should

XVI Congreso Internacional de Interacción Persona-Ordenador | Doctoral Colloquium

| 131 |
 Intelligent Playful Environments for Animals
P. Pons | J. Jaen | A. Catala
intelligently make use of its composing devices in order to create digital systems. We have already defined these environments,
suitable playful activities for different purposes. The intelligent their features and potential [6,7]. Our immediate future work will
environment should gather reliable information about the animal, be conducting a set of experiments in order to answer question Q1
its preferences, physical features and its surrounding context. in Section 3.3. For this purpose, we will study how different
With this knowledge, the intelligent environment should create an devices and stimuli affect the attention and motivation of the
engaging playful activity for a specific situation, deciding which animal for starting a playful activity. Based on the results of these
devices and interaction mechanisms would be more suitable for experiments, we will design suitable activities in order to conduct
the given context. There are plenty of scenarios where intelligent future studies to answer the remaining research questions.
playful environments could help to improve animals’ wellbeing,
e.g., create playful activities to alleviate the stress and anxiety of 5. REFERENCES
animals alone at home, zoos or shelters. It could also encourage [1] Hu, F., Silver, D., and Trude, A.LonelyDog@Home.
these animals to perform physical exercise by means of play. The 2007 IEEE/WIC/ACM International Conferences on Web
system should adapt the game to the animal’s attributes such as Intelligence and Intelligent Agent Technology - Workshops, IEEE
species, weight or age - old animals should not perform as much (2007), 333–337.
physical exercise as young ones. An intelligent playful [2] Huizinga, J.Homo ludens. Wolters-Noordhoff,
environment could also assist in animal training, making the Groningen, The Nederlands, 1985.
activity more amusing and less repetitive, and without requiring
human supervision. The environment could chose the optimal [3] Mancini, C.Animal-computer interaction: a manifesto.
time of the day to perform training, i.e., when the animal’s Magazine interactions 18, 4 (2011), 69–73.
attention and motivation seems to be higher. [4] Mancini, C.Animal-computer interaction (ACI):
changing perspective on HCI, participation and sustainability.
3.2 Definition CHI ’13 Extended Abstracts on Human Factors in Computing
As part of the planning stage of the research, we have already Systems, ACM Press (2013), 2227–2236.
defined the scope of the systems being developed [6,7]:
[5] Matsuzawa, T.The Ai project: historical and ecological
An intelligent playful environment for animals, or IPE4A, is an contexts. Animal cognition 6, 4 (2003), 199–211.
animal-centered ecosystem with intelligent capabilities which is [6] Pons, P., Jaen, J., and Catala, A.Animal Ludens :
able to learn from the animals’ behaviors and interactions, using Building Intelligent Playful Environments for Animals. 11th
the acquired knowledge to adapt itself to the context, creating Conference on Advances in Computer Entertainment - Workshop
engaging playful activities which do not necessarily need human on Animal Human Computer Interaction, (2014).
mediation to evolve.
[7] Pons, P., Jaen, J., and Catala, A.Envisioning Future
We have also outlined the different features and dimensions of Playful Interactive Environments for Animals. In A. Nijholt, ed.,
intelligent playful environments which can affect the design and More Playful User Interfaces. Springer, 2015.
development of these systems [6,7]: number of participants,
[8] Robinson, C., Mancini, C., van der Linden, J., Guest,
species of the participants, human participation, human presence,
C., and Harris, R.Empowering assistance dogs : an alarm interface
control, information acquisition, learning inputs, types of stimuli,
for canine use. Intelligent Systems for Animal Welfare, (2014).
purpose of the activity.
[9] Rumbaugh, D.M., Gill, T. V., Brown, J. V., et al.A
3.3 Research questions computer-controlled language training system for investigating
This thesis will give answer to several fundamental questions on the language skills of young apes. Behavior Research Methods &
how to provide successful interactions and communication Instrumentation 5, 5 (1973), 385–392.
between the animal and the intelligent system. Some of these [10] Westerlaken, M. and Gualeni, S.Felino: The
research questions are: Philosophical Practice of Making an Interspecies Videogame. The
• Q1: Which are the most appropriate interactive Philosophy of Computer Games Conference, (2014), 1–12.
mechanisms for the animal, e.g. a cat or a dog? [11] Wingrave, C.A., Rose, J., Langston, T., and LaViola,
J.J.J.Early explorations of CAT: canine amusement and training.
• Q2: How can the system gather information
CHI ’10 Extended Abstracts on Human Factors in Computing
unobtrusively about the animals’ preferences, actions
Systems, (2010), 2661–2669.
and context?
[12] SpeakDolphin. http://www.speakdolphin.com.
• Q3: How can the system use all these information in
order to adapt the playful activity to the context and the
animals’ preferences and wills?
• Q4: How can we objectively measure the success of the
system?

4. CONCLUSIONS AND FUTURE WORK

This thesis proposes the definition, design and development of
intelligent playful environments for animals, focusing on pet
animals. These systems aim to automatically create and adapt
playful digital experiences for animals, improving their wellbeing
and providing a better understanding of their interactions with

XVI Congreso Internacional de Interacción Persona-Ordenador | Doctoral Colloquium

| 132 |
 Multi-Display Environments to Foster Emotional Intelligence in Hospitalized Children
F. Garcia-Sanjuan | J. Jaen | A. Catala

Multi-Display Environments to Foster Emotional

Intelligence in Hospitalized Children
Fernando Garcia-Sanjuan Javier Jaen Alejandro Catala
[email protected] [email protected] [email protected]
DSIC/ISSI, Universitat Politècnica de València
Camí de Vera, S/N, 46022 Valencia (Spain)

ABSTRACT better workspace awareness, since face-to-face settings allow

Long-term and frequent hospitalized children are under high loads users to know what the others are doing.
of emotional stress, which affects their well-being in addition to Despite research showing these advantages, it is rare to see
the illness they are suffering. This thesis proposes and will focus tabletops embedded in real settings. This is due to a number of
on an approach to use Multi-Display Environments (MDE) in disadvantages: their high cost; their limited workspace
pediatric hospitalization contexts to improve patients’ emotional dimensions, which can only accommodate a certain number of
intelligence so they can deal with the negative emotions produced participants; and the fact that their form factor complicates
by their situation. mobility. Therefore, this work will address the development of a
collaborative play environment based on multiple interactive
Categories and Subject Descriptors surfaces to support the learning of emotional intelligence in
H.5.2, H.5.3 [Information Interfaces and Presentation (e.g., pediatric hospitalization contexts aimed at reducing the negative
HCI)]: User Interfaces – Input devices and strategies, interaction emotional impact caused by long-term or frequent hospital stays.
styles. Group and Organization Interfaces – Computer-supported
cooperative work. 2. RELATED WORK
K.3.1 [Computers and Education]: Computer Users in 2.1 Emotional Intelligence and Tabletops
Education – Collaborative learning, Computer-assisted Several previous works have considered technology as a way of
instruction (CAI). acquiring emotional intelligence abilities. Morris et al. [6], for
example, involve several users collectively around big interactive
General Terms surfaces in an emotional identification task, through the tagging of
Design, Experimentation, Human Factors, Standardization different images according to the emotions they cause. During the
course of the activity, the researchers observe the users speaking
Keywords with one another about their feelings towards the images, and,
Multi-Display Environments (MDE), Hospitalization, Emotional before tagging an image, they self-reflect and ask themselves
Intelligence, Child-Computer Interaction, Socialization questions like: “Am I happy?”, “Am I sad?” According to the
authors, their explorations show people’s desire of having
1. INTRODUCTION technologies that allow the complex expression of emotions
According to hospital morbidity statistics from INE for 2013, the through a game experience. However, most of previous works in
number of pediatric hospital stays in 2012 in Spain amounted to this area target adults or adolescents, and the activities performed
1,813,009. Besides the fears associated to their illness, frequent aim mainly at identification of emotions, whereas the practical use
and long periods of hospitalization provoke on children high loads of those emotions to solve problems has not yet been fully
of emotional stress and insecurity for being away from their home explored.
and their comfort zone with parents and friends. To tackle these
problems, there are previous studies that demonstrate that play Others target children with autistic spectrum disorders (ASD) to
activities allow children to face the anxiety, conflicts, and fear increase socialization through technology. As an example, Gal et
caused by hospital experiences [4]. However, the application of al. [1] make use of an interactive tabletop to perform storytelling
new technologies in the context of hospital schools has been activities to foster collaboration among kids with ASD. They
limited mostly to scenarios of individual use in which emotional observe an increment of the number of approaches between the
stress is addressed in a simplistic way by using technologies to children, a higher load of positive affect as well as a higher
enable communication channels between children and their homes tendency to express emotions. These type of works, however, are
or external schools. However, we not only aim at deviating the not aimed at making these children reason about their emotions in
children’s attention from the clinical processes they are subject to, a way they can overcome an unfavorable personal situation and
but also at facilitating the emotions that help them accept their they rely on interactive tabletops, which may be complex to
new medical situation and adapt their mental state to face the implant in real hospital environments, as stated previously.
issues that could arise by means of collaborative game activities. 2.2 Multi-Display Environments (MDE)
Since emotional intelligence is mainly of social nature, traditional MDE have been object of much previous research. Traditionally,
desktop computers are not the best option to support its training their main purpose was enlarging the visualization space via the
because these devices are mostly designed for individual use. On physical union of several smaller screens (e.g. [8]), and the
the other hand, digital tabletops have been shown to be very interaction played a more secondary role, therefore leading to
suitable tools for use in collaborative environments (e.g., [3]). mono-user environments with difficulties to favor collaboration,
Their multi-touch capabilities allow simultaneous interaction, which is fundamental in emotional learning. Besides, the
which contributes to a more democratized participation [9] and a configurations tended to be static (i.e., the number of screens was

XVI Congreso Internacional de Interacción Persona-Ordenador | Doctoral Colloquium

| 133 |
 Multi-Display Environments to Foster Emotional Intelligence in Hospitalized Children
F. Garcia-Sanjuan | J. Jaen | A. Catala
fixed a priori), and to not allow the mobility of the participants. interact with many devices at once. In order to achieve more
The posterior popularization of mobile devices, such as phones dynamic scenarios users will be able to modify the resulting
and PDAs, introduced a higher degree of mobility and multiuser surface by moving the displays through the environment during
(parallel) interaction since each user could interact with their own the course of the collaboration. And, in order to obtain richer
device, which they carried with them. Mandryk et al. [5] study the ways of interaction, our proposal will support interacting with
impressions of teenagers when using PDAs in a collaborative alternative media such as tangibles, which can result more
game. Because of the reduced dimensions of these devices, the intuitive for children as they can grab and manipulate actual
researchers encouraged them to enlarge the visualization space by physical objects. Moreover, it will also accept aerial interactions
joining the screens. The participants, however, reported low sense using gestures with fingers or dedicated objects with the purpose
of liberty of movements because they had to maintain the devices of avoiding interference with another user’s actions.
together during the course of the activity. Therefore, the mobility In sum, our goal is to design activities to foster emotional
was achieved in detriment of the visualization space. This intelligence in hospitalized children. Since this type of
problematic is due to the fact that the approaches commented so intelligence is mainly social, we propose to build collaborative
far rely basically on joining smaller screens to form a bigger one environments using several affordable surfaces such as tablets and
(and, therefore, rectangular). In this respect, some existing works smartphones. The dimensions identified previously will then be
have proposed alternatives. For instance, Pinch [7] is a technique explored beyond their technical aspects regarding how they
to couple smartphones and tablets via a manual gesture, and it impact collaboration among the children and how they foster
supports free-form topologies and dynamic modification of the emotional intelligence that might help them overcome the
device group to the extent of not needing the displays to be stressful situations they experience during their hospitalization.
physically joint once coupled. A limitation of this work, however,
is that the resulting topology is still flat and the only interaction 4. ACKNOWLEDGEMENTS
technique supported is touch. As this analysis suggests, previous This work is supported by Spanish MINECO (TIN2010-20488
research on MDE has mainly focused on technical aspects, and and TIN2014-60077-R), from Universitat Politècnica de València
their capabilities have not yet been proven in real settings but (UPV-FE-2014-24), and from GVA (APOSTD/2013/013 and
rather in experimental contexts in order to show the performance ACIF/2014/214).
of a given approach. This implicates that, in their design, the
ultimate activity to be performed by the users with these MDE has 5. REFERENCES
not been considered deeply enough. In our context, the software [1] Gal, E., Bauminger, N., Goren-Bar, D., et al. Enhancing
system and underlying interaction techniques will be considered Social Communication of Children with High-functioning
effective not only if the children are able to perform quickly the Autism Through a Co-located Interface. AI & Society 24, 1
manipulations and they understand how to make them, but also if (2009), 75–84.
the system supports effectively the construction of experiences [2] Hornecker, E., Marshall, P., Dalton, N.S., and Rogers, Y.
that develop emotional intelligence through games. Collaboration and Interference: Awareness with Mice or
Touch Input. CSCW '08, ACM (2008), 167–176.
3. RESEARCH OBJECTIVES
From previous works’ limitations, we have identified several [3] Kaminski, M., Pellino, T., and Wish, J. Play and Pets: The
technical dimensions that will be addressed in our work in order Physical and Emotional Impact of Child-Life and Pet
to successfully build collaborative MDE for children in Therapy on Hospitalized Children. Children’s Health Care
hospitalized environments. These are classified into dimensions 31, 4 (2002), 321–335.
regarding the construction of the environment and interaction [4] Mandryk, R.L., Inkpen, K.M., Bilezikjian, M., Klemmer,
requirements. S.R., and Landay, J.A. Supporting children’s collaboration
With respect to the former, our approach will allow several across handheld computers. CHI EA '01, ACM (2001), 255–
displays to form a common environment of any topology, not 256.
necessarily the traditional rectangular one, and possibly [5] Morris, M.E., Marshall, C.S., Calix, M., Al Haj, M.,
discontinuous to remove the need for the devices to be physically MacDougall, J.S., and Carmean, D.M. PIXEE: Pictures,
joint. This way, the users will be able to choose more freely their Interaction and Emotional Expression. CHI EA ’13, ACM
location. The number of devices will be unbounded, and their (2013), 2277–2286.
annexation to one another will be done in a transparent way in
order to avoid complex registration methods. Concretely, our [6] Ohta, T. and Tanaka, J. Pinch: an interface that relates
approach to address these requirements will rely on using the applications on multiple touch-screen by ‘pinching’ gesture.
built-in camera of tablets and smartphones to track their own ACE '12, Springer-Verlag (2012), 320–335.
position with respect to a wallpaper situated on the ceiling using [7] Ohta, T.Dynamically reconfigurable multi-display
computer vision algorithms, allowing the system to build a logical environment for CG contents. ACE '08, ACM (2008), 416.
map of all the tablets in the group. Since the only requirement is [8] Rick, J., Marshall, P., and Yuill, N.Beyond One-size-fits-all:
having the wallpaper/marker partially in sight, this approach can How Interactive Tabletops Support Collaborative Learning.
also allow some 3D topologies by simply placing the displays in IDC '11, ACM (2011), 109–117.
different planes.
Regarding the interactions that our MDE will allow, the most
basic one, in order to benefit from the advantages of tabletops in
terms of collaboration, is to consider all displays as part of a
whole, public surface, and therefore to permit many users interact
with the same display at the same time, and also a given user to

XVI Congreso Internacional de Interacción Persona-Ordenador | Doctoral Colloquium

| 134 |
 KINDERTIVITY: Using Interactive Surfaces to Foster Creativity in Pre-kindergarten Children
V. Nacher | J. Jaen

KINDERTIVITY: Using Interactive Surfaces to Foster

Creativity in Pre-kindergarten Children
Vicente Nacher, Javier Jaen
ISSI Group, Dpto. Sistemas Informáticos y Computación, Universitat Politècnica de València.
{vnacher, fjaen}@dsic.upv.es

ABSTRACT individual development [6]. Moreover, the technological support

Taking into account the existent educative and pedagogical to be used when developing creative characteristics in pre-
techniques, which have proved its effectiveness to foster the kindergarten children is critical since the choice of the underlying
innovation and creativity, this thesis poses to develop, experiment supporting technology has a great impact on the nature of the
and evaluate a new technological framework based on interactive pedagogical activities to be performed in a creative learning
surfaces to be applied in the context of preschool education. The context.
goal is to facilitate the three factors required for creative learning: The initial hypothesis for this thesis is that it is possible to trigger
knowledge, creative thinking and motivation but taking into in a more effective way creative collective constructivist
account the cognitive and interaction limitations of these very processes in pre-kindergarten children through the participation of
young users. multiple users in physical spaces of shared games based on
interactive surfaces. Therefore, we pretend to study the use of
Categories and Subject Descriptors these surfaces in the early phase of the cognitive development and
H.5.2. [Information interfaces and presentation]: User its suitability for the creation of educational tools.
Interfaces - Evaluation/methodology, Interaction Styles
2. RELATED WORK
General Terms Multi-touch technology has evolved quickly in recent years, from
Design, Experimentation, Human Factors, Standardization the adoption of graphical user interfaces to its wide acceptance
nowadays [1]. This technology offers new sophisticated input and
Keywords processing mechanisms that enable users to interact in a more
Pre-kindergarten, Touch interaction, Education. natural and intuitive way [19]. These characteristics have
1. INTRODUCTION triggered a new approach to developing applications for even very
young children. Supporting this evidence, the Horizon report [10]
The European Union recognizes creativity as a key factor for the
identifies mobile devices (smartphones and tablets) as one of the
economic development, hence, “Increase the creativity and the
two emerging technologies suitable for children under two years
innovation, including the entrepreneurship spirit, in all levels of
old and Rideout pointed out that children between zero and eight
the education and training” is the 4th goal of the strategic
years old are frequent users of digital media and they meet with
framework for the European cooperation in the field of education
touch technology often before they can even speak [4].
and training and it is one of the constituent elements of the new
R&D strategy in the 2020 Horizon [3]. This is a necessary This inherent ability of touch systems to engage children’s
strategy because, as Cropley points out [5], the traditional attention is being widely exploited to promote learning activities
education systems tend to frequently assume the existence of one from pre-kindergarteners to adolescents [12, 18, 11]. Moreover,
valid answer (or, at least, a preferred one) for any type of some studies have demonstrated that the technology can also be
problem. This fact prevents the generation of new ideas and used to promote collaboration between peers [16, 17, 7] and to
innovation processes. According to this, a creative student, or with foster creativity [8, 2].
a different cultural basis, can be considered as a distortion source However, the increasing interest in multi-touch technology has
or distraction in the knowledge acquisition process imparted by not as yet given rise to studies on the design of multi-touch
the teacher. However, while the individual ability to obey exactly systems for the youngest age range (2-3 years old) [9].
the given orders with discipline were in line with necessities of an
industrial society and massive production systems, the ability to Taking into account the previous works, we believe our proposal
be part of collaborative processes and direct the divergent is a step forward because of the following reasons: (1) the
thinking as the motor of creativity and innovation are essentials suitability of the interactive surfaces to support social learning
nowadays in the information society. since several subjects share the same physical space and, as it
happens in traditional technology-free games, the communication
Fostering creativity must be addressed from a very early age, even during the creation process, experimentation and reflection is
in the preschool phase, since the main cognitive processes direct and no mediated by a computer. (2) The collaborative
associated to creativity have their sources in this phase of the nature of the technologic infrastructure, in which users can carry
Permission to make digital or hard copies of all or part of this work for out different tasks in parallel and on the same table; and (3) the
personal or classroom use is granted without fee provided that copies are creative nature of the infrastructure in which users select the game
not made or distributed for profit or commercial advantage and that elements and the reactive behavior that its offer. This allows
copies bear this notice and the full citation on the first page. To copy
educators to have a direct feedback about the evolution of
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee. children’s creative mental models. These models are internalized
Interaccion’15, September 7–9, 2015, Vilanova i la Geltrú, Spain. in a collective way since the activity itself is based on reflection,
Copyright 2010 ACM 1-58113-000-0/00/0010 …$15.00. creation and experimentation processes. The goal is that also

XVI Congreso Internacional de Interacción Persona-Ordenador | Doctoral Colloquium

| 135 |
 KINDERTIVITY: Using Interactive Surfaces to Foster Creativity in Pre-kindergarten Children
V. Nacher | J. Jaen
educators have a tool on which they can measure in an effective [2] Catala, A., Jaen, J., van Dijk, B., and Jordà, S. Exploring
way the level of knowledge development, in depth and breadth, of tabletops as an effective tool to foster creativity traits. In
the divergent/convergent thinking processes and the motivation of Proc. of TEI’12, pp. 143–150.
users in the activity. [3] Comisión Europea. Conclusiones del Consejo de 12 de mayo
3. CONTRIBUTION de 2009 sobre un marco estratégico para la cooperación
In order to assess the actual skills of pre-kindergarten children europea en el ámbito de la educación y la formación («ET
with interactive surfaces, we have performed an evaluation of a 2020»). 2009.
set of basic multi-touch gestures [14]. The results showed that [4] Common Sense Media. Zero to Eight: Childrens Media Use
although only the tap and drag gestures are used in commercial in America 2013. 2013.
applications targeted to pre-kindergarten children, there are [5] Cropley, A.J. Creativity in Education and Learning: A Guide
additional multi-touch gestures that can be performed by them for Teachers and Educators. Kogan Page, (2001).
(one finger rotation, scale up and scale down). In addition, this
study provides a set of design guidelines to define and improve [6] Damon, W., Lerner, R.M., Kuhn, D., and Siegler, R.S., eds.
the interactions of these particular users. The application of Handbook of Child Psychology, Volume 2, Cognition,
several of these design guidelines [15] showed that more Perception, and Language. Wiley, 2006.
problematic gestures, such as double tap and long press, can be [7] Fleck, R., Rogers, Y., Yuill, N., et al. Actions speak loudly
suitable for pre-kindergarten children too. with words. Proc. of ITS’09, pp. 189-196.
On the other hand, we have carried out another study addressing [8] Helmes, J., Cao, X., Lindley, S.E., and Sellen, A. Developing
communicability of multi-touch gestures [13] and the results the story. Proc. of ITS’09, pp. 49–52.
proved that the use of animated languages to communicate [9] Hourcade, J.P. Interaction Design and Children. Foundations
gestures to pre-kindergarten children is possible. This opens a new and Trends® in Human-Computer Interaction 1, 4 (2007),
opportunity to new studies with training sessions in order to 277–392.
evaluate the acquisition ability of this type of languages with these
young users. [10] Johnson, L., Adams, S., and Cummins, M. The NMC Horizon
Report: 2012 K-12. The New Media Consortium, Austin,
These preliminary results have been published in international Texas, 2012.
forums and allow tackling with a lot of motivation the important
milestones that this thesis poses. To sum up, the main milestones [11] Khandelwal, M. and Mazalek, A. Teaching table: a tangible
mentor for pre-k math education. Proc. of TEI’07, 191–194.
of the thesis are the following:
[12] Mansor, E.I., De Angeli, A., and De Bruijn, O. Little fingers
-Contextualization: the evaluation of the types of interaction that on the tabletop: A usability evaluation in the kindergarten.
can be performed by pre-kindergarten children in multi-touch Proc. of TABLETOP'08, 93–96.
surfaces.
[13] Nacher, V., Jaen, J., & Catala, A. (2014). Exploring Visual
-Assistance: the definition of assistive techniques of interaction Cues for Intuitive Communicability of Touch Gestures to
that allow the increase of effectiveness of pre-kindergarten actions Pre-kindergarten Children. Proc. of ITS'14, 159–162.
with interactive surfaces.
[14] Nacher, V., Jaen, J., Navarro, E., Catala, A., and González, P.
-Communicability: the definition of effective mechanisms to Multi-touch gestures for pre-kindergarten children.
communicate which actions are expected from the user through International Journal of Human-Computer Studies 73
animated languages that allow children to be autonomous when (2015), 37-51.
using the multi-touch technology without the continuous [15] Nacher, V., Jaen, J., Catala, A., Navarro, E., and Gonzalez, P.
supervision of adults. Improving Pre-Kindergarten Touch Performance. Proc. of
-Adaptability: the definition of strategies that allow the ITS ’14, 163–166..
adaptation of the Assistance and Communicability mechanisms to [16] Rick, J., Francois, P., Fields, B., Fleck, R., Yuill, N., and
the specific skills of each user. Carr, A. Lo-fi prototyping to design interactive-tabletop
-Creativity: the construction and validation of environments that applications for children. Proc. of IDC’10, pp. 138–146.
foster creativity and allow collaboration between users (pre- [17] Rick, J. and Rogers, Y. From DigiQuilt to DigiTile: Adapting
kindergarten children) to obtain in the future more adequate educational technology to a multi-touch table. Proc. of
educational tools for these users. TABLETOP'08, pp. 73–80.
[18] Sluis, R.J.W., Weevers, I., van Schijndel, C.H.G.J., Kolos-
4. ACKNOWLEDGMENTS Mazuryk, L., Fitrianie, S., and Martens, J.B.O.S. Read-It:
Work supported by the MINECO (grants TIN2010-20488 and
Five-to-seven-year-old children learn to read in a tabletop
TIN2014-60077-R) and from GVA (ACIF/2015/075).
environment. Proc. of IDC’04, pp. 73–80.
5. REFERENCES [19] Smith, S.P., Burd, E., and Rick, J. Developing, evaluating
[1] Buxton, B. Multi-touch systems that I have known and loved. and deploying multi-touch systems. International Journal of
2013. http://billbuxton.com/multitouchOverview.html. Human-Computer Studies 70, 10 (2012), 653–656.

XVI Congreso Internacional de Interacción Persona-Ordenador | Doctoral Colloquium

| 136 |
 Facial Emotion Analysis in Down’s syndrome children in classroom
P. Torres-Carrión | C. González-González

Facial Emotion Analysis in Down’s syndrome children in

classroom
Pablo Torres-Carrión Carina González-González
Universidad Técnica Particular de Loja Universidad de La Laguna
Ecuador Spain
[email protected] [email protected]
ABSTRACT have expanded considerably in the last three decades [14]
Sentiment analysis has allowed a degree of machines "intelligent" improving your quality of life standards.
interaction, although its development is still at an early stage. This People with DS usually have well-developed emotional
document presents the progress of the research project in this line, intelligence, maintain good social interaction skills. They are
with a particular focus on children with Down Syndrome (DS). communicative and respond well to the demands of the
Their singularities are considered through personalized learning environment [2]. The emotions of the people seldom are
resources, and interaction with Kinect HCI, and the Tango:H expressed in words, so it is necessary to interpret them from non-
platform. In this first moment we made an initial interaction, verbal channels such as: the gestures, tone of voice and facial
resulting in video images of the interaction, and a subjective expression [7]. People with SD shows emotional sensitivity that
assessment of emotions through one extension of EMODIANA. allows them to capture emotions that others go unnoticed [2].
The next step will be the comparison of the patterns of facial Each person's emotional state can be expressed in different ways;
recognition applied in people without SD, to identify whether the facial expression is one of the most common ways to show
there are significant differences. others our feeling with respect to any act or thing [4]. In the case
Categories and Subject Descriptors of persons with SD, we have to consider that certain facial
characteristics distinguish them from the rest of population [3,9],
K.4.2. [Computers and Society]: Computing Milieux – Social
therefore, studies and identification of emotions patterns based on
Issues: Assistive technologies for persons with disabilities.
face, may not extend and apply to this population in general.
General Terms 2. AVANCE DE INVESTIGACION
Human Factors, Theory. Research in the field of the SEN, and specifically in people with
DS have increased over the last three decades [14]. However they
Keywords have not been significant in the area of human-computer
Down Syndrome, Emotion, Affective Computing, AI, Digital interaction (HCI) with affective computing (AC), and the
Learning. consideration of emotions during interaction.
1. INTRODUCTION 2.1 Metodology
The human person as a social entity, learn from inside the The methodology that guides this research, is a continuous work
environment that develops and lives; it is necessary therefore to qualitatively and quantitatively (see Table1). In the first time we
develop the skill of reading behaviors and emotions to infer the plan doing the emotional assessment in environments of
possible intentions [15]. In this context, the emotional intelligence interaction in the classroom. This work was done with a sample of
as part of the multiple intelligences of Gardner [6], joins two types six children (3 male and 3 female) of the Down Tenerife
of intelligences: intrapersonal and interpersonal; that refer to how Association, located in the autonomous community of Canary,
we know ourselves, the way in which we managed to control and Spain.
regulate our feelings; the ability to understand others, what you
feel and the skills to relate to others[7], respectively. The pre-test phase included meetings with teachers to identify the
population, for the experimental group and the control sample,
With the development of these intelligences, teachers and keeping the equity in terms of motor and cognitive profiles. We
psychologists comprehenden (understand from inside) to each organized an initial intervention, composed of three working
other and their environment (social field). Explicitly in the sessions. The experimental group used the platform Tango:H and
learning process, the student and teacher through educational MS Kinect; the control group used conventional resources.
activities can know, appreciate and share, generating one compre-
h-ensión integral (comprehensive understanding), which is The first data were obtained from a subjective assessment by
reflected in a significant learning [15]. teachers, about the emotions displayed by the children during the
interaction; it was carried out according to the methodology of the
The above applies to the entire student population, from initial EMODIANA [8] and their assessment scales. This instrument was
training to top, without making a cognitive or mental age adapted in a proprietary format, which allowed to measure
discrimination. Against this background, in this paper we focus on continuously the 10 basic emotions, and their possible causes:
the people with DS, that within the population with Special subject, activity and external.
Educational Needs (SEN), represent one of the largest and most
vulnerable around the world [10]. The process of educational and
social adaptation of this population requires an individual
treatment, due to psychological, cognitive and kinesthetic abilities
[1]. Scientific research about their characteristics and possibilities
of performance in different areas of life: social, work and school,

XVI Congreso Internacional de Interacción Persona-Ordenador | Doctoral Colloquium

| 137 |
 Facial Emotion Analysis in Down’s syndrome children in classroom
P. Torres-Carrión | C. González-González
Table 1. Methodology for the emotional assessment[13] Additionally, there is the need to relate these algorithms in
Fase Tool Actors classrooms and in our case with children with SD.
Pre-test - Competency Test - Testers 4. REFERENCES
- DS’s people [1] Amadó, A., Benejam, B., Mezuca, J., Serrat, E., and Vallès-
Majoral, E.Socio-cognitive abilities in children with Down’s
Test - EMODIANA - Testers
syndrome: results of a preliminary study. International
- Rehabilitation platform (cognitive - Teachers Medical Review on Down Syndrome 16, 3 (2012), 34–39.
and physical) TANGO:H. - Physiotherapist [2] Angulo A. Luna, M., M.G. and Prieto, I.Manual de Atención
- Record of videos - DS’s children al Alumnado con Necesidades Específicas de Apoyo
Post-test - Competency Test - Testers Educativo Derivadas de Sindrome Down. Junta de
Andalucía, Sevilla, 2010.
- EMODIANA - DS’s people
- Observation Method (videos)
[3] Carvajal, F., Fernández-Alcaraz, C., Rueda, M., and Sarrión,
L.Processing of facial expressions of emotions by adults with
- HER Tool (videos) Down syndrome and moderate intellectual disability.
For a further analysis all sessions were recorded on video. These Research in developmental disabilities 33, 3 (2012), 783–
videos will undergo evaluation patterns, to contrast with results 790.
obtained by these tools in environments of interaction with non- [4] Cerezo, E. and Hupont, I.Emotional Facial Expression
SD. This contrast, in addition to the subjective assessment Classification for Multimodal User Interfaces. In Articulated
performed with the EMODIANA, will allow establish scientific Motion and Deformable Objects. Springer Berlin Heidelberg,
findings about the possible differences between people with and Berlin, Heidelberg, 2006, 405–413.
without SD, in terms of patterns and AI algorithms on recognition
of emotions previously validated. [5] Dibeklioğlu, H., Salah, A.A., and Gevers, T.Recognition of
Genuine Smiles. IEEE Transactions on Multimedia 17, 3
The objective pursued is to customize in real time, both teaching (2015), 279–294.
resources such as teaching strategies, based on the cognitive,
motor and emotional profile of the student with SD. We believe [6] Gardner, H.Estructuras de la mente: La teoría de las
that learning will be more significant. inteligencias múltiples. Fondo de Cultura Económica,
México, 2001.
3. BRIEF REVIEW OF LITERATURE [7] Goleman, D.Inteligencia emocional. Kairo´s, Barcelona,
3.1 HCI and Affective Computing 2011.
Every human relationship contains many and varied doses of [8] González-González, C.S., Cairós-González, M., and
affection, knowledge and interaction [15]. The literature about Navarro-Adelantado, V.EMODIANA: Un instrumento para
"affective" includes research and study areas, where relates: la evaluación subjetiva de emociones en niños y niñas. Actas
psychology, science of consciousness, neuroscience, engineering, del XIV Congreso Internacional de Interacción
computer science, sociology, philosophy, and medicine; from this PersonaOrdenador, (2013).
epistemological context seeks to understand the feelings, such as: [9] Hippolyte, L., Barisnikov, K., Van der Linden, M., and
emotion, motivation, attention, memory, among others [11]. Detraux, J.-J.From facial emotional recognition abilities to
3.2 Emotions through the face and gestures emotional attribution: A study in Down syndrome. Research
Research about emotion recognition are held mainly in facial in developmental disabilities 30, 5 (2009), 1007–1022.
expressions, intonation of voice and gestural traits [11]. In the [10] Hodapp, R.M. and Freeman, S.F.N.Advances in educational
corresponding face, muscle movements associated with strategies for children with Down syndrome. CURRENT
sentimental or emotional situation have been related to patterns OPINION IN PSYCHIATRY 16, 5 (2003), 511–516.
that relate basic emotions like raised them by Ekman: joy, [11] Picard, R.W., Papert, S., Bender, W., et al.Affective learning
surprise, disgust, anger, sadness and fear, along with a Neutral - a manifesto. BT Technology Journal 22, 4 (2004), 253–268.
option [4]. These emotions can be extended up to levels of
valuation of complex emotions, product of its combination and [12] Tapia, S.A.A., Gomez, A.H.F., Corbacho, J.B., et al.A
semantic reading of the sequence in which certain gestures are contribution to the method of automatic identification of
shown [12]. human emotions by using semantic structures. 2014
International Conference on Interactive Collaborative
The recognition of genuine emotions [5] against those that occur Learning (ICL), IEEE (2014), 60–70.
in environments where the user poses specifically for an
evaluation of the emotion, varies significantly. Therefore, it is [13] Torres-Carrion, P., González-González, C.S., and Mora
necessary that these studies are carried out in the natural Carreño, A.Methodology of emotional evaluation in
environment of interaction, avoiding as far as possible the education and rehabilitation activities for people with Down
variation of expressions by external agents [11]. syndrome. XV International Conference on Human
Computer Interaction, (2014), 12–15.
In the TED-Woman 2015 event, the PhD. Rana el Kaliouby of the
Affectiva company, presented the Affdex SDK, that allows the [14] Troncoso, M.V. and Flórez, J.Comprensión en la lectura de
recognition of emotions in real time, supported in a study of las personas con síndrome de Down. Revista Síndrome de
approximately two million videos, coming from more than 75 Down: Revista española de investigación e información
countries. This is one of the reference works in the area. sobre el Síndrome de Down, 2011, 50–59.

XVI Congreso Internacional de Interacción Persona-Ordenador | Doctoral Colloquium

| 138 |
 Facial Emotion Analysis in Down’s syndrome children in classroom
P. Torres-Carrión | C. González-González
De Zubiría, M.Inteligencia Emocional. Fundación Alberto
Meraní para el Desarrollo de la Inteligencia, Bogotá, 1999.

XVI Congreso Internacional de Interacción Persona-Ordenador | Doctoral Colloquium

| 139 |
CHILD COMPUTER INTERACTION
 Métodos y técnicas para la evaluación de la experiencia emocional de niños y niñas con videojuegos activos
C. S. González-González | V. Navarro-Adelantado

Métodos y técnicas para la evaluación de la experiencia

emocional de niños y niñas con videojuegos activos
Carina S. González-González Vicente Navarro-Adelantado
Departamento de Ingeniería Departamento de Didácticas
Informática y de Sistemas Específicas
Universidad de La Laguna Universidad de La Laguna
[email protected] [email protected]

ABSTRACT La definición, el modelado y la representación de las emociones

En este trabajo se presenta una propuesta metodológica para la sigue siendo objeto de estudio en el campo de la psicología y de la
evaluación de la experiencia emocional de niños y niñas con computación afectiva [6]. Existen dos enfoques
videojuegos activos. Esta propuesta pretende dar respuesta a fundamentalmente [7]: el enfoque categórico o discreto y el
algunos de los problemas más frecuentes encontrados en la enfoque dimensional. El enfoque categórico se basa en la
evaluación de productos interactivos con menores. Para ello, se ha clasificación de emociones en categorías [8], por lo que supone
realizado un análisis de los principales inconvenientes de las una representación discreta de las emociones incapaz de reflejar el
técnicas tradicionales de evaluación de la usabilidad, calidad de amplio rango de emociones complejas que un ser humano puede
uso y de la experiencia de usuario y emocional en niños y niñas, expresar. El segundo enfoque es dimensional [9], y establece que
así como de las relaciones entre las emociones, los videojuegos y las emociones conforman un espacio continuo multidimensional
los juegos activos. En base a este análisis, se ha elaborado una en el que unas y otras no son independientes sino que se
propuesta de evaluación que contiene: un conjunto de emociones relacionan de forma sistemática, siendo el modelo tridimensional
más representativas para los videojuegos activos y diferentes (valencia, activación y control) [10] uno de los más extendidos.
instrumentos y técnicas creados y/o adaptados para el caso de los Existen distintos instrumentos que realizan este tipo de
juegos motores con mediciones de tipo subjetiva (verbal, no evaluaciones, entre los que encontramos SAM [11] o Premo©
verbal), objetiva (biométrica) y de interacción social (Product Emotion Measurement Instrument) [12]. Este último se
(comportamiento), organizados en diferentes momentos de una basa a su vez en SAM, pero está especialmente diseñado para
sesión de evaluación. Esta propuesta permite conocer si las medir la experiencia emocional de los usuarios con productos.
respuestas emocionales son debidas a razones internas a la Aunque este método ha sido utilizado con éxito en niños, la
persona, a la actividad realizada o externas (contexto). cantidad de emociones que tiene (14) es potencialmente
problemático para su uso como un método de auto-reporte durante
Categories and Descriptors una actividad [13]. Esto se debe a que la carga cognitiva que
H.5.2 [User Interfaces]: Evaluation/Methodology produce es significativa para los niños y niñas, ya que deben
identificar, distinguir y seleccionar sus respuestas emocionales
Terms generals entre 14 emociones. Igualmente, existen dificultades de
Measurement, Human Factors asociación entre el lenguaje utilizado por los niños y su asociación
con la representación gráfica de la emoción [13]. Por este motivo
Palabras clave nuestra metodología utiliza un instrumento denominado
EMODIANA [14], específicamente diseñado y validado para
Evaluación de emociones en niños y niñas, metodología de tener en cuenta las dificultades cognitivas de los niños y niñas, y
evaluación emocional, evaluación de la experiencia de usuario que se basa en las fortalezas de los métodos anteriores. Además,
(UX) en niños y niñas, videojuegos activos, juegos motores. como videojuego activo utilizaremos una plataforma de juegos de
rehabilitación cognitiva y física basada en interacción gestual
1. INTRODUCCIÓN creada por el equipo de investigación, llamada TANGO:H
La evaluación emocional intenta recopilar y medir información
(http://tangoh.iter.es).
sobre aspectos cualitativos y cuantitativos de la experiencia de un
usuario [1]. El instrumento de medición dependerá en gran Para poder crear una metodología adecuada al caso de
medida de lo se busca obtener, pudiendo distinguir entre videojuegos activos y niños, primero se deben identificar los
instrumentos no verbales (objetivos) y verbales (subjetivos). problemas más frecuentes encontrados en la evaluación de la
Actualmente existen distintas técnicas de medición emocional, experiencia del usuario (UX) con niños y cuáles son las técnicas
tales como técnicas que permiten analizar las expresiones faciales, más idóneas para la evaluación con menores. Luego se deben
técnicas que miden reacciones fisiológicas (por ejemplo, latido del identificar las emociones relacionadas con los juegos y
corazón, sudor, dilatación de la pupila o las ondas cerebrales) o videojuegos, específicamente activos, para ver las técnicas e
las técnicas de medición subjetiva de sentimientos a través de instrumentos que permitan medir de forma fiable una intervención
cuestionarios, entrevistas y auto-informes [2-5]. Nuestra lúdica-educativa utilizando juegos activos con niños. Por ello, a
metodología de evaluación emocional utiliza técnicas objetivas y continuación, se describen los problemas más frecuentes de la
también subjetivas. evaluación emocional en niños y las técnicas de UX más
recomendadas. Posteriormente, analizaremos el conjunto de
XVI Congreso Internacional de Interacción Persona-Ordenador | Child Computer Interaction
| 141 |
 Métodos y técnicas para la evaluación de la experiencia emocional de niños y niñas con videojuegos activos
C. S. González-González | V. Navarro-Adelantado
emociones relacionadas al juego motor y videojuego activo. Método Ventajas Inconvenientes
Luego, presentaremos la metodología creada específicamente para
Observación -La interacción con -La estructura de la
este caso y las conclusiones obtenidas.
pasiva el sistema es más observación puede
2. TRABAJOS RELACIONADOS (estructurada) natural ya que el influir en los resultados
niño/a no se obtenidos.
2.1 Evaluación de la experiencia interactiva y encuentra
emocional en niños y niñas Observación presionado por la -Con alguna frecuencia
pasiva (no evaluación y los aparecen cuestiones
La diversión es un atributo de la experiencia de usuario que es
estructurada) evaluadores. fuera del ámbito de la
importante medir, ya que es una de las principales motivaciones
guía de observación que
que tienen los niños/as para interactuar con la tecnología [15] y -Permite observar la pueden pasar
además, es uno de los factores más relevantes asociados con los experiencia del desapercibidas
juegos. Por tanto, podemos decir que la diversión es una variable niño/a en su entorno
que se requiere medir en un estudio de la experiencia del usuario (familiar). -Los datos obtenidos
con los niños/as. Existen varios métodos de evaluación de la suelen ser demasiado
-Adecuado para difíciles de analizar.
experiencia del usuario con niños/as, entre los que podemos
descubrir problemas
encontrar: tarjetas gráficas para la identificación de problemas
inesperados en la
(Problem Identification Picture Cards), el kit de herramientas de
interacción como
diversión (Fun Toolkit) y la escala (Laddering) [16]. Muchos de
son los patrones de
estos nuevos métodos de evaluación de la experiencia del usuario
uso.
se basan en el uso de instrumentos como encuestas u otras
técnicas relacionadas. El problema es que existen grandes -Resultados fiables.
diferencias en las habilidades cognitivas y de desarrollo entre los
niños de la misma edad [17], lo cual en las técnicas subjetivas, se
pone en juicio la validez y fiabilidad de las respuestas de los niños Pensamiento -Los niños/as - Puede ser exigente
[18]. Al evaluar a niños/as encontramos problemas tales como en voz alta pueden explicar por cognitivamente e
complaciencia (satisficing), sugestionabilidad (suggestibility) e qué encuentran el incómodo para los
incomprensión (misunderstanding) [19]. Por ello, es necesario producto fácil, niños/as.
Método divertido, aburrido,
trabajar en la línea de maximizar la fiabilidad de las respuestas de
conductor difícil, etc.
los niños/as con el fin de asegurar la validez e integridad de los
- Las respuestas suelen
resultados. -Se puede adecuar la ir ligadas a cuestiones
sesión de evaluación específicas planteadas y
La investigación ha evaluado el Fun Toolkit con distintos métodos
para hacerla cómoda no sobre el proceso
de evaluación de juegos [16]. Los resultados de este estudio
y agradable, interactivo real.
mostraron que los dos métodos produjeron resultados muy
adecuada al niño/a.
similares y fueron comparables para identificar las preferencias en
el juego. También se han evaluado técnicas de evaluación de -El evaluador puede
usabilidad usadas en adultos con los niños, tales como las técnicas influenciar al niño/a -Normalmente se
de pensar en voz alta, entrevistas y cuestionarios [20]. Se ha y guiar el proceso realizan en laboratorio,
demostrado que los niños pueden identificar y reportar los hacia un fuera del contexto
problemas de usabilidad. Por ejemplo, los niños fueron capaces de determinado familiar al niño/a.
detectar problemas de usabilidad que ayudarían en el diseño de resultado o proceso.
juegos para niños de 4-9 años [21]. Sin embargo, en la realización Mago de Oz -Permite evaluar un -Se requiere la
de una investigación de usabilidad con los niños hay una serie de producto no participación de un
retos que deben tenerse en cuenta. Así, en un estudio [22] de 70 terminado evaluador que conozca
niños sólo 28 de ellos hicieron observaciones verbales durante la todo el sistema que
prueba de usuario. Esto puede atribuirse a su personalidad, ya que -Permite descubrir
aspectos de la cumpla dicho rol y
un estudio mostró que las características de personalidad influyen brinde las respuestas
en el número de problemas identificados [23], por consiguiente, se interacción no
contemplados interactivas simuladas.
hace necesario investigar para comprender las limitaciones y
aplicaciones en niños/as con relación a diferentes métodos. En anteriormente
Padilla Zea et al (2012) [24], podemos encontrar una clasificación -Para los niños/as
de diferentes métodos de evaluación de productos interactivos puede resultar
procedentes de la evaluación de usabilidad y calidad de uso, y los divertido y fácil
pros y contras de su utilización en niños y niñas. Asimismo, los realizar un juego de
mismos autores proponen una metodología para la evaluación de rol, por lo que se
la jugabilidad en videojuegos educativos en niños y niñas de 3 a 7 aumentan las
años. emociones positivas
hacia el producto
Tabla 1. Métodos de evaluación de la experiencia interactiva
con productos para con niños (Adaptado de Padilla-Zea et al.
[24]). Cuestionarios - Se puede preguntar - Poca fiabilidad en la
Método Ventajas Inconvenientes a los niños a la vez y respuesta de los

XVI Congreso Internacional de Interacción Persona-Ordenador | Child Computer Interaction

| 142 |
 Métodos y técnicas para la evaluación de la experiencia emocional de niños y niñas con videojuegos activos
C. S. González-González | V. Navarro-Adelantado
Método Ventajas Inconvenientes con videojuegos activos incluyendo elementos de interacción
social y colaborativa. Por ello, en este trabajo se propone una
obtener gran niños/as.
metodología específicamente diseñada para la evaluación de este
cantidad de datos.
tipo de productos con niños y niñas.
Entrevistas -Permite conocer en -Dificultad de los
profundidad la Las respuestas emocionales pueden darse a través de diversos
niños/as en comprender canales. El canal tradicional de detección emocional han sido las
experiencia las preguntas.
individual de cada imágenes faciales, pero en los últimos años se han ido
participante. -Análisis de datos incorporado otros canales como el sonoro (voz) y, muy
laborioso, con mucha especialmente, los fisiológicos [26]. Asimismo, existen diversos
-Se pueden obtener información de tipo tipos de instrumentos para medir las respuestas emocionales. Por
datos cualitativos cualitativa. ejemplo para las respuestas de tipo verbal, algunos de los
más ricos que los instrumentos más comunes son la escala Likert, escala de
obtenidos en otros diferenciador semántico, el perfil emocional estandarizado (SEP,
métodos. el perfil de reacción (Reaction Profile) o la escala sentimiento
Grupos - Permite debatir hacia anuncios (Feelings Toward Ad Scale). Para las respuestas de
focales sobre diversos temas tipo no verbal, los instrumentos más comunes son PrEmo,
en profundidad EmoCards, SAM,, LemTool o GEW. Y para medir las respuestas
fisiológicas, para las expresiones faciales algunos instrumentos
más comunes son: Affdex Facial Coding, FaceAPI, Facesense,
- El evaluador guía Facereader, FaceSDK; para señales cerebrales el Emotiv-EPOC y
la discusión hacia para otras variables biométricas los Affective wearables [27].
los temas que
interesan evaluar, En Méndez et al (2015) [27] se presenta un marco de referencia
permitiendo ahondar como complemento a la evaluación de sistemas interactivos con
en variables tres componentes principales de evaluación: comportamiento,
cualitativas de la fisiológico y subjetivo. El componente de comportamiento se
experiencia del centra en evaluar el comportamiento del usuario, considerando
usuario específicamente las características relacionadas con la expresión
(satisfacción, facial, dirección de la mirada, postura, movimientos de la cabeza
impacto emocional, y gestos. El componente fisiológico se centra en los aspectos
etc.) fisiológicos de los usuarios, considerando específicamente la
actividad cerebral del usuario durante la evaluación. Por último, el
Métodos -Permite la -Las heurísticas se componente subjetivo se centra en recoger la propia opinión del
basados en evaluación cuando centran tradicionalmente usuario, al respecto de las emociones que el usuario considera que
heurísticas la participación de en aspectos pragmáticos evocó mientras realizaba las diferentes tareas durante la
los niños/as es (usabilidad) y se puede evaluación del sistema interactivo. Este enfoque metodológico no
difícil. perder información contempla el componente social y cooperativo de los usuarios,
Métodos hedónica (diversión, cuestión que sí es abordada en nuestra propuesta.
basados en - Ayuda a obtener
información con un entretenimiento, etc.)
recorridos 2.2 Emociones, videojuegos y juegos activos
número reducido de - Pueden mostrar
expertos (5) problemas que no Aunque existe una gran variedad de interpretaciones y
existen y se puede definiciones del concepto de emoción [28], podemos decir que se
-Son válidos en trata de un estado complejo del organismo caracterizado por una
evaluaciones perder los problemas
que realmente si están. excitación o perturbación que predispone a una respuesta
tempranas de organizada [29], de acuerdo con la evaluación subjetiva que
prototipos - Los expertos pueden realiza cada persona del significado del evento que la ha originado
desconocer lo que [30].
realmente siente al
interaccionar un niño/a Los videojuegos son poderosos generadores de emociones. Varios
con el producto autores, se han centrado en analizar qué emociones pueden
interactivo. generarse, y de cómo surgen [31]. Es posible distinguir dos
niveles en la vivencia de emociones [32]:
-Dificultad de encontrar
expertos en los diversos • El jugador como observador-participante: Se refiere a