Multitouch

An increasing number of today's consumer devices such as mobile phones or tablet computers are equipped with various sensors. The extraction of useful information such as gestures from sensor-generated data based on mainstream imperative languages is a notoriously difficult task. Over the last few years, a number of domain-specific programming languages have been proposed to ease the development of gesture detection. Most of these languages have adopted a declarative approach allowing programmers to describe their gestures rather than having to manually maintain a history of event data and intermediate gesture results. While these declarative languages represent a clear advancement in gesture detection, a number of issues are still unresolved. In this paper we present relevant criteria for gesture detection and provide an initial classification of existing solutions based on these criteria in order to foster a discussion and identify opportunities for future gesture programming languages.

Saggio su interazione uomo macchina, usabilità, valore euristico dell'arte, le più diffuse tecnologie d'interfaccia, mems, gesture recognition.
Contiene i seguenti case studies: smart laser scanner, physical motion interface, sensorizzazione di una protesi, Apple multitouch screen, Nintendo WiiMote.

In particolare, al momento della pubblicazione nell'Aprile 2006, il saggio conteneva alcuni indizi di retro-ingegneria sul controller Wii, procurato tramite l'Università quando ancora non era disponibile al pubblico (nei negozi a Dicembre 2006).

Multi-touch interaction with computationally enhanced surfaces has received considerable recent attention. Approaches to the implementation of multi-touch interaction such as Frustrated Total Internal Reflection (FTIR) and Diffused Illumination (DI) have allowed for the low cost development of such surfaces, leading to a number of technology and application innovations. Although many of these techniques have been presented in an academic setting, the practicali-ties of building a high quality multi-touch enabled surface, both in terms of the software and hardware, are not trivial. This document aims to summarize the knowledge and experience of developers of multi-touch technology who gathered at the Bootcamp on Construction & Implementation of Optical Multi-touch Surfaces at Tabletop 2008 in Ams-terdam, and seeks to provide hints and practical advice to people seeking to " build your own " multi-touch surface. We mostly focus on technical aspects that are important in the construction of optical multi-touch surfaces, including: infrared illumination, silicone compliant surfaces, projection screens, cameras, filters, and projectors. In addition, we outline how to integrate this hardware to allow users to create a solid multi-touch surface, and provide an overview of existing software libraries for the implementation of multi-touch applications. In addition, we discuss the problem of latency introduced by the different parts of the system. A brief description of most of the common technologies to realize (multi-) touch surfaces is provided; however, the main focus is on those that utilise optical approaches.

Recent advances in sensing technology have enabled a new generation of tabletop displays that can sense multiple points of input from several users simultaneously. However, apart from a few demonstration techniques [17], current user interfaces do not take advantage of this increased input bandwidth. We present a variety of multifinger and whole hand gestural interaction techniques for these displays that leverage and extend the types of actions that people perform when interacting on real physical tabletops. Apart from gestural input techniques, we also explore interaction and visualization techniques for supporting shared spaces, awareness, and privacy. These techniques are demonstrated within a prototype room furniture layout application, called RoomPlanner.

The information society manufactures, manipulates, and commodifies information. Heritage is one such area that is undergoing digital transformation. Heritage is increasingly being transmuted through digitisation devices such as laser and structured light scans into multiple representations of information. The rich information of a heritage object or an environment can be restructured, transmitted, and recomposed into a mediated form both textual and non-textual. Once digitised, it becomes free from its physical predecessor; it enters another world that defies the physical laws of nature where the imagination of the maker is a limit. Such worlds accompanied by their objects are accessible in new yet intuitive ways via surface computers. The horizontal nature of the multitouch-multiuser surface computer then becomes the mirror that links both worlds, allowing access into a virtual space via the touch-table computing paradigm. This article explores 3D surface computing, its technology, capabilities and limits with developments of two multitouch applications incorporating heritage objects and environments, and the observation of the reactions of initial users. It addresses new issues and challenges surrounding the use of surface computing and how the access and transmission of heritage information via multitouch-multiuser tables are able to contribute to the accessibility, teaching and learning of heritage.

This paper presents the findings of a user study investigating conversational patterns across three conditions of table-based interaction (direct touch interactive table, pantograph interactive table and non-digital table) for different types of educational activities. Findings demonstrate that communication style is significantly affected by interaction techniques. The direct touch technique stimulated conversations based around the topic and pedagogical method.

"Analogous perceptual experiences in multiple sensory modalities provide possibilities for abstract expression using modern computer-based audio-visual systems. This work examines the phenomenon of such cross-modal links with a focus on mappings between the auditory and visual realms.
The design and implementation of a graphics-based sound synthesis controller using such inter-sensory associations is presented. A review of literature relevant to the design of computer-based musical instruments is provided, including discussions of parameter mapping, the use of graphical displays and computer vision as a gesture capture mechanism.
An analysis of the software instruments realised and the physical interface setup is provided. System improvements and possible applications are also discussed with the direction of future work with the system being suggested."

Although multi-touch applications and user interfaces have become increasingly common in the last few years, there is no agreed-upon multi-touch user interface language yet. In order to gain a deeper understanding of the design of multi-touch user interfaces, this paper presents semiotic analysis of multi-touch applications as an interesting approach to gain deeper understanding of the way users use and understand multi-touch interfaces. In a case study example, user tests of a multi-touch tabletop application platform called MuTable are analysed with the Communicability Evaluation Method to evaluate to what extent users understand the intended messages (e.g., cues about interaction and functionality) the MuTable platform communicates. The semiotic analysis of this case study shows that although multi-touch interfaces can facilitate user exploration, the lack of well-known standards in multi-touch interface design and in the use of gestures makes the user interface difficult to use and interpret. This conclusion points to the importance of the elusive balance between letting users explore multi-touch systems on their own on one hand, and guiding users, explaining how to use and interpret the user interface, on the other.► We present semiotic analysis as a useful approach to study multi-touch interaction. ► A tabletop user interface is studied from the perspective of Semiotic Engineering. ► We propose design characteristics for a multi-touch user guidance system.

In this paper we discuss our experiences of conducting user evaluations of technological installations for the new Robert Burns Birthplace museum and the challenges associating with doing this. We believed it to be important to avoid a sterile lab situation, and to be able to observe the social dynamics that evolve around museum installations. As we were unable to conduct studies in the museum itself we devised ways of taking them out of the lab by basing them in a more museum-like or semi-realistic setting. We took efforts to recreate the installation setup when this was likely to influence interaction patterns and could reveal usability issues. Moreover, we invited young families and adult groups to take part in the evaluation sessions to investigate how they would interact with these as a group. We explain our motivations and the tradeoffs involved. We focus on discussing the effects that the semi-realistic setting may have had on the user feedback obtained, reflect on the differing user behaviours that might emerge in the genuine museum setting and make recommendations for future studies.

3D modeling has been revolutionized in recent years by the
advent of computers. While computers have become much more affordable and accessible to the masses, computer modeling remains  a complex task involving  a  steep learning curve and extensive training.  In this paper we describe  the MozArt Table, our effort to redefine the interface for computer modeling to make  it more accessible to lay users. We have explored both the  hardware and software aspects  of the interface, specifically, the use of intuitive speech commands and multitouch gestures on an inclined interactive surface.  The paper describes our approach,
hardware setup and the technology used to make it work.

Abstract This research examines phonics instruction on multi-touch display technology for collaborative learning. The paper provides a brief discussion of technology, collaborative learning and literacy instruction and then focuses on the design of a Phonics application using collaborative learning interactions. The paper discusses the methodology of a multi-phase evaluation of the usability and the instructional design of the application. A survey was used for the usability study and a focus group consisting of experts in the field of ...

MorphOSC is a new toolkit for building graphical user interfaces for the control of sound using morphing between parameter presets. It uses the multidimensional interpolation space paradigm seen in some other systems, but hitherto unavailable as open-source software in the form presented here. The software is delivered as a class library for the Processing Development Environment and is cross-platform for desktop
computers and Android mobile devices. This paper positions the new library within the context of similar software, introduces the main features of the initial code release and details future work on the project.

In this paper we present the table-based tangible interface application Xenakis which uses probability models in order to compose music in a way that can be strongly influenced by the user. Our musical sequencing application is based on a framework for tangible interfaces with an architecture that is strongly inspired by the model-view-controller pattern. In addition, we developed a hardware setup for tangible interfaces and used MatraX for tracking markers. The sequencer is the first implementation based on this framework. It allows users to create music simply by moving tangibles on the table. The graphics engine Horde3D is used to visualize the user-interaction and to show the relationships between the tangible objects on the table, creating an appealing audio-visual experience. An evaluation with 37 first time users was conducted in order to discover the strong and the weak points of such tangible user interfaces, especially in the context of our application.

Two technology trends address the rising costs of healthcare systems in aging societies: Serious Games for Healthcare and Ambient Assisted Living Environments. Surprisingly, these concepts are rarely combined and the users’ perception and use of Serious Games in Ambient Assisted Living environments is insufficiently understood. We present the evaluation of a serious game for stimulating cognitive abilities for elderly with regard to technology acceptance (based on the UTAUT2 model), performance and preference for an interaction device (tablet, table, wall). The results suggest that acceptance of serious games is independent of gender, technical expertise, gaming habits, and only weakly influenced by age. Determinants for acceptance are perceived fun and the feeling that the users can make playing the game a habit. Performance within the game is explained by age and previous gaming experience. All investigated interaction devices were rated as useful and easy to learn, although the wall-sized display had lower approval levels. The article concludes with guidelines for successfully introducing serious games for healthcare to residents in ambient assisted living environments.

This research applies Socio-Cultural theory and Distributed Cognition/Activity theory to conceptualize the design of collaborative learning activities in large shared digital spaces. The paper begins by providing a summary of previous work in the creation of a technology platform for large shared digital spaces. It then details how Socio-Cultural theory and Distributed Cognition/Activity theory can provide significant insights into the educational utility of such collaborative technology and how design issues relating to collaborative learning may be addressed. The paper then reports on an experimental study involving a Phonics literacy application, including data from a survey of 19 participants and focus groups sessions by 6 discipline experts. Themes related to Socio-Cultural theory and Distributed Cognition/Activity theory are highlighted in the results reported. Conclusions and further research opportunities involving large shared digital learning spaces are then discussed.

Over the past few years, multi-touch user interfaces emerged from research prototypes into mass market products. This evolution has been mainly driven by innovative devices such as Apple's iPhone or Microsoft's Surface tabletop computer. Unfortunately, there seems to be a lack of software engineering abstractions in existing multi-touch development frameworks. Many multi-touch applications are based on hard-coded procedural low level event processing. This leads to proprietary solutions with a lack of gesture extensibility and cross-application reusability. We present Midas, a declarative model for the definition and detection of multi-touch gestures where gestures are expressed via logical rules over a set of input facts. We highlight how our rule-based language approach leads to improvements in gesture extensibility and reusability. Last but not least, we introduce JMidas, an instantiation of Midas for the Java programming language and describe how JMidas has been applied to implement a number of innovative multi-touch gestures.

The penetration of stereoscopic 3D (S3D) output devices is becoming widespread. S3D screens range in size from large cinema screens, to tabletop displays, TVs in living rooms, and even mobile devices are nowadays equipped with autostereoscopic 3D screens. As a consequence, the requirement for interacting with 3D content is also increasing, with " touch " being one of the dominant input methods. However, the design requirements and best practices for interaction with S3D touch screen user interfaces are only now evolving. To understand the challenges and limitations S3D technology brings to interaction design, and in particular the additional demands they place on the users of such interfaces, we present our research that considers stereoscopic touch screens of different sizes in static and mobile usage contexts. We report on perceptual, cognitive and behavioral aspects revealed by user studies and present interaction design requirements based on the findings.

This paper describes a comparative study between the usage of low-fidelity and a high-fidelity prototyping for the creation of multi-user multi-touch interfaces. The multitouch interface presented in this paper allows users to collaboratively search for existing multimedia content, create new compositions with this content, and finally integrate it in a layout for presenting it. The study we conducted consists of a series of parallel user tests using both low-fidelity and high-fidelity prototypes to inform the design of the multi-touch interface. Based on a comparison of the two test sessions, we found that one should be cautious in generalising high-level user interactions from a low towards a high-fidelity prototype. However, the lowfidelity prototype approach presented proved to be very valuable to generate design ideas concerning both high and low-level user interactions on a multi-touch tabletop.

This paper describes a comparative study between the usage of low-fidelity and a high-fidelity prototyping for the creation of multi-user multi-touch interfaces. The multitouch interface presented in this paper allows users to collaboratively search for existing multimedia content, create new compositions with this content, and finally integrate it in a layout for presenting it. The study we conducted consists of a series of parallel user tests using both low-fidelity and high-fidelity prototypes to inform the design of the multi-touch interface. Based on a comparison of the two test sessions, we found that one should be cautious in generalising high-level user interactions from a low towards a high-fidelity prototype. However, the lowfidelity prototype approach presented proved to be very valuable to generate design ideas concerning both high and low-level user interactions on a multi-touch tabletop.

Existing interaction paradigms are associated with direct manipulation with indirect input devices such as the mouse.
By accounting the emergency of multitouch interactivity on reactive surfaces, it becomes relevant to discuss upon the changes brought by the direct relation of the body, trough the hand, with the contents and objects represented on surface.
From the study of the scientific literature of the elements of the interaction and the principal gestural interaction techniques, we propose a methodological framework for the design gestural interaction based on three main approaches: empirical, metaphorical and phenomenological.

—We are motivated to find a multi-touch gesture detection algorithm that is efficient, easy to implement, and prospectively scalable to real-time applications using 3D environments. Our approach tries to solve the recognition for gestures with the use of feature extraction without the need of any previous learning samples. Before showing our proposed solution, we describe some algorithms that attempt to solve similar problems. Finally, we describe our code to accomplish off-line gesture recognition.

Decisions on settlement location in the face of climate change and coastal inundation may have resulted in success, survival or even catastrophic failure for early settlers in many parts of the world. In this study, we investigate various questions related to how individuals respond to a palaeoenvironmental simulation, on an interactive tabletop device where participants have the opportunity to build a settlement on a coastal landscape, balancing safety, and access to resources, including sea and terrestrial foodstuffs, while taking into consideration the threat of rising sea levels. The results of the study were analyzed to consider whether decisions on settlement were predicated to be near to locations where previous structures were located, stigmergically, and whether later settler choice would fare better, and score higher, as time progressed. The proximity of settlements was investigated and the reasons for clustering were considered. The interactive simulation was

Physical computing empowers people to design and customize electronic hardware tailored to their individual needs. This often involves ”tinkering” with components connections, but due to the intangible nature of electricity, this can be difficult, especially for novices. We use a multi-stage design process to design, build and evaluate a physical prototyping workspace for novices to learn about real physical computing hardware. The workspace consists of a horizontal surface that tracks physical components like sensors, actuators, and microcontroller boards and augments them with additional digital information in situ. By digitally exploring various means of connecting components, users can experientially learn how to build a functioning circuit and then transition directly to building it physically. In a user study, we found that this system motivates learners by encouraging them and building a sense of competence, while also providing a stimulating experience.

In educational settings, current digital technologies often work counter-productive because people using them experience separation and isolation. This paper describes a set of multi-touch multimedia interaction applications that especially were designed to enhance co-located collaboration between users. We present the underlying framework for creating such applications. Our applications were created for supporting typical collaborative tasks performed by secondary students. We present our findings on the usage of these applications by these users in the settings of a secondary school classroom.

This paper presents two low-fidelity prototyping methods for multi-touch tabletop applications. The prototypes, only consisting of basic 'physical' materials such as paper, cardboard, markers, scissors, etc. were designed to evaluate distinct aspects of the same multi-touch platform. One prototype allowed observing tabletop interaction in general: how users interact with each other and the surface they work on. The second prototype allowed the design team to test how users react to new, innovative multi-touch controls. Introducing these prototyping methods, this paper describes a way to iterate and test designs at an early stage in the development cycle.

Deictic gestures are ubiquitous when people work over tables and whiteboards, but when collaboration occurs across distributed surfaces, the embodiments used to represent other members of the group often fail to convey the details of these gestures. Although both gestures and embodiments have been well studied, there is still little information available to groupware designers about what components and characteristics of deictic gesture are most important for conveying meaning through remote embodiments. To provide this information, we conducted three observational studies in which we recorded and analysed more than 450 deictic gestures. We considered four issues that are important for the design of embodiments on surfaces: what parts of the body are used to produce a deictic gesture, what atomic movements make up deixis, where gestures occur in the space above the surface, and what other characteristics deictic gestures exhibit in addition to pointing. Our observations provide a new design understanding of deictic gestures. We use our results to identify the limitations of current embodiment techniques in supporting deixis, and to propose new hybrid designs that can better represent the range of behavior seen in real-world settings.

In everyday life, our interactions with objects on real tables include how our fingertips feel those objects. In comparison, current digital interactive tables present a uniform touch surface that feels the same, regardless of what it presents visually. In this paper, we explore how tactile interaction can be used with digital tabletop surfaces. We present a simple and inexpensive device-the Haptic Tabletop Puck-that incorporates dynamic, interactive haptics into tabletop interaction. We created several applications that explore tactile feedback in the area of haptic information visualization, haptic graphical interfaces, and computer supported collaboration. In particular, we focus on how a person may interact with the friction, height, texture and malleability of digital objects.

Abstract Natural user interfaces (NUI) such as multitouch and surface computing are positioned as the next major evolution in computing and user interfaces. Just graphical user interfaces (GUIs) brought unprecedented interaction capabilities to their command-line predecessors, we believe multitouch and surface computing will spawn novel ways to interact with media and improve social usage patterns. Since experimentation and deployment are currently limited, the exploration of applications and interfaces in this area ...

In everyday life, our interactions with objects on real tables include how our fingertips feel those objects. In comparison, current digital interactive tables present a uniform touch surface that feels the same, regardless of what it presents visually. In this paper, we explore how tactile interaction can be used with digital tabletop surfaces. We present a simple and inexpensive device -the Haptic Tabletop Puck -that incorporates dynamic, interactive haptics into tabletop interaction. We created several applications that explore tactile feedback in the area of haptic information visualization, haptic graphical interfaces, and computer supported collaboration. In particular, we focus on how a person may interact with the friction, height, texture and malleability of digital objects.

This paper introduces a collaborative learning game called Futura: The Sustainable Futures Game, which is implemented on a custom multi-touch digital tabletop platform. The goal of the game is to work with other players to support a growing population as time passes while minimizing negative impact on the environment. The design-oriented research goal of the project is to explore the novel design space of collaborative, multi-touch tabletop games for learning. Our focus is on identifying and understanding key design factors of importance in creating opportunities for learning. We use four theoretical perspectives as lenses through which we conceptualize our design intentions and inform our analysis. These perspectives are: experiential learning, constructivist learning, collaborative learning, and game theory. In this paper we discuss design features that enable collaborative learning, present the results from two observational studies, and compare our findings to other guidelines in order to contribute to the growing body of empirically derived design guidelines for tangible, embodied and embedded interaction.

This research demonstrates the use of multi-touch display technology for collaborative tasks in the classroom via task breakdown and role assignment for individual learners. Multi-touch displays can provide a large shared digital workspace and multiple simultaneous user inputs. This paper provides a brief background discussion of improvements to a hardware and software multi-touch platform and then focuses on the design of collaborative learning software using the example of a Phonics literacy application. Collaborative learning design issues discussed include; i) learning task analysis, ii) learning task decomposition and role assignment, iii) engaging collaborative communication mechanisms, and iv) multi-person interface issues. The paper then discusses a pilot study of the usability and analysis of the data gathered. Conclusions and further research opportunities are then discussed.

We present the Rizzo, a multi-touch virtual mouse that has been designed to provide the fine grained interaction for information visualization on a multi-touch table. Our solution enables touch interaction for existing mouse-based visualizations. Previously, this transition to a multi-touch environment was difficult because the mouse emulation of touch surfaces is often insufficient to provide full information visualization functionality. We present a unified design, combining many Rizzos that have been designed not only to provide mouse capabilities but also to act as zoomable lenses that make precise information access feasible. The Rizzos and the information visualizations all exist within a touch-enabled 3D window management system. Our approach permits touch interaction with both the 3D windowing environment as well as with the contents of the individual windows contained therein. We describe an implementation of our technique that augments the VisLink 3D visualization environment to demonstrate how to enable multi-touch capabilities on all visualizations written with the popular prefuse visualization toolkit.

Most tabletop research presents findings from lab-based user studies, focusing on specific interaction techniques. This means we still know little about how these new interfaces perform in real life settings and how users appropriate them.

This paper presents findings from a field study of an existing interactive table in a museum of natural history. Visitors were found to employ a wide variety of gestures for interacting; different interface elements invited different types of gesture. The analysis highlights challenges and design conflicts in the design of tabletop interfaces for public settings, such as latency times and side-effects of ‘frame-less’ content, which had some users struggling to learn how to interact. While the majority of visitors engaged at least briefly with the table, which enabled browsing question-answer text about animal species, talk amongst visitors dealt mainly with how to interact and evoked few comments, indicating shallow
engagement with content.

Designing and assessing systems to support neighborhood participation in design is difficult due to the challenges of involving real participants and the fragile nature of early instantiations of technologies aimed at supporting open-ended and ill-structured design tasks. We report on a scenario-based, semi-realistic field trial of two prototypes of the Envisionment and Discovery Collaboratory, an environment for supporting community involvement in design activities. By engaging subjects in playing participant roles, we have been able to gain some crucial insights into the facets of the design at multiple levels as part of an ongoing design process.

Qualitative analyses further revealed how other factors can affect how participants contribute to the task. The findings are discussed in terms of how the design of the physicaltechnological set-up can affect the desired form of collaboration.