Minimally invasive therapy (MIT) is one of the most important trends in modern medicine. It inclu... more Minimally invasive therapy (MIT) is one of the most important trends in modern medicine. It includes a wide range of therapies in videoscopic surgery and interventional radiology and is performed through small incisions. It reduces hospital stay-time by allowing faster recovery and offers substantially improved cost-effectiveness for the hospital and the society. However, the introduction of MIT has also led to new problems. The manipulation of structures within the body through small incisions reduces dexterity and tactile feedback. It requires a different approach than conventional surgical procedures, since eye-hand co-ordination is not based on direct vision, but more predominantly on image guidance via endoscopes or radiological imaging modalities. ARIS*ER is a multidisciplinary consortium developing a new generation of decision support tools for MIT by augmenting visual and sensorial feedback. We will present tools based on novel concepts in visualization, robotics and haptics providing tailored solutions for a range of clinical applications. Examples from radio-frequency ablation of liver-tumors, laparoscopic liver surgery and minimally invasive cardiac surgery will be presented. Demonstrators were developed with the aim to provide a seamless workflow for the clinical user conducting image-guided therapy.
Applications in the fields of virtual and augmented reality as well as image-guided medical appli... more Applications in the fields of virtual and augmented reality as well as image-guided medical applications make use of a wide variety of hardware devices. Existing frameworks for interconnecting low-level devices and high-level application programs include high-frequency event streaming frameworks as well as dynamically typed message passing systems. Neither of these approaches exploits the full potential for processing events coming from arbitrary sources and neither is easily generalizable. In this paper, we will introduce a new multi-modal event processing methodology that uses dynamically-typed event attributes for event passing between multiple devices and systems. The existing OpenTracker framework was modified to incorporate a highly flexible and extensible event model, which can store data that is dynamically created and arbitrarily typed at runtime. The software architecture that is introduced in this work provides multi-modal event streaming, network and file logging support, and the ability to merge separate event streams. Due to the implementation of a self-descriptive event serialization mechanism, the architecture does not put any constraints on new data types to be used within the library. The main factors impacting the library's latency and throughput were determined experimentally and the overall performance was shown to be sufficient for most typical applications. Several sample applications were developed to take advantage of the new dynamic event model that is provided by the library, thereby demonstrating its flexibility and expressive power.
Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI '14, 2014
Content on computer screens is often inaccessible to users because it is hidden, e.g., occluded b... more Content on computer screens is often inaccessible to users because it is hidden, e.g., occluded by other windows, outside the viewport, or overlooked. In search tasks, the efficient retrieval of sought content is important. Current software, however, only provides limited support to visualize hidden occurrences and rarely supports search synchronization crossing application boundaries. To remedy this situation, we introduce two novel visualization methods to guide users to hidden content. Our first method generates awareness for occluded or out-ofviewport content using see-through visualization. For content that is either outside the screen's viewport or for data sources not opened at all, our second method shows off-screen indicators and an on-demand smart preview. To reduce the chances of overlooking content, we use visual links, i.e., visible edges, to connect the visible content or the visible representations of the hidden content. We show the validity of our methods in a user study, which demonstrates that our technique enables a faster localization of hidden content compared to traditional search functionality and thereby assists users in information retrieval tasks.
The First IEEE International Workshop Agumented Reality Toolkit,, 2002
We present a mobile augmented reality (AR) system to guide a user through an unfamiliar building ... more We present a mobile augmented reality (AR) system to guide a user through an unfamiliar building to a destination room. The system presents a world-registered wire frame model of the building labeled with directional information in a see-through heads-up display, and a three-dimensional world-in-miniature (WIM) map on a wrist-worn pad that also acts as an input device. Tracking is done using a combination of wall-mounted ARToolkit markers observed by a head-mounted camera, and an inertial tracker. To allow coverage of arbitrarily large areas with a limited set of markers, a structured marker re-use scheme based on graph coloring has been developed.
IEEE Transactions on Visualization and Computer Graphics, 2000
As heterogeneous data from different sources are being increasingly linked, it becomes difficult ... more As heterogeneous data from different sources are being increasingly linked, it becomes difficult for users to understand how the data are connected, to identify what means are suitable to analyze a given data set, or to find out how to proceed for a given analysis task. We target this challenge with a new model-driven design process that effectively codesigns aspects of data, view, analytics, and tasks. We achieve this by using the workflow of the analysis task as a trajectory through data, interactive views, and analytical processes. The benefits for the analysis session go well beyond the pure selection of appropriate data sets and range from providing orientation or even guidance along a preferred analysis path to a potential overall speedup, allowing data to be fetched ahead of time. We illustrate the design process for a biomedical use case that aims at determining a treatment plan for cancer patients from the visual analysis of a large, heterogeneous clinical data pool. As an example for how to apply the comprehensive design approach, we present Stack'n'flip, a sample implementation which tightly integrates visualizations of the actual data with a map of available data sets, views, and tasks, thus capturing and communicating the analytical workflow through the required data sets.
The Virtual Table presents stereoscopic graphics to a user in a workbench-like setting. For this ... more The Virtual Table presents stereoscopic graphics to a user in a workbench-like setting. For this device, a user interface and new interaction techniques have been developed based on transparent props -a tracked hand-held pen and a pad 12 . These props, particularly the pad, are augmented with 3D graphics from the Virtual Table's display that can serve as a s suitability for design tasks. Traditional CAD dialogue can be combined with intuitive rapid sketching of geometry on the pad. Additionally, the resulting events and objects can be associated with scene details below the translucent tablet.
EDT 2006: IEEE VR 2006 Workshop on Emerging Display Technologies March 26, 2006 10: 30–12: 00 Ses... more EDT 2006: IEEE VR 2006 Workshop on Emerging Display Technologies March 26, 2006 10: 30–12: 00 Session II Upponurkka: An Inexpensive Immersive Display for Public VR Installations............................ 13 T. Lokki, T. Ilmonen, W. Mäkelä, and T. Takala La Cueva Grande: a 43-Megapixel Immersive System................................................... 17 C. Canada, T. Harrington, R. Kares, D. Modl, L. Monroe, S. Stringer, and B. Tomlinson Display Research at the University of Southern California....................... ...
F orensic-case analysis relies heavily on digital information for documentation, especially to re... more F orensic-case analysis relies heavily on digital information for documentation, especially to reconstruct accident and crime scenes and to present forensic findings in court. 1 Although forensic investigations have used photography for decades, interest has been recently increasing in 3D medical-imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI). 2,3 The obvious benefit of these modalities is the possibility of imaging a whole 3D anatomy from the inside of the subject. This lets investigators retrieve additional forensic information that's often not visible from the outside.
Minimally invasive therapy (MIT) is one of the most important trends in modern medicine. It inclu... more Minimally invasive therapy (MIT) is one of the most important trends in modern medicine. It includes a wide range of therapies in videoscopic surgery and interventional radiology and is performed through small incisions. It reduces hospital stay-time by allowing faster recovery and offers substantially improved cost-effectiveness for the hospital and the society. However, the introduction of MIT has also led to new problems. The manipulation of structures within the body through small incisions reduces dexterity and tactile feedback. It requires a different approach than conventional surgical procedures, since eye-hand co-ordination is not based on direct vision, but more predominantly on image guidance via endoscopes or radiological imaging modalities. ARIS*ER is a multidisciplinary consortium developing a new generation of decision support tools for MIT by augmenting visual and sensorial feedback. We will present tools based on novel concepts in visualization, robotics and haptics providing tailored solutions for a range of clinical applications. Examples from radio-frequency ablation of liver-tumors, laparoscopic liver surgery and minimally invasive cardiac surgery will be presented. Demonstrators were developed with the aim to provide a seamless workflow for the clinical user conducting image-guided therapy.
Applications in the fields of virtual and augmented reality as well as image-guided medical appli... more Applications in the fields of virtual and augmented reality as well as image-guided medical applications make use of a wide variety of hardware devices. Existing frameworks for interconnecting low-level devices and high-level application programs include high-frequency event streaming frameworks as well as dynamically typed message passing systems. Neither of these approaches exploits the full potential for processing events coming from arbitrary sources and neither is easily generalizable. In this paper, we will introduce a new multi-modal event processing methodology that uses dynamically-typed event attributes for event passing between multiple devices and systems. The existing OpenTracker framework was modified to incorporate a highly flexible and extensible event model, which can store data that is dynamically created and arbitrarily typed at runtime. The software architecture that is introduced in this work provides multi-modal event streaming, network and file logging support, and the ability to merge separate event streams. Due to the implementation of a self-descriptive event serialization mechanism, the architecture does not put any constraints on new data types to be used within the library. The main factors impacting the library's latency and throughput were determined experimentally and the overall performance was shown to be sufficient for most typical applications. Several sample applications were developed to take advantage of the new dynamic event model that is provided by the library, thereby demonstrating its flexibility and expressive power.
Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI '14, 2014
Content on computer screens is often inaccessible to users because it is hidden, e.g., occluded b... more Content on computer screens is often inaccessible to users because it is hidden, e.g., occluded by other windows, outside the viewport, or overlooked. In search tasks, the efficient retrieval of sought content is important. Current software, however, only provides limited support to visualize hidden occurrences and rarely supports search synchronization crossing application boundaries. To remedy this situation, we introduce two novel visualization methods to guide users to hidden content. Our first method generates awareness for occluded or out-ofviewport content using see-through visualization. For content that is either outside the screen's viewport or for data sources not opened at all, our second method shows off-screen indicators and an on-demand smart preview. To reduce the chances of overlooking content, we use visual links, i.e., visible edges, to connect the visible content or the visible representations of the hidden content. We show the validity of our methods in a user study, which demonstrates that our technique enables a faster localization of hidden content compared to traditional search functionality and thereby assists users in information retrieval tasks.
The First IEEE International Workshop Agumented Reality Toolkit,, 2002
We present a mobile augmented reality (AR) system to guide a user through an unfamiliar building ... more We present a mobile augmented reality (AR) system to guide a user through an unfamiliar building to a destination room. The system presents a world-registered wire frame model of the building labeled with directional information in a see-through heads-up display, and a three-dimensional world-in-miniature (WIM) map on a wrist-worn pad that also acts as an input device. Tracking is done using a combination of wall-mounted ARToolkit markers observed by a head-mounted camera, and an inertial tracker. To allow coverage of arbitrarily large areas with a limited set of markers, a structured marker re-use scheme based on graph coloring has been developed.
IEEE Transactions on Visualization and Computer Graphics, 2000
As heterogeneous data from different sources are being increasingly linked, it becomes difficult ... more As heterogeneous data from different sources are being increasingly linked, it becomes difficult for users to understand how the data are connected, to identify what means are suitable to analyze a given data set, or to find out how to proceed for a given analysis task. We target this challenge with a new model-driven design process that effectively codesigns aspects of data, view, analytics, and tasks. We achieve this by using the workflow of the analysis task as a trajectory through data, interactive views, and analytical processes. The benefits for the analysis session go well beyond the pure selection of appropriate data sets and range from providing orientation or even guidance along a preferred analysis path to a potential overall speedup, allowing data to be fetched ahead of time. We illustrate the design process for a biomedical use case that aims at determining a treatment plan for cancer patients from the visual analysis of a large, heterogeneous clinical data pool. As an example for how to apply the comprehensive design approach, we present Stack'n'flip, a sample implementation which tightly integrates visualizations of the actual data with a map of available data sets, views, and tasks, thus capturing and communicating the analytical workflow through the required data sets.
The Virtual Table presents stereoscopic graphics to a user in a workbench-like setting. For this ... more The Virtual Table presents stereoscopic graphics to a user in a workbench-like setting. For this device, a user interface and new interaction techniques have been developed based on transparent props -a tracked hand-held pen and a pad 12 . These props, particularly the pad, are augmented with 3D graphics from the Virtual Table's display that can serve as a s suitability for design tasks. Traditional CAD dialogue can be combined with intuitive rapid sketching of geometry on the pad. Additionally, the resulting events and objects can be associated with scene details below the translucent tablet.
EDT 2006: IEEE VR 2006 Workshop on Emerging Display Technologies March 26, 2006 10: 30–12: 00 Ses... more EDT 2006: IEEE VR 2006 Workshop on Emerging Display Technologies March 26, 2006 10: 30–12: 00 Session II Upponurkka: An Inexpensive Immersive Display for Public VR Installations............................ 13 T. Lokki, T. Ilmonen, W. Mäkelä, and T. Takala La Cueva Grande: a 43-Megapixel Immersive System................................................... 17 C. Canada, T. Harrington, R. Kares, D. Modl, L. Monroe, S. Stringer, and B. Tomlinson Display Research at the University of Southern California....................... ...
F orensic-case analysis relies heavily on digital information for documentation, especially to re... more F orensic-case analysis relies heavily on digital information for documentation, especially to reconstruct accident and crime scenes and to present forensic findings in court. 1 Although forensic investigations have used photography for decades, interest has been recently increasing in 3D medical-imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI). 2,3 The obvious benefit of these modalities is the possibility of imaging a whole 3D anatomy from the inside of the subject. This lets investigators retrieve additional forensic information that's often not visible from the outside.
Uploads
Papers by D. Schmalstieg