Human-Computer Interaction
Track A: Direct manipulation and graphical interaction
Clemens Nylandsted Klokmose – September 27th 2021
Plan for the day
• Direct Manipulation
• Direct manipulation and immersive environments
• (2D) Graphical interaction
Learning goals
• Understand
• What direct manipulation is
• Different levels of directness in direct manipulation
• Direct manipulation in immersive environments
• What input tasks in 2D interfaces are, and what types there are
• What modal interfaces are
• Be able to …
• Describe the level of directness in a direct manipulation interface
• Describe the input tasks in a a user interface
• Describe modes in an interface
Direct Manipulation
Shneiderman 1983
• Continuous representation of the objects of interest
• Physical actions instead of complex syntax
• Rapid, incremental, reversible actions whose effects
on the objects of interest are immediately visible
• Layered approach to discover the interface progressively, so that it can
be used with little previous knowledge
Degrees of directness
Translational distance
Weak
Medium
Strong
Immersive
Direct Manipulation in
Immersive environments
Virtual reality
“The presence aspect of virtual reality
breaks the physical limitations of space
and allows users to act as though they
are somewhere else”
Virtual reality
• Head-tracking
• Outside in
• Inside out
• Hand-tracking
• Controllers (gyro, accelerometer, camera)
• Finger tracking (camera and/or gloves)
Augmented Reality
Lets users see the world with a digital overlay.
Immersive
- E.g., Hololens 2
Handheld
- E.g., Smartphone ortablet
Based on cameras, depth cameras, LIDAR,
gyroscopes, accelerometers
Mixed reality
Milgram & Kishino (1994)
Tangible computing
Physical objects are used to
represent digital objects.
Position and rotation, e.g., on an
interactive table is tracked and
mapped to digital properties.
Digital content overlaid adjacent to
or through projection mapping
Tangible computing
• Can make interaction more direct
• Also can add another layer of
indirection if not done well
• Difficult to make general purpose
• Often used for specific installations
• Based on cameras,
markers, RFID
(2D) Graphical interaction
Input and feedback
Pointing
Selection (single/multiple click, with/without modifier)
Input and feedback
Drag
“ghosts” or
shadows
ink
http://jamesfriend.com.au/pce-js/
Basic interaction tasks
• Input tasks
• Selection tasks
• Trigger tasks
• Navigation tasks
• Property setting tasks
• Transformation tasks
Input tasks
Input text
Text field + keyboard
Input a single value
Slider
Input position
Pointing
Input a path
Sampling (feedback = ink)
Input task
Adapt input to the value type
Example: size of a table
Selection tasks
Choose one or several items within a set
Fixed- or variable size set
Small or large set
Fixed-size set
Menu, checkboxes, radio buttons
Variable-size set
Pointing, list, text input
Selection task
Multiple selection
by group by interval
add/remove
Combine techniques
Point elements in list / use keyboard to input prefix
Selection tasks
Linear menus
Menu bar +
pulldown menus
Fixed palette
Pop-up (or contextual) menus
Hierarchical menus
Pull-off menus
Selection tasks
Radial menus
Pie Menus (Hopkins)
Selection is faster in a radial menu than in a linear menu
But radial menu limited (in practice) to 8 items
Marking menus (Kurtenbach)
Natural transition from novice to expert
perform the selection gesture faster – the menu does not display
Trigger tasks
Buttons and Menus
Drag and drop
Action depends on both the source and destination
Gestural input
Specifies the object and command simultaneously
Trigger tasks
Gestural input
Navigation tasks
Scrollbars
Direction of scrolling
Division of attention
Direct scrolling
Move with the hand
Automatic scrolling
Zooming
Zooming in to see detail
Zooming out to see context
Property and parameter specification tasks
Dialog boxes
Fields + OK / Apply / Cancel buttons
Modal or non-modal
Temporal and spatial decoupling between the specification of the command, its parameters, and
its execution
Optional Parts Tabs
Problem with OK button!
Property and parameter specification tasks
Inspectors
Inspector is always displayed
Content of inspector changes according to selected item
Editing properties immediately affects the selected items
Transformation tasks
Manipulation handles
Modes
• Mode = state of the interface where user actions are interpreted consistently, and differently than in other
modes
• Problems: Mode visibility, mode change
• Examples
• “vi” text editor :
• Type “Esc” to switch between input/command modes
• “emacs” text editor :
• User Control and Meta modifiers to enter commands
• Tool palettes:
• Select a tool to activate it
Modes
• Temporal modes
• The same user action performed at different times has different effects
• Problem: Initiative of the mode change
• Spatial modes
• The same user action performed at different locations has different effects
• Quasi-modes
• Temporal modes activated as long as a physical action is sustained
An interface is a collection of modes !
Mode exercise
• Go to https://bellard.org/jslinux/vm.html?url=alpine-
x86.cfg&mem=192
• Open a file in vi with vi vi.txt, write Hello from vi! in it, save and
exit
• Open a file in emacs with emacs emacs.txt write Hello from
emacs! in it, save and exit
• Feel free to use google
What do I expect you to have learned?
• Understand
• What direct manipulation is
• Different levels of directness in direct manipulation
• Direct manipulation in immersive environments
• What input tasks in 2D interfaces are, and what types there are
• What modal interfaces are
• Be able to …
• Describe the level of directness in a direct manipulation interface
• Describe the input tasks in a a user interface
• Describe modes in an interface
