REVIEW ON IHRC :

AN AR-BASED
INTERFACE FOR
INTUITIVE,
INTERACTIVE AND
COORDINATED TASK
SHARING BETWEEN
HUMANS AND
ROBOTS IN BUILDING
CONSTRUCTION
AGENDA

• Introduction
• Background
• Methodological Understanding
• Evaluation in AEC
• Instructive Human Robot Collaboration
• Details
• Contribution and Relevance
• Q&A

2
 PUBLICATION AT:
HTTPS://WWW.RESEARCHGATE.NET/PUBLICATIO
N/355809540

CONFERENCE PAPER:
NOVEMBER 2021

CITATIONS:
30

JOURNAL:
ISARC. PROCEEDINGS OF THE INTERNATIONAL
SYMPOSIUM ON AUTOMATION AND ROBOTICS IN
CONSTRUCTION

VOLUME:
38

PAGES:
25-32

PUBLISHER:
IAARC PUBLICATIONS
AUTHORS:
FELIX AMTSBERG
INSTITUTE FOR COMPUTATIONAL DESIGN
UNIVERSITÄT STUTTGART
34 PUBLICATIONS, 179 CITATIONS

XILIU YANG
INSTITUTE FOR COMPUTATIONAL DESIGN AND CONSTR
UCTION
UNIVERSITÄT STUTTGART
14 PUBLICATIONS, 82 CITATIONS

LIOR SKOURY
INSTITUTE FOR COMPUTATIONAL DESIGN
UNIVERSITÄT STUTTGART
PUBLICATIONS, 83 CITATIONS

HANS JAKOB WAGNER

43 PUBLICATIONS, 491 CITATIONS
INSTITUTE FOR COMPUTATIONAL DESIGN AND CONSTR
UCTION
UNIVERSITÄT STUTTGART
FELIX AMTSBERG

UNIVERSITY STUTTGART
Bio: Felix Amtsberg is Postdoctoral research
associate at the Institute for Computational
Design & Construction and the Cluster of
Excellence IntCDC, where he is coordinating
the timber prefabrication platform and leads
the research for collaboration.
He earned his PhD with distinction in 2016
under the supervision of Prof. Stefan Peters
and Prof. Martin Bechthold (Harvard GSD).
Between 2016 and 2019 Felix was working as
a postdoc at Singapore University of
Technology & Design and MIT investigating
new combinations of unprocessed natural
construction materials and digital fabrication.
Research interests:
Novel prefabrication methods of load carrying
structures, made from natural grown resources
INTRODUCTION:
•The paper introduces the concept of Instructive Human Robot
Collaboration (iHRC), specifically designed for project-based
construction industry. The research highlights a case in timber
prefabrication, where humans and robots work together using
AR tools to handle complex tasks with a significant increase in
flexibility and robustness. The system’s interface, Vizor, allows
easy communication between workers and robots by offering a
digital twin of tasks and giving real-time feedback and control.

•A case study was conducted using a 14-axis robotic fabrication

setup in which humans and robots jointly assembled timber
structures. The study demonstrated the use of AR head-
mounted displays (HMDs) to facilitate human intervention in
automated tasks particularly in addressing issues like material
inconsistencies or robotic path errors. The system's flexibility
6
allows for a dynamic reallocation of tasks between human and
BACKGROUND:
Nowadays, especially the timber-based
manufacturers are investing in
prefabrication lines with a high Level of
Automation (LoA) for a higher accuracy
and process reliability which previously
relied on the execution of human labor
while such solutions are often not
sufficient to solve the demand for the
housing market, especially in the inner
cities, where various onsite conditions are
requested, to fill the vacant spaces or
extend the building vertically by adding
new stories on the existing ones.
Compared to the whole market only 13 %
of the upcoming projects fall under the
concept of modular building solutions and
hence to address this challenge, more
flexible automation approaches are
METHODOLOGICAL UNDERSTANDING:
• The research employs humans and robots work alongside each
other as separate units executing pre-determined task while
Human as an instructor, robot as an augmenting tool involving
software development, system integration and a real-world case
study to test the feasibility of iHRC. A key part of the methodology
is the task-skill analysis framework where tasks are divided
between humans and robots based on their respective skill sets.
The researchers used AR to create a seamless interface between
the two where workers can interact with robots in real-time
without technical expertise in robot programming.

• The approach integrates KUKA industrial robots, a Hololens AR

headset, and a server-based communication system. The AR
8
interface enhances human control by offering a digital twin of both
the work object and robot movements. The dynamic task-sharing
EVALUATION IN AEC:

AR in AEC research:
Shared model visualization was introduced to enhance design
communication and collaboration of hidden parts like water or
electric conduits in facility management.

AR-enhanced craft:
Visually guided pick and place of bricks and manual bending of
structural elements (i.e. steel tubes) to erect structures were
successful case for it.

AR-based human-machine collaboration:

Integration
9
of human labor and industrial robots to join their
individual strength and skill sets for smarter and more flexible
CONTD..

Humans and robots as separate units:

During prefabrication of steel structures, i.e. I-Beam, H-Beam etc.,
the skilled labor screws and robotic welding takes place to fix the
workpiece while human and robot do not share the same
workspace, i.e. material supply, workpiece unloading etc.

Human as an instructor and robot as an augmenting tool:

Sensors are used to identify and localize the human intervention
Convert the instructions to actions executed by the machine
For example, a Kinect RGB-D camera attached to the end effector
of an industrial robot was used for the project “Iterative clay
forming” to localize the finger
10
CONTD..
Human and robot as collaborative units:
Low payload industrial robots (e.g KUKA iiwa, Universal Robots)
were developed for low-risk hand in collaboration between human
and robot.

Instructive Human robot collaboration and Multi-Unit Task

Sharing
A novel method for the integration of humans and High LoA
technology in a collaborative fabrication setup combining their
individual strengths to extend skill set based fabrication planning

From Task-skill analysis to Multi-Unit Task-sharing

The fabrication tasks are distributed based on the individual skill
set of these units. If the skill set required by a task is satisfied by
more than one unit, the task can be flexibly redistributed among
11
these units depending on other criteria.
INSTRUCTIVE HUMAN ROBOT COLLABORATION:

With instructive Human Robot Collaboration (iHRC) this paper introduces an

additional layer to the existing levels of collaboration. This extends the
concept of sequential collaboration, enabling the craftsmen to instruct the
automated fabrication units e.g. industrial robots and a communication
protocol is proposed, which connects the machine controllers with the human
craftsmen via AR-technology. The developed Hololens interface “Vizor” is
used as the central facilitator and doesn’t request any programming skills
from its user. Providing a digital twin of the workpiece, it further extends the
users skill in terms of positioning accuracy. This enables a craftsman-
controlled fabrication process, breaking free from the constraints of a linear
predefined workflow. Tasks can be taken over, corrected or reassigned,
depending on various conditions, such as the unit’s skill set (tool, reach,
payload, etc.), resulting in a dynamically adaptive fabrication process and an
12
extremely flexible fabrication system.
DETAILS:
Communication:
• A server that dispatches tasks, two KUKA robot controllers
Setup: connected to the server by KukaVarProxy via TCP/IP
connection using the Python KukaVarProxy library.
A communication • According to the specific task, the server knows which
strategy and an data it needs to send for the task execution.
interface for the HMD • For the robot to know what to do with the received
were developed, to information, a unique KRL code is used.
connect the human • In this KRL some global variables are defined at the
being and beginning of the program.
communicate the • Then an infinite loop is triggered, and according to the
received data, it causes a switch case to execute the
fabrication data to desired task.
the units for • A Hololens Version 1 HMD connected to the server by
execution rosbridge via a WebSocket connection.
• The information related to each task like description,
13
deadline, and geometric information are sent to the
human worker as the server dispatches them.
 Display of the Digital Twins:
CONTD… A digital twin of the fabrication setup is built within
Grasshopper using the plugin Virtual Robot and the
plugin developed within this research Vizor to support
simulation of the fabrication process. The digital twin
shows the virtual representation of the model. It shows
where to place the timber slat for the robot to pick it
AR Interface Vizor: and its final position also containing the information if a
screw or nail was used, for disassembly and recycling.
Developed application “Vizor“ facilitates Digital twin (robot):
bi-directional communication between
The digital twin builds the virtual representation of the
multiple fabrication units. The High LoA system showing the execution of a specific
Grasshopper plugin allows users to task. It also keeps the craftsman informed of the robot’s
next movement even for an operator unfamiliar with
prototype workflows directly using GH pre-programmed robot paths.
components. The companion HoloLens
Task window:
application is built using Unity. It
The Task window provides information on the
includes a digital twin of the TIM setup fabrication process which task is currently executed or if
as well as modular user interface the system is paused for the craftsman to execute. A
components that let a user interact with clock shows the runtime of the current job.
tasks, markers, model geometry and Task list:
system status. The information is The Task list gives the human control over the
provided and visually presented in four upcoming task and their executing unit. Whenever the
human decides to take over a specific task, they can
components on the Head Mounted interfere and change the fabrication sequence and
14
Display, in this case a Hololens assign a specific task to themselves.
PICK AND PLACE PROCEDURE EXECUTED BY ROBOT
TASKS AT DIFFERENT SKILL SET BY A HOLOLENS-EQUIPPED HUMAN

15
CONTRIBUTION AND RELEVANCE:

• This research is especially relevant in today’s world where

automation in industries like construction is rapidly advancing.
The study shows how AR and robotics can work together to
improve productivity, quality, and flexibility in timber
prefabrication. It makes a significant contribution to the fields of
robotics, construction automation, and human-robot interaction.
•This research aligns well with smart systems and collaboration
between human and robotic workers are seen as key to improving
efficiency and reducing costs in construction. The case study's
successful implementation of the proposed system provides a
strong foundation for further research and development in AR-
enhanced human-robot collaboration.
16
IMPROVEMENT AREA:

• Software Development: Only 1 task happens at a time

which will allow the human to pause a running sequence,
intervene in the process (e.g. update target position, correct
errors), and then continue fabrication.
• Skill set descriptions are required of more complex
fabrication setups and of humans for the coordination of
several human and robotic units for fabrication sequences
with higher complexity.
• The developed interface can be applied to tasks requiring
hybrid human and robotic labor to execute, though it has
certain limitations at the current stage with untrained labor
for validation and improvement.
•17The AR-device Hololens 1 is suitable for applications in the
controlled research environment only.
 Q&A

18
THANK YOU
Md Shahadat Hossain
409-812-8049
[email protected]
Under Supervision of:
Dr. Wenhao Yang