Virtual Assistant Task Guide

m
fro ls
0 2 3 ie
5, 2 d Dan
Assessing the Helpfulness
e m
2
ber Davi Sa
tur
of Responses
v y IP d
y , No 4.9 b 17 ay, N
da .25 2.1
tur 72.16 6.2 ovem
a
S 1 54 ber
.9
IP by 25,
Da 202
Task Instructions vid 3 f
Da rom
nie
ls
A Virtual Assistant is a service or mobile app that can understand queries from a user, and give
visual results, audio responses, or take actions on the user's behalf. The primary method of
interacting with a Virtual Assistant is to speak or type queries, listen to audio responses, and look
at or interact with results on the smartphone. Examples of Virtual Assistants available today are:
Google Assistant, a personal assistant service on Android devices, and Siri, a personal assistant
service on iOS devices.

In this task, you will see a query a user might speak or type to a Virtual Assistant, and a
corresponding response from the assistant. The response might include audio spoken by the
assistant, visual results provided by the assistant, and/or a text description of an action taken by
the assistant. Below the last response, you may see a suggestion chip: a word or phrase inside a
rectangle with rounded corners. Suggestion chips allow a user to issue a follow-up query to the
Assistant. These are not functional in the rating environment but would be for the user.

Your job is to evaluate the response from the assistant by rating its usefulness on the Needs Met
rating scale. More detail is given below.

Needs Met Rating

Please rate how well each response meets the needs of users according to the following scale.

Rating Category Explanation

m
Sa
t 3 fro ls
IP urFully
d All or almost all users would5,be 202 immediately
ie and fully
17 ay, N Meets (FullyM) 2 D an
2.1 r vi d
6.2 ovem satisfied by the result. Users e
mb y Da
would not need to see
54 ber v e
o .9 b need.
.9
by 25, other results to satisfy
, N 54their
Da 202 a y
rd 6.2
vid 3 f
ro
Da (HM) Satu 172.1
Highly Meets nie m Very helpful for
IP many or most users. Some users
ls may prefer to see other results.

Moderately Meets (MM) Helpful for many users OR very helpful for some.
Some or many users may prefer to see other results.

Slightly Meets (SM) Helpful for fewer users. There is a connection

between the query and the result, but the result is
unhelpful or unsatisfying in some way. Many or most
users would prefer to see other results.

Fails to Meet (FailsM) Completely fails to meet the needs of users. All or
almost all users would prefer to see other results.

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 Additionally, please review the sections below for examples of Answer and Action
rom
responses from a Virtual 2Assistant.
3 f ls 0 ie
2 5, 2 d Dan
ber Davi Sa
t
Query v e m y Response IP urday Rating Discussion
No 4.9 b 17
y , 5 2.1 , Nov
rda 6.2 6.2 em
[how tall is Satu 72.1 54 beFully Meets This response fully
1 .9 r
barack obama]IP by 25, answers the user's query.
Da 202
vid 3 f
Da rom
nie
ls

[SP500] Highly Meets to This response would

Fully Meets satisfy most users issuing
this query, but some users
might want additional
information.

[seattle Moderately Meets This is a broad query for a

washington] city. Different users may
be looking for different
things. The response
shows a card with
information about Seattle,
Washington. This
Sa f rom
t s
IP urday , 2 023 nielresponse may be helpful
17 25 a
2.1 , Nov er avid
D for some users.
6.2 em mb D
54 ber
.9 Nove .9 by
by 25, , 4
Da 202
vid 3 f ur day 16.25
t .
Da rom Sa 172
nie I P
ls

[who is the Slightly Meets Most users issuing this

president?] query want to know the
name of the current
president, not the
definition of the office.
This response would be
helpful for few users.

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 Query Response Rating Discussion
m
3 fro ls
[dogs] 0 2 ie Fails to Meet This is a broad
2 5, 2 d Dan informational query for an
ber Davi Sa
t
v e m y IP urday animal. Different users
No 4.9 b 17
y ,
da .25 2.1 , Nov may be looking for
tur 72.16 6.2 em
a
S 1 54 ber different things. This
.9
IP by 25, response shows
Da 202
vid 3 f visit-in-person information
Da rom
nie for three different dog
ls
care providers near the
user location. However,
the query is very broad
and it is unlikely users
want to go anywhere in
person.

Query Response Rating Discussion

[play lady gaga poker face Action Response: Play Media Fully Meets The device plays the requested song
on spotify] Artist: Lady Gaga from the correct source. This Fully
Meets the user’s need.
Song: Poker Face
Content URL: URL

[play all along the watch Action Response: Play Media Highly Meets The device played the song by Jimi
tower] Artist: Jimi Hendrix Hendrix, which would satisfy many
users. Some users might want a
Song: All Along the Watch
different version of the song (for
Tower
example, by Bob Dylan).
Content URL: URL

[play jazz] Action Response: Play Media Moderately Meets to The device plays a jazz song, which
Sa Artist: Miles Davis Slightly Meets om
would befrsatisfying to some users.
t 3 ls
IP urday Most 0 2
2 users i e
n would prefer a jazz playlist
17
2.1 , Nov Song: Kind of Blue
r 25, id Da
6.2 em m Dbe to abe
v played.
54 ber
.9 Nove .9 by
by 25, , 4
Da 202
vid 3 f ur day 16.25
t .
Da romAction Response: Play Video Sa 172
[play thriller music video] nie Slightly Meets
I P The audio-only version of this song
ls
Artist: Michael Jackson was played. Although some users
Song: Thriller could be satisfied by this audio-only
Content URL: URL version, most users would prefer a
music video like this one to be played
on their mobile device.

[play mumford and sons Action Response: Set a Fails to Meet The user wanted to play a specific
reminder] Reminder Time: Please specify song, and the device instead set a
reminder. No users would be satisfied
a time
with this response.

Note: You may also see a side-by-side rating question. In this case, please rate which side better meets the user
need. If you can't hear the words spoken in an audio query, please use your best judgment to give a rating.

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.