Category Maturity Scorecard - Testing Group::Accessibility Category - FY21-Q1 - Obtaining an accessibility report from an MR
-
Research issue: ux-research#807 (closed)
-
Walkthrough:
{{add link to YouTube video or walkthrough document}}TBD -
Recommendations:
{{add link to your recommendation issue/s}}TBD
Category Maturity Scorecard Checklist
Learn more about Category Maturity Scorecards
-
Review the Category Maturity Scorecard handbook page and follow the process as described. Reach out to the UX Researcher for your stage if you have questions. -
Document the results of each participant's session using the CM Scorecard Results Template -
Add Zoom links for each participants' session recordings. These can be found in the Recordings section of the Zoom web UI. -
If the participant has not granted permission to share the recording publicly, ensure the sharing settings are set to GitLab-only. -
Summarize the results and give the final grade using this template. -
[Create a recommendation issue](https://gitlab.com/gitlab-org/gitlab-design/issues/new? issuable_template=UX%20Scorecard%20Part%202) for these sessions.
Summary
This is an issue for tracking, discussing, and identifying the Primary and Secondary JTBDs within the Accessibility Category in the Testing Group.
These JTBD will be used to judge the quality of experiences within the Accessibility Category after work is done to fulfill the initial category opportunity canvas.
CM Scorecard Results Template
Jobs to be done
JTBD for this scorecard
| JTBD | Scenario | Persona | Epic to address |
|---|---|---|---|
| When I am making a change to a project, I want to see how those changes in the MR impacted a11y of the project, so that I can ensure a11y stays/improves quickly. | You made a change that impacted a11y of the project. Investigate the change and how you would find information to resolve a degradation before merging. | - Sasha (Software Developer) | Actionable a11y data in every Merge Request |
JTBD for future scorecards
| JTBD | Scenario | Persona | Epic to address |
|---|---|---|---|
| When I want to see the overall a11y of my projects, I want to have as much data as possible about the issues in the latest build, so that I can write up quality / actionable issues for the development team to fix. | You are responsible for testing a11y of a project and writing issues for problems. Investigate the issues of the current project and write up an issue for the team to make an improvement. | - Sasha (Software Developer){at Small Orgs} - Parker (Product Manager){at Small Orgs} - QA Engineer |
|
| When I am feature planning, I want to see the trend of a11y of a project, so that I can make informed tradeoffs about non a11y features and a11y improvements. | You are planning the next milestone / iteration for a project. Find information about how accessible your project is and if you need to work on improving a11y. | - Parker (Product Manager) - Delaney (Development Team Lead){Certain Orgs} |
|
| When I am feature planning, I want to see the trend of a11y of a project, so that I can make informed tradeoffs about non a11y features and a11y improvements. | You are planning the next milestone / iteration for a project. Find actionable a11y issues that can be addressed to write as issues. | - Parker (Product Manager) - Delaney (Development Team Lead){Certain Orgs} |
|
| When I am strategy planning, I want to see the trend in overall a11y of my teams projects, so that I can make informed tradeoffs about investments in a11y improvements OR other items. | You are responsible for P&L of multiple projects and the board is making a11y of the apps/site a priority. Make a decision about if a11y should be an area of investment and why. | * Parker (Product Manager) |
Script
Scenarios
Scenario 1
In this scenario, you're a developer working on a static website. Let's assume that you just opened a Merge Request for a small change you just did to that website.
Now you want to check the accessibility of this website up to the current state of your feature branch. Your job is to find the accessibility job, download the artifact, and obtain the number of accessibility problems that the website currently has.
** Number of Meaningful Clicks for this scenario **: 6 (via exposed artifacts) / 8 (via jobs)
Notes:
- You don't need to create the MR. There's already an MR in the project. You can assume that's your MR.
- This scenario is based on the MVC of this feature that checks the whole state of a page. Accessibility for diffs it's an upcoming feature but is not part of this scenario.
- Assume that the jobs and CI are correctly configured and giving you access to the website as seen from your branch. No need to dig into the configuration details.
- We will provide you a link to the project. Please share your screen while you complete the scenario.
Participant: Sourav
Background
Sourav is the Head of Engineering at a small startup in Bangalore, India. He had used GitLab at other companies, but his company had been using GitLab for the past 2 months.
Video
https://youtu.be/vg_EVJQlo3I
Scenario JTBD 1 (Accessibility Information Retrieval)
- Ease of completion (post-task / observed): Somewhat easy
- Task completion confidence: Very confident
- Meaningful clicks: 6 clicks
- Heuristic evaluation (How well did the participant complete the scenario?): Successfully
Other notes
The participant was also asked to attempt a different user path. He struggled to figure out this path but ultimately was successful (Successfully, but with some unnecessary steps or difficulty)
Participant: Vincent
Background
Frontend Engineer at Inrupt, doing Developer Tooling for customers to write applications against the Solid standard.
Video
https://youtu.be/-6WOvt-3-Rc
Scenario JTBD 1 (Accessibility Information Retrieval)
- Ease of completion (post-task / observed): Very easy
- Task completion confidence: Very confident
- Meaningful clicks: 9 clicks (via jobs)
- Heuristic evaluation (How well did the participant complete the scenario?): Successfully
Other notes
The participant was also asked to attempt a different user path. He was able to detect the alternative user path correctly and complete the steps successfully.
Participant: Fred
Background
Lead Programmer for PMCTire, an E-Commerce Website in Canada that sells tires.
Video
https://youtu.be/XD1q1GIIH_c
Scenario JTBD 1 (Accessibility Information Retrieval)
- Ease of completion (post-task / observed): Not difficult or easy
- Task completion confidence: Not confident
- Meaningful clicks: +15 clicks (via jobs)
- Heuristic evaluation (How well did the participant complete the scenario?): Successfully, but with some unnecessary steps or difficulty
Other notes
The participant was also asked to attempt a different user path. He approached this new path with low-confidence but was able to successfully complete it, especially since he had previous knowledge of the first task. The participant verbalized some of his doubts while performing the task. (Successfully, but with some unnecessary steps or difficulty)
Participant: Lorin
Background
Founder at BarCraftUnited, an eSports Event Management Platform. He mostly deals with frontend tasks but his responsibilities are all across the stack and supporting all the technical people in the company.
Video
https://youtu.be/3_HMh_u6hMI
Scenario JTBD 1 (Accessibility Information Retrieval)
- Ease of completion (post-task / observed): Somewhat easy
- Task completion confidence: Very confident
- Meaningful clicks: 9 clicks (via exposed artifacts)
- Heuristic evaluation (How well did the participant complete the scenario?): Successfully
Other notes
The participant verbalized his reasoning which included the process of reviewing the MR. This added extra clicks, but they weren't necessarily meaningful towards the main scenario. The participant was also asked to attempt a different user path. He was able to detect the right alternative path but couldn' successfully complete it due to low visibility/salience of the call to actions. (Unsuccessful)
Participant: Nathan
Background
Nathan is the founder of a company that creates software solutions for Educational institutions. He is responsible of managing developers and designers. He is deeply passionate about accessibility, particularly because it's crucial in his space.
Video
https://youtu.be/TuL94O-_adE
Scenario JTBD 1 (Accessibility Information Retrieval)
- Ease of completion (post-task / observed): Very easy
- Task completion confidence: Very confident
- Meaningful clicks: 7 clicks (via exposed artifacts)
- Heuristic evaluation (How well did the participant complete the scenario?): Successfully
Other notes
The participant verbalized his reasoning during the walkthrough. He was capable of achieving the proposed flow and also the alternative flow. He was very knowledgeable of the underlying artifact retrieval mechanics.
Overall Grade:
-
4/5 Participants (80%) completed the scenarios with no more than 20% clicks above the minimum
-
4/5 Participants (80%) completed the scenario with "absolute confidence" (observed and asked).
-
5/5 Participants (100%) completed at least one of the possible scenario flows successfully.
Based on the observed results and the existing grading rubric this study meets the overall expectations of the original MVCs and features proposals of this category. In general, all participants were able to retrieve the artifacts without much trouble.
We are aware that discoverability, in general, can be improved but this explicit flow and scenario were completed successfully by 100% of the participant, even in the absence of particular accessibility knowledge. This means that although the feature provides accessibility data, most customers can rely on it and obtain useful information without previous domain knowledge and using their existing GitLab understanding.
B (Meets Expectations)
This grade represents the average of the ratings for all participants based on the grading rubric