FY20-Q2 Quality Department OKR: Make self-managed mission critical => 94%

Certify 10K environment, improve defect SLA visibility and detection, generate report on missed SLAs, add 2 metrics against bugs with customer label for every group's dashboard.

• Key result: Roll-out triage training recording to ensure bugs with customer label are triaged in a timely manner. https://gitlab.com/gitlab-org/quality/team-tasks/issues/148 => 85%
• Key result: Triage all existing customer ~S1 ~S2 bug and have them scheduled with a milestone. gitlab-org/quality/triage-ops#177 (closed) => 100% Results https://gitlab.com/gitlab-org/gitlab-ce/issues/63136 still waiting for PM on half of the issues.
• Key result: Design and implement customer defect SLO detection process. gitlab-org/quality/triage-ops#137 (closed) => 100%
• Key result: Enablement: Conduct testing on support team's 10k user reference env with customer reference traffic load. gitlab-org/quality/performance#3 (closed) => 100%
  • Follow-up task: Investigate four slow api endpoints discovered during load testing https://gitlab.com/gitlab-org/gitlab-ce/issues/64335
• Key result Enablement: Publish guidelines for customers. gitlab-org/quality/performance#14 (closed) => 100%

Retrospective

Good

• We completed 10K reference architecture testing.

Bad

• Test planning started a bit late into the quarter which made us sprinted to catchup.

Try

• Break down Q3 OKRs into sub-issues earlier.
• Sync AMA session has a high cost to setup. Try a small recording and open an issue for feedback and discussion after posting the recording.
• Build a set of performance baseline so that performance testing is well-documented and has clear pass/fail criterias.