In T342930, we ran an A/B test of Reference Check that demonstrated it was effective at causing:
- Newcomers to publish new content edits that include references while lowering the likelihood those edits would be reverted
- Newcomers to be more likely to return to edit again
...all the while NOT causing degradations in other metrics like, block rate, edit completion rate, etc.
This task involves the work of running another A/B test (or potentially an A/B/C test) of the Reference Check with one key change: removing the constraint on how many Reference Checks people have the potential to see within a single edit.
Decision to be made
This A/B test will help us make the following decision: What – if any – changes in the Reference Check UX will we make to ensure people seeing multiple Checks within a single edit continue to experience the benefits the first iteration of the feature caused ?
Hypotheses
| ID | Hypothesis | Metric(s) for evaluation |
|---|---|---|
| KPI | The quality of new content edits newcomers and Junior Contributors make in the main namespace will increase because a greater percentage of these edits will include a reference or an explicit acknowledgement as to why these edits lack references. | 1) Proportion of published edits that add new content and include a reference or explicit acknowledgement of why a citation was not added, 2) Proportion of published edits that add new content (T333714) and are reverted within 48 hours (or have a high revision risk score) if we use revision risk model (T317700, T343938)) |
| Curiosity #1 | New account holders will be more likely to publish an unreverted edit to the main namespace within 24 hours of creating an account because they will be made aware of the need to accompany new text they're attempting to publish with a reference, when they don't first think/know to do so themselves | Constructive activation |
| Curiosity #2 | Newcomers and Junior Contributors will be more aware of the need to add a reference when contributing new content because the visual editor will prompt them to do so in cases where they have not done so themselves. | Increase in the proportion of newcomers and Junior Contributors that publish at least one new content edit that includes a reference. |
| Curiosity #3 | Newcomers and Junior Contributors will be more likely to return to publish a new content edit in the future that includes a reference because Edit Check will have caused them to realize references are required when contributing new content to Wikipedia. | 1) Proportion of newcomers and Junior Contributors that publish an edit Edit Check was activated within and successfully and return to make an unreverted edit to a main namespace within 7 and 30 days., 2) Proportion of newcomers and Junior Contributors that publish an edit Edit Check was activated within and return to make a new content edit with a reference to a main namespace within 7 and 30 days. |
Leading indicators
See T388731.
Guardrails
This section describes the metrics we will use to make sure other important parts/dimensions of the "editing ecosystem" are not being negatively impacted by people being able to see Multiple Reference Checks in a single edit. The scenarios named in the chart below emerged through T325851.
| ID | Name | Metric(s) for Evaluation |
|---|---|---|
| 1) | Edit quality decrease (T317700) | Proportion of published edits that add new content and are still reverted within 48hours (or have a low revision risk score if we use the revision risk model (T317700)). Will include a breakdown of revert rate of published edits with and without a reference added. |
| 2) | Edit completion rate drastically decreases | Proportion of edits that are started (event.action = init) and are successfully published (event.action = saveSuccess) |
| 3) | Edit abandonment rate drastically increases | Proportion of contributors that are presented Edit Check feedback and abandon their edits (indicated by event.action = abort and event.abort_type = abandon). |
| 4) | People shown Edit Check are blocked at higher rates | Proportion of contributors blocked after publishing an edit where Edit Check was shown |
| 5) | High false positive rate | Proportion of contributors that dismiss adding a citation and select "I didn't add new information" or other indicator that their edit doesn't require a citation |
A/B Test: Decision Matrix
| ID | Scenario | Indicator(s) | Plan of Action |
|---|---|---|---|
| 1) | Reference Check is disrupting, discouraging, or otherwise getting in the way of volunteers. Read: people are less likely to publish the edits they start. | Significant drop in edit completion and spike in edit abandonment in edit sessions where Reference Check is activated. | Pause scaling plans; investigate changes to UX |
| 2) | Reference Check is increasing the likelihood that people will publish destructive edits | Increase in proportion of contributors blocked after publishing an edit where Reference Check is activated, Increase in proportion of published edits where Reference Check was activated and are reverted within 48 hours relative to new content edits Reference Check was NOT activated within. | Pause scaling plans, review edits to try to identify pattern in abuse and propose changes to UX to mitigate them |
| 3) | Reference Check is causing people to publish edits that align with project policies | Increase in the proportion of edits Reference Check was activated within that include a reference and are not reverted within 48 hours relative to new content edits without a reference Reference Check was NOT activated within | Move forward with scaling plans |
| 4) | Reference Check is effective at causing people to accompany new content edits that include a reference, but those references are unreliable | Increase in the proportion of published edits Reference Check was activated within that include a reference and increase or no change in the proportion of these edits that are reverted within 48 hours | Block scaling plans on reference reliability work (T276857) |
| 5) | Reference Check is not effective at causing people to accompany new content edits that include a reference but is not disrupting to volunteers. | No change or decrease in the proportion of published edits Reference Check was activated within that include reference and A) no significant drop in edit completion or abandonment rate or B) no significant spike in block or revert rate | Move forward with scaling plans |
WARNING: For each metric named above, we need to be able to filter them by the number of Reference Checks shown within a given edit.