Skip to content

Mas i1001 locationtest#1375

Merged
martinsumner merged 5 commits into
develop-3.0from
mas-i1001-locationtest
Jun 12, 2023
Merged

Mas i1001 locationtest#1375
martinsumner merged 5 commits into
develop-3.0from
mas-i1001-locationtest

Conversation

@martinsumner

Copy link
Copy Markdown

Test v4 claim

There were some issues with the original test:

- Config changes to riak_core where put under default bucket properties, and so were not applied.  In particular, although the test was run 3 times on different ring sizes, only the default ring size (64) was actually tested.

- In testing the genuine ring sizes, it was uncovered that the Ring9 test will fail on ring sizes other than 64.

- As the test did not wait for transfers, they were not a genuine system test, and there was potential for false success and false failure due to this.  The test now waits for transfers to complete; but does so in a fairly cumbersome way as `rt:wait_until_transfers_complete/1` is not a reliable function (in particular when it is called before any transfers have started.

- After adding the wait for transfers, and correcting the vnode_inactivity_timeout, it was uncovered that there exists a race condition whereby a vnode may be simultaneously be exiting (triggered by a join) and be required (to receive a handoff).  This could lead to transfers that got permanently stuck due to `Transfers: [{stopped,'[email protected]',6}]`.  A timeout of 60s is sufficient to make the test relatively reliable, but significantly extends the time required to run the test.
@martinsumner
martinsumner merged commit ad245b9 into develop-3.0 Jun 12, 2023
@martinsumner
martinsumner deleted the mas-i1001-locationtest branch June 12, 2023 12:07
martinsumner added a commit that referenced this pull request Jun 22, 2023
* Test reliability fix

There were some issues with the original test:

- Config changes to riak_core where put under default bucket properties, and so were not applied.  In particular, although the test was run 3 times on different ring sizes, only the default ring size (64) was actually tested.

- In testing the genuine ring sizes, it was uncovered that the Ring9 test will fail on ring sizes other than 64.

- As the test did not wait for transfers, they were not a genuine system test, and there was potential for false success and false failure due to this.  The test now waits for transfers to complete; but does so in a fairly cumbersome way as `rt:wait_until_transfers_complete/1` is not a reliable function (in particular when it is called before any transfers have started.

- After adding the wait for transfers, and correcting the vnode_inactivity_timeout, it was uncovered that there exists a race condition whereby a vnode may be simultaneously be exiting (triggered by a join) and be required (to receive a handoff).  This could lead to transfers that got permanently stuck due to `Transfers: [{stopped,'[email protected]',6}]`.  A timeout of 60s is sufficient to make the test relatively reliable, but significantly extends the time required to run the test.

* Update location.erl

* Add leave test

* Update leave test

* Update core_all
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant