Ceph » rgw

The transactions that are associated with object creation and deletion within versioned buckets involve OLH objects, instance objects, and several types of index entries. It is intrinsically possible that one or more of these resources is left in an inconsistent state when a transaction terminates abnormally. Recent work has attempted to fix known, easily reproducible error modes that caused these sorts of issues and to add rollback logic so that the inconsistencies are normally cleaned up immediately on request threads [1]. Even so, it is important to acknowledge that there is no existing mechanism that can ensure that the index and associated objects are eventually consistent when admitting the possibility of unidentified bugs or fault scenarios (e.g. the process dies in the middle of a transaction and there is no opportunity to rollback).

For clusters that have systematically reproduced the types of bugs/conditions that evoke the aforementioned inconsistencies (typically due to particular patterns of client behavior), there can be significant consequences. Most obviously, extraneous objects and index entries can cause bloat in space consumption. Secondarily, the nature of the bogus index entries that are left behind in these scenarios gives rise to severe performance degradation for bucket listings.

This intent of this issue is to propose new radosgw-admin commands for the purpose of identifying and cleaning up the aforementioned inconsistencies with versioned bucket indexes and associated objects.

[1] https://github.com/ceph/ceph/pull/51700