[xextension/storage] Add Walker interface for storage extensions#15190
Conversation
Introduce Walker interface, WalkFunc type, and SkipAll error value to support iterating over storage key/value entries. This enables consumers to implement storage migrations or any functionality that involves ranging through the storage entries. One of the examples of such functionality may be a TTL-based garbage collection. Values can have a TTL marker, a routine runs with an interval iterating through entries searching for expired entries. WalkFunc produces Delete operations for each expired entry.
a90a5fb to
ec99e0d
Compare
Merging this PR will not alter performance
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #15190 +/- ##
==========================================
- Coverage 91.32% 91.14% -0.18%
==========================================
Files 700 701 +1
Lines 45316 45798 +482
==========================================
+ Hits 41383 41742 +359
- Misses 2782 2843 +61
- Partials 1151 1213 +62 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| // SkipAll is used as a return value from WalkFunc to indicate that | ||
| // all remaining storage entries are to be skipped. | ||
| // The pending operations are still applied. | ||
| var SkipAll = errors.New("skip everything and stop the walk") //nolint:revive,staticcheck // mimics the existing API in the standard library, see filepath |
There was a problem hiding this comment.
There is a linter issue that failed the CI:
Error: extension/xextension/storage/storage.go:87:5: error-naming: error var SkipAll should have name of the form ErrFoo (revive)
var SkipAll = errors.New("skip everything and stop the walk")
^
I'd rather prefer to keep the name as it is since it mimics the existing API in the standard library.
|
@songy23 please let me know if I can do anything to help with the review. |
|
This looks good to go @open-telemetry/collector-maintainers |
|
FYI @carsonip |
|
Should we apply similar interface as we have for opentelemetry-collector/pdata/pcommon/map.go Line 255 in d3b2c86 |
…n-telemetry#15190) #### Description Introduce `Walker` interface, `WalkFunc` type, and `SkipAll` error value to support iterating over storage key/value entries. This enables consumers to implement storage migrations or any functionality that involves ranging through the storage entries. One of the examples of such functionality may be a TTL-based garbage collection. Values can have a TTL marker, a routine runs with an interval iterating through entries searching for expired entries. `WalkFunc` produces `Delete` operations for each expired entry. The type names and the interface is close to the ones in [`filepath`](https://pkg.go.dev/path/filepath#WalkFunc) package of the standard library. Example: TTL-based garbage collection using `Walker` This example shows how to use the `Walker` interface to implement a TTL-based garbage collection routine. Each stored value carries a TTL marker (an expiration timestamp prefix). A periodic routine walks all entries, and the `WalkFunc` produces `Delete` operations for every expired entry. Values are prefixed with an 8-byte big-endian Unix timestamp (seconds) that represents the expiration time. The rest of the bytes are the actual payload. ```go import ( "encoding/binary" "time" ) func encodeWithTTL(value []byte, ttl time.Duration) []byte { expireAt := time.Now().Add(ttl).Unix() buf := make([]byte, 8+len(value)) binary.BigEndian.PutUint64(buf[:8], uint64(expireAt)) copy(buf[8:], value) return buf } func decodeExpiration(value []byte) (time.Time, []byte) { ts := binary.BigEndian.Uint64(value[:8]) return time.Unix(int64(ts), 0), value[8:] } ``` Garbage collection with `Walk`: ```go import ( "context" "fmt" "time" "go.opentelemetry.io/collector/extension/xextension/storage" ) func collectExpired(ctx context.Context, client storage.Client) error { walker, ok := client.(storage.Walker) if !ok { return fmt.Errorf("storage client does not implement Walker") } now := time.Now() return walker.Walk(ctx, func(key string, value []byte) ([]*storage.Operation, error) { if len(value) < 8 { return nil, nil } expireAt, _ := decodeExpiration(value) if now.After(expireAt) { return []*storage.Operation{storage.DeleteOperation(key)}, nil } return nil, nil }) } ``` All `Delete` operations returned by the `WalkFunc` are collected during the walk and applied atomically once the iteration completes. If the callback returns a non-nil error (other than `SkipAll`), or if the walk encounters an internal error, no operations are applied. <!--Describe the documentation added.--> #### Documentation The package's README.md has been updated. The new types have been fully documented. Closes open-telemetry#15191
You can easily implement that interface with the new one. Are you thinking of a QoL thing? |
Description
Introduce
Walkerinterface,WalkFunctype, andSkipAllerror value to support iterating over storage key/value entries.This enables consumers to implement storage migrations or any functionality that involves ranging through the storage entries.
One of the examples of such functionality may be a TTL-based garbage collection. Values can have a TTL marker, a routine runs with an interval iterating through entries searching for expired entries.
WalkFuncproducesDeleteoperations for each expired entry.The type names and the interface is close to the ones in
filepathpackage of the standard library.Example: TTL-based garbage collection using
WalkerThis example shows how to use the
Walkerinterface to implement a TTL-basedgarbage collection routine. Each stored value carries a TTL marker (an expiration
timestamp prefix). A periodic routine walks all entries, and the
WalkFuncproduces
Deleteoperations for every expired entry.Values are prefixed with an 8-byte big-endian Unix timestamp (seconds) that
represents the expiration time. The rest of the bytes are the actual payload.
Garbage collection with
Walk:All
Deleteoperations returned by theWalkFuncare collected during the walkand applied atomically once the iteration completes. If the callback returns a
non-nil error (other than
SkipAll), or if the walk encounters an internalerror, no operations are applied.
Documentation
The package's README.md has been updated. The new types have been fully documented.
Closes #15191