Skip to content

IntervalShardingAlgorithm performance is too bad #14047

@Ahoo-Wang

Description

@Ahoo-Wang

Feature Request

When I was about to use IntervalShardingAlgorithm to integrate CosId, I checked the source code and found the following problems:

  • Ease of use: The IntervalShardingAlgorithm implementation is to first convert to a string and then convert to LocalDateTime, the conversion success rate is affected by the time formatting characters
  • Performance: The method of nested traversal checking determines whether the conditions are met, and it is accompanied by the conversion process of LocalDateTime. The performance problem is fatal. The performance of PreciseShardingValue is even lower than 7000 ops/s (it is lower than the storage layer MySql, ShardingSphere-JDBC becomes the bottleneck, which is obviously unbearable)

Code implementation of benchmark report

I even doubt whether there is a problem with the way the benchmark report is implemented,But I have tried my best to eliminate the test noise

Interval-based-sharding-algorithm-jmh

gradle cosid-shardingsphere:jmh
# JMH version: 1.29
# VM version: JDK 11.0.13, OpenJDK 64-Bit Server VM, 11.0.13+8-LTS
# VM options: -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/work/CosId/cosid-shardingsphere/build/tmp/jmh -Duser.country=CN -Duser.language=zh -Duser.variant
# Blackhole mode: full + dont-inline hint
# Warmup: 1 iterations, 10 s each
# Measurement: 1 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
Benchmark                                                          Mode  Cnt         Score   Error  Units
IntervalShardingAlgorithmBenchmark.cosid_precise_local_date_time  thrpt       66276995.822          ops/s
IntervalShardingAlgorithmBenchmark.cosid_precise_timestamp        thrpt       24841952.001          ops/s
IntervalShardingAlgorithmBenchmark.cosid_range_local_date_time    thrpt        3344013.803          ops/s
IntervalShardingAlgorithmBenchmark.cosid_range_timestamp          thrpt        2846453.298          ops/s
IntervalShardingAlgorithmBenchmark.office_precise_timestamp       thrpt           6286.861          ops/s
IntervalShardingAlgorithmBenchmark.office_range_timestamp         thrpt           2302.986          ops/s

So I re-implemented the time range sharding algorithm based on time interval to improve the ease of use and performance.

https://github.com/Ahoo-Wang/CosId/releases/tag/v1.4.5

CosIdIntervalShardingAlgorithm

  • DateIntervalShardingAlgorithm
    • type: COSID_INTERVAL_DATE
  • LocalDateTimeIntervalShardingAlgorithm
    • type: COSID_INTERVAL_LDT
  • TimestampIntervalShardingAlgorithm
    • type: COSID_INTERVAL_TS
  • TimestampOfSecondIntervalShardingAlgorithm
    • type: COSID_INTERVAL_TS_SECOND
  • SnowflakeIntervalShardingAlgorithm
    • type: COSID_INTERVAL_SNOWFLAKE

Is your feature request related to a problem?

NO

Describe the feature you would like.

If you think this implementation is good, I can submit a PR

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions