[ntuple] Overhaul tuning and default settings when writing by jblomer · Pull Request #8703 · root-project/root

jblomer · 2021-07-20T06:20:37Z

This Pull request:

The PR sets new defaults for the cluster size and page size of RNTuple. The default should work well in the majority of cases but can be adjusted if needed. The idea is to give target sizes for clusters and pages (measured in bytes). RNTuple will try make good decisions and approximate the target sizes. The PR replaces previous defaults for cluster size and page size given in number of entries and number of elements resp.

Changes or fixes:

The PR sets three new defaults:

Target size for compressed clusters of 50MB. In general, larger clusters provide room for more and larger pages and should improve compression ratio and speed. However, clusters also need to be buffered during write and (partially) during read, so larger cluster increase the memory footprint.
Maximum size for uncompressed clusters of 512MiB. Prevents very compressible clusters from growing too large. That is mostly a problem for writing.
Target size for uncompressed pages of 64KiB. In general, larger pages give better compression ratios. Smaller pages, however, reduce the memory footprint. When reading, every active column requires at least one page buffer. For the number of read requests, the page size does not matter because pages of the same column are written consecutively and therefore read in one go.

Given the three settings, writing works as follows: when the current cluster is larger than the maximum uncompressed size, it will be flushed unconditionally. When the current cluster size reaches the estimate for the compressed cluster size, it will be flushed, too. The estimated compression ratio for the first cluster is 0.5 if compression is used, and 1 otherwise. The following clusters use the compression ratio of the last cluster as estimate.

Pages are filled until the target size and then flushed. If a column has enough elements to fill at least half a page, there is a mechanism to prevent undersized tail pages: writing uses two page buffers in turns and flushes the previous buffer only once the next buffer is at least at 50%. If the cluster gets flushed with an undersized tail page, the small page is appended to the previous page before flushing. Therefore, tail pages sizes are between [0.5 * target size .. 1.5 * target size].

phsft-bot · 2021-07-20T06:20:47Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

ghost · 2021-07-20T06:48:49Z

DeepCode's analysis on #9dbf29 found:

⚠️ 1 warning 👇

Top issues

Description	Example fixes
Potential nullptr dereference. Null flows from nullptr literal. Consider adding a check. Occurrences: RColumn.hxx:91	🔧 Example fixes

👉 View analysis in DeepCode’s Dashboard | Configure the bot

👉 The DeepCode service and API will be deprecated in August, 2021. Here is the information how to migrate. Thank you for using DeepCode 🙏 ❤️ !

If you are using our plugins, you might be interested in their successors: Snyk's JetBrains plugin and Snyk's VS Code plugin.

phsft-bot · 2021-07-20T07:00:20Z

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft10.dyndns.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Errors:

[2021-07-20T06:21:55.108Z] stderr: error: could not read '.git/rebase-apply/head-name': No such file or directory

Failing tests:

projectroot.roottest.python.cpp.roottest_python_cpp_cpp

phsft-bot · 2021-07-22T07:48:29Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-07-22T08:48:12Z

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft11.dyndns.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

projectroot.roottest.python.cpp.roottest_python_cpp_cpp

Axel-Naumann

It would be good to have the PR description as documentation somewhere. Maybe split for cluster and page flushing?

tree/ntuple/v7/inc/ROOT/RColumn.hxx

tree/ntuple/v7/src/RColumn.cxx

tree/ntuple/v7/src/RField.cxx

tree/ntuple/v7/src/RNTuple.cxx

Co-authored-by: Axel Naumann <[email protected]>

phsft-bot · 2021-07-22T11:38:50Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

Co-authored-by: Axel Naumann <[email protected]>

phsft-bot · 2021-07-22T11:41:39Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

Co-authored-by: Axel Naumann <[email protected]>

phsft-bot · 2021-07-22T11:43:19Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-07-22T13:29:39Z

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft10.dyndns.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Errors:

[2021-07-22T12:51:42.350Z] stderr: error: could not read '.git/rebase-apply/head-name': No such file or directory

Failing tests:

projectroot.roottest.python.cpp.roottest_python_cpp_cpp

phsft-bot · 2021-07-22T14:10:06Z

Build failed on mac11.0/cxx17.
Running on macphsft20.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

phsft-bot · 2021-09-17T07:34:46Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

…geImpl

phsft-bot · 2021-09-17T07:44:25Z

Build failed on windows10/cxx14.
Running on null:C:\build\workspace\root-pullrequests-build
See console output.

Errors:

[2021-09-17T07:44:24.243Z] CMake Error at cmake/modules/RootBuildOptions.cmake:407 (message):
[2021-09-17T07:44:24.243Z] CMake Error at C:/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1117 (message):

phsft-bot · 2021-09-17T08:16:29Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

…WritePage

phsft-bot · 2021-09-17T09:23:31Z

Build failed on mac11.0/cxx17.
Running on macphsft20.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

phsft-bot · 2021-09-17T09:26:44Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

…ator

phsft-bot · 2021-09-17T10:05:14Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-09-17T10:15:29Z

Build failed on windows10/cxx14.
Running on null:C:\build\workspace\root-pullrequests-build
See console output.

Errors:

[2021-09-17T10:15:29.154Z] CMake Error at cmake/modules/RootBuildOptions.cmake:407 (message):
[2021-09-17T10:15:29.154Z] CMake Error at C:/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1117 (message):

phsft-bot · 2021-09-17T11:11:38Z

Build failed on mac11.0/cxx17.
Running on macphsft20.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

phsft-bot · 2021-09-19T11:15:40Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-09-19T11:22:45Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

jblomer · 2021-09-19T11:26:34Z

@pcanal As discussed, the estimator for the compressed cluster size is now the average compression ratio of all so-far written clusters. I also added a stub checklist for the future RNTuple validation.

phsft-bot · 2021-09-19T11:33:12Z

Build failed on windows10/cxx14.
Running on null:C:\build\workspace\root-pullrequests-build
See console output.

Errors:

[2021-09-19T11:33:11.882Z] CMake Error at cmake/modules/RootBuildOptions.cmake:407 (message):
[2021-09-19T11:33:11.882Z] CMake Error at C:/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1117 (message):

phsft-bot · 2021-09-19T12:29:25Z

Build failed on mac11.0/cxx17.
Running on macphsft20.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

pcanal

Approved pending resolution of 2 left over conversations.

jblomer added 10 commits July 19, 2021 11:26

[ntuple] Default page size on write to 64kB (WIP)

43876e7

[ntuple] Return new cluster ID from CommitCluster()

86d5a35

[ntuple] Return number of unzipped bytes added from RField::Append()

f5198f5

[ntuple] Add RClusterDescriptor::GetBytesOnStorage()

2573cd8

[ntuple] Add RNTupleWriteOptions::fMaxUnzippedClusterSize

83d4123

[ntuple] Return number of bytes written to storage from CommitCluster()

c30995f

[ntuple] Auto-tune cluster size according to expected zipped size

b6e8b8e

[ntuple] Use approximate target cluster size instead of min/max

04b9101

[ntuple] Adjust cluster size tunables

553b413

[ntuple] Avoid small tail pages

346123d

jblomer self-assigned this Jul 20, 2021

jblomer requested a review from pcanal as a code owner July 20, 2021 06:20

jblomer requested review from Axel-Naumann and eguiraud July 20, 2021 06:21

[ntuple] Prevent current reading code from choking on 0-byte clusters

ee3e1bc

Axel-Naumann reviewed Jul 22, 2021

View reviewed changes

[ntuple] Simplify setting compression estimate for 1st cluster

fa06f3a

Co-authored-by: Axel Naumann <[email protected]>

[ntuple] Add description to R_ASSERT in AppendImpl

9d3657e

Co-authored-by: Axel Naumann <[email protected]>

[ntuple] Use more concise expression for flapping index

c9a6b0e

Co-authored-by: Axel Naumann <[email protected]>

[ntuple] Add tuning.md initially explaining cluster/page sizes

dc4f48d

[ntuple] More concise implementation of RPageSinkDaos::CommitSealedPa…

847186c

…geImpl

jblomer added 4 commits September 17, 2021 09:46

[ntuple] Improve input check ordering in RNTupleWriteOptions

831d6c7

[ntuple] Remove unnecessary include

b2be479

[ntuple] Rename RColumn::fHeadPage --> RColumn::fWritePage

a60570a

[ntuple] Rename RColumn::fCurrentPage --> RColumn::fReadPage

560c4a8

jblomer added 3 commits September 17, 2021 10:25

[ntuple, NFC] Fix code comment

f493c17

[ntuple] Rename RColumn::SwapHeadPages --> RColumn::SwapWritePages

db95daf

[ntuple] Rename RColumn::FlushShadowHeadPage --> RColumn::FlushShadow…

87c59d8

…WritePage

[ntuple] Consolidate check for full pages in RColumn

55ace11

[ntuple, NFC] Add discussion section on compressed cluster size estim…

dd6c0a9

…ator

jblomer requested a review from jalopezg-git September 17, 2021 10:21

[ntuple] Use avg compression ratio of all clusters as estimate for next

806637b

[ntuple, doc] Add validation checklist stub

5a02334

pcanal approved these changes Sep 20, 2021

View reviewed changes

jalopezg-git approved these changes Sep 20, 2021

View reviewed changes

jblomer merged commit 61124f2 into root-project:master Sep 21, 2021

jblomer deleted the ntuple-write-autotuning branch September 21, 2021 12:44

Conversation

jblomer commented Jul 20, 2021 • edited by pcanal Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This Pull request:

Changes or fixes:

Uh oh!

phsft-bot commented Jul 20, 2021

Uh oh!

ghost commented Jul 20, 2021 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DeepCode's analysis on #9dbf29 found:

Top issues

👉 View analysis in DeepCode’s Dashboard | Configure the bot

👉 The DeepCode service and API will be deprecated in August, 2021. Here is the information how to migrate. Thank you for using DeepCode 🙏 ❤️ !

Uh oh!

phsft-bot commented Jul 20, 2021

Errors:

Failing tests:

Uh oh!

phsft-bot commented Jul 22, 2021

Uh oh!

phsft-bot commented Jul 22, 2021

Failing tests:

Uh oh!

Axel-Naumann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phsft-bot commented Jul 22, 2021

Uh oh!

phsft-bot commented Jul 22, 2021

Uh oh!

phsft-bot commented Jul 22, 2021

Uh oh!

phsft-bot commented Jul 22, 2021

Errors:

Failing tests:

Uh oh!

phsft-bot commented Jul 22, 2021

Failing tests:

Uh oh!

phsft-bot commented Sep 17, 2021

Uh oh!

phsft-bot commented Sep 17, 2021

Errors:

Uh oh!

phsft-bot commented Sep 17, 2021

Uh oh!

phsft-bot commented Sep 17, 2021

Failing tests:

Uh oh!

phsft-bot commented Sep 17, 2021

Uh oh!

phsft-bot commented Sep 17, 2021

Uh oh!

phsft-bot commented Sep 17, 2021

Errors:

Uh oh!

phsft-bot commented Sep 17, 2021

Failing tests:

Uh oh!

phsft-bot commented Sep 19, 2021

Uh oh!

phsft-bot commented Sep 19, 2021

Uh oh!

jblomer commented Sep 19, 2021

Uh oh!

phsft-bot commented Sep 19, 2021

Errors:

Uh oh!

phsft-bot commented Sep 19, 2021

jblomer commented Jul 20, 2021 •

edited by pcanal

Loading

ghost commented Jul 20, 2021 •

edited by ghost

Loading