Skip to content

analyze function that collects various statistics #4478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
jangorecki wants to merge 34 commits into
base: master
Choose a base branch
from
Draft

analyze function that collects various statistics #4478

jangorecki wants to merge 34 commits into from

Conversation

@jangorecki
Copy link
Member

@jangorecki jangorecki commented May 23, 2020 •
edited
Loading

Built on top of lazy-forder branch, diff will include lazy-forder diff as well!
For useful diff see https://github.com/Rdatatable/data.table/compare/lazy-forder...analyze?expand=1
After merging lazy-forder will need rebase.
Closes #2879

x = data.table(id1=1:5, id2=c(NA,2:5), id3=rep(NA_integer_, 5L), id4=c("a","b","c","b","a"), id5=c("a","b","\u221A","d","e"), v1=1L/2, v2=c(NaN,2:5/2), v3=c(1:4/2, NaN), v4=c(-Inf,2:4,Inf))
cols = list("id1", "id2","id3", c("id1","id2"), "id4", "id5", c("id4","id5"), "v1", "v2", "v3", "v4")

a = analyze(x, cols, flat=TRUE)
a
#       cols ncols             coltype            colclass sorted uniqueN maxgrpN  anyNA anyInfNaN anyNotASCII anyNotUTF8  anyNF unique  const  allNA
#     <char> <int>              <char>              <char> <lgcl>   <int>   <int> <lgcl>    <lgcl>      <lgcl>     <lgcl> <lgcl> <lgcl> <lgcl> <lgcl>
# 1:     id1     1             integer             integer   TRUE       5       1  FALSE     FALSE       FALSE      FALSE  FALSE   TRUE  FALSE  FALSE
# 2:     id2     1             integer             integer   TRUE       5       1   TRUE     FALSE       FALSE      FALSE   TRUE   TRUE  FALSE  FALSE
# 3:     id3     1             integer             integer   TRUE       1       5   TRUE     FALSE       FALSE      FALSE   TRUE  FALSE   TRUE   TRUE
# 4: id1,id2     2     integer,integer     integer,integer   TRUE       5       1   TRUE     FALSE       FALSE      FALSE   TRUE   TRUE  FALSE  FALSE
# 5:     id4     1           character           character  FALSE       3       2  FALSE     FALSE       FALSE      FALSE  FALSE  FALSE  FALSE  FALSE
# 6:     id5     1           character           character  FALSE       5       1  FALSE     FALSE        TRUE      FALSE  FALSE   TRUE  FALSE  FALSE
# 7: id4,id5     2 character,character character,character  FALSE       5       1  FALSE     FALSE        TRUE      FALSE  FALSE   TRUE  FALSE  FALSE
# 8:      v1     1              double             numeric   TRUE       1       5  FALSE     FALSE       FALSE      FALSE  FALSE  FALSE   TRUE  FALSE
# 9:      v2     1              double             numeric   TRUE       5       1  FALSE      TRUE       FALSE      FALSE   TRUE   TRUE  FALSE  FALSE
#10:      v3     1              double             numeric  FALSE       5       1  FALSE      TRUE       FALSE      FALSE   TRUE   TRUE  FALSE  FALSE
#11:      v4     1              double             numeric   TRUE       5       1  FALSE      TRUE       FALSE      FALSE   TRUE   TRUE  FALSE  FALSE

a = analyze(x, cols) ## flat=FALSE returns first and last rows as nested DT

No tests yet. No manual yet. I will follow up with this after some initial feedback.

jangorecki added 30 commits April 18, 2020 19:38
@jangorecki
lazy forder
b0efcf5
@jangorecki
fix tests
aaead65
@jangorecki
fix tests
b0858df
@jangorecki
respect use.index option
9df2668
@jangorecki
bmerge timings
62982ed
@jangorecki
codecov
eec4971
@jangorecki
helper function
9affbfa
@jangorecki
reduce diff to master
b36136a
@jangorecki
rename fix
3442b78
@jangorecki
setindex writes groups (retGrp=TRUE)
dfe883e 
forder C set index directly
smart opt for index retGrp=T/F
no tests updated yet
@jangorecki
calc order, not groups
efb3319
@jangorecki
expect to reach optimization
68b378d
@jangorecki
skip opt for list
0bd0e1e
@jangorecki
override retGrp=F by retGrp=T is legit use
71f5aeb
@jangorecki
more backward compatiblility, no retGrp from getindex
de916aa
@jangorecki
more verbose messages during opts in forderLazy
0a3da1d
@jangorecki
recycle order 1/-1 argument in one place
068c1a9
@jangorecki
precise verbose messages
326695d
@jangorecki
recycle arg once _by_ arg is known
8be2d53
@jangorecki
copy paste typo fix
3383bbf
@jangorecki
forder writing index disabled
cc7a2b6
@jangorecki
fix tests
514913a
@jangorecki
code coverage
ca2823d
@jangorecki
minor update for safer use of internal option
4e9bbb3
@jangorecki
fix bad name in unit test
4d0dec3
@jangorecki
retGrp=F requires downgrade idx and it seems to be costly
1606046
@jangorecki
NA stats from forder
a9b01ff
@jangorecki
keyOpt fix, and existing tests
ded1e1a
@jangorecki
fixes for na.last in key and setting idx
53cce98
@jangorecki
filling tests for na.last=T and possible fixes
1e92366
@jangorecki jangorecki added the WIP label May 23, 2020
@codecov
Copy link

codecov bot commented May 23, 2020 •
edited
Loading

Codecov Report

Attention: Patch coverage is 90.68966% with 27 lines in your changes missing coverage. Please review.

Project coverage is 99.42%. Comparing base (b1b1832) to head (9382cb8).
Report is 1620 commits behind head on master.

Files with missing lines Patch % Lines
R/setkey.R 34.14% 27 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #4478      +/-   ##
==========================================
- Coverage   99.61%   99.42%   -0.20%     
==========================================
  Files          72       72              
  Lines       13916    14153     +237     
==========================================
+ Hits        13862    14071     +209     
- Misses         54       82      +28

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@MichaelChirico MichaelChirico marked this pull request as draft February 19, 2024 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MichaelChirico MichaelChirico Awaiting requested review from MichaelChirico MichaelChirico will be requested when the pull request is marked ready for review MichaelChirico is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

collect more statistics about the data

2 participants

@jangorecki @MichaelChirico