Skip to content

[BEAM-12593] Verify DataFrame API on pandas 1.3 (with container update)#15525

Merged
TheNeuralBit merged 3 commits intoapache:masterfrom
TheNeuralBit:pandas-1.3
Sep 28, 2021
Merged

[BEAM-12593] Verify DataFrame API on pandas 1.3 (with container update)#15525
TheNeuralBit merged 3 commits intoapache:masterfrom
TheNeuralBit:pandas-1.3

Conversation

@TheNeuralBit
Copy link
Copy Markdown
Member

@TheNeuralBit TheNeuralBit commented Sep 16, 2021

Re-apply #15165 (which was reverted in #15397) with additional commits to bump the pandas version in containers.

ValidatesRunner compliance status (on master branch)

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- Build Status Build Status Build Status Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Python --- Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status ---
XLang Build Status Build Status Build Status Build Status Build Status ---

Examples testing status on various runners

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- --- --- --- --- --- ---
Java --- Build Status
Build Status
Build Status
--- --- --- --- ---
Python --- --- --- --- --- --- ---
XLang --- --- --- --- --- --- ---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go Java Python
Build Status Build Status Build Status
Build Status
Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status Build Status Build Status
Portable --- Build Status Build Status --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@TheNeuralBit
Copy link
Copy Markdown
Member Author

Run Python 3.8 PostCommit

@TheNeuralBit
Copy link
Copy Markdown
Member Author

Run Python 3.6 PostCommit

@codecov
Copy link
Copy Markdown

codecov Bot commented Sep 16, 2021

Codecov Report

Merging #15525 (50a3b4c) into master (e11b80c) will decrease coverage by 0.01%.
The diff coverage is 92.85%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #15525      +/-   ##
==========================================
- Coverage   83.78%   83.76%   -0.02%     
==========================================
  Files         444      444              
  Lines       60397    60410      +13     
==========================================
+ Hits        50601    50604       +3     
- Misses       9796     9806      +10     
Impacted Files Coverage Δ
sdks/python/apache_beam/dataframe/frames.py 94.85% <92.85%> (-0.02%) ⬇️
sdks/python/apache_beam/utils/interactive_utils.py 87.80% <0.00%> (-7.32%) ⬇️
sdks/python/apache_beam/internal/metrics/metric.py 90.42% <0.00%> (-1.07%) ⬇️
...pache_beam/runners/interactive/interactive_beam.py 75.60% <0.00%> (-0.98%) ⬇️
...runners/interactive/display/pcoll_visualization.py 85.85% <0.00%> (-0.51%) ⬇️
...eam/runners/interactive/interactive_environment.py 91.03% <0.00%> (-0.36%) ⬇️
...hon/apache_beam/runners/worker/bundle_processor.py 93.26% <0.00%> (-0.25%) ⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py 89.16% <0.00%> (+0.15%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 13d1d27...50a3b4c. Read the comment docs.

@TheNeuralBit
Copy link
Copy Markdown
Member Author

Run Python PreCommit

@TheNeuralBit TheNeuralBit changed the title [WIP][BEAM-12593] Verify DataFrame APi on pandas 1.3 (with container update) [BEAM-12593] Verify DataFrame API on pandas 1.3 (with container update) Sep 16, 2021
@TheNeuralBit
Copy link
Copy Markdown
Member Author

R: @tvalentyn

@TheNeuralBit
Copy link
Copy Markdown
Member Author

Note 20cd6b5978a673e00d4b9455d27b59305ec56a3d is a pure re-apply of #15165, which was already reviewed.

rohdesamuel and others added 2 commits September 27, 2021 14:37
…I on pandas 1.3.0

* [BEAM-12593] Verify DataFrame API on pandas 1.3.0

* Fills correct partitioning for fillna
* Add version number to lib
* Condition on incompatible version features
* Due to BEAM-12495 value_counts with dropna=False does not work on
frames
* Added NotImplementedError when detecting BEAM-12495

* fix insert/value_counts doc test

* update Pandas doctests for groupby for 1.3.x Pandas

* test dropna&normalize combinations

Co-authored-by: Brian Hulette <[email protected]>
@TheNeuralBit
Copy link
Copy Markdown
Member Author

Run Python 3.8 PostCommit

@TheNeuralBit
Copy link
Copy Markdown
Member Author

R: @ibzib

@tvalentyn
Copy link
Copy Markdown
Contributor

Sorry I remember looking at the PR but looks like I forgot to approve.

@TheNeuralBit
Copy link
Copy Markdown
Member Author

No worries! Thanks for the review :)

@TheNeuralBit
Copy link
Copy Markdown
Member Author

Run Python 3.8 PostCommit

@TheNeuralBit TheNeuralBit merged commit 6f6efa5 into apache:master Sep 28, 2021
dmitriikuzinepam pushed a commit to dmitriikuzinepam/beam that referenced this pull request Nov 2, 2021
…e) (apache#15525)

* Merge pull request apache#15165 from [BEAM-12593] Verify DataFrame API on pandas 1.3.0

* [BEAM-12593] Verify DataFrame API on pandas 1.3.0

* Fills correct partitioning for fillna
* Add version number to lib
* Condition on incompatible version features
* Due to BEAM-12495 value_counts with dropna=False does not work on
frames
* Added NotImplementedError when detecting BEAM-12495

* fix insert/value_counts doc test

* update Pandas doctests for groupby for 1.3.x Pandas

* test dropna&normalize combinations

Co-authored-by: Brian Hulette <[email protected]>

* Upgrade to pandas==1.3.3 in Python container

Co-authored-by: Sam sam <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants