[BEAM-12593] Verify DataFrame API on pandas 1.3.0 by rohdesamuel · Pull Request #15165 · apache/beam

rohdesamuel · 2021-07-13T17:40:26Z

Fills correct partitioning for fillna
Add version number to lib
Condition on incompatible version features
Due to BEAM-12495 value_counts with dropna=False does not work on
frames
Added NotImplementedError when detecting BEAM-12495

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Samza	Twister2
Go	---	---	---
Java
Python	---	---	---
XLang		---	---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

rohdesamuel · 2021-07-13T17:58:02Z

@robertwb

rohdesamuel · 2021-07-13T18:40:20Z

@aaltay

rohdesamuel · 2021-07-13T18:40:39Z

This is a continuation from #15008

codecov · 2021-07-16T19:08:44Z

Codecov Report

Merging #15165 (2ebdf65) into master (7b54b0b) will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #15165      +/-   ##
==========================================
+ Coverage   83.86%   83.89%   +0.03%     
==========================================
  Files         441      441              
  Lines       59724    60472     +748     
==========================================
+ Hits        50086    50735     +649     
- Misses       9638     9737      +99

Impacted Files	Coverage Δ
sdks/python/apache_beam/dataframe/frames.py	`94.93% <100.00%> (+0.06%)`	⬆️
sdks/python/apache_beam/utils/interactive_utils.py	`87.80% <0.00%> (-7.32%)`	⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py	`94.02% <0.00%> (-2.24%)`	⬇️
sdks/python/apache_beam/io/source_test_utils.py	`88.47% <0.00%> (-1.39%)`	⬇️
...pache_beam/runners/interactive/interactive_beam.py	`75.53% <0.00%> (-1.07%)`	⬇️
...runners/interactive/display/pcoll_visualization.py	`85.26% <0.00%> (-0.53%)`	⬇️
...eam/runners/interactive/interactive_environment.py	`90.33% <0.00%> (-0.38%)`	⬇️
...ython/apache_beam/examples/kafkataxi/kafka_taxi.py	`0.00% <0.00%> (ø)`
...s/python/apache_beam/io/gcp/bigquery_file_loads.py	`88.07% <0.00%> (+0.57%)`	⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b54b0b...2ebdf65. Read the comment docs.

aaltay

LGTM.

There is one open comment and failing tests. Could you ping me once those are resolved?

rohdesamuel · 2021-07-28T23:29:51Z

Run Python_PVR_Flink PreCommit

aaltay · 2021-07-29T01:40:12Z

Run Python PreCommit

aaltay · 2021-07-29T01:40:52Z

Thank you. Ping me or any other committer to merge once tests pass.

rohdesamuel · 2021-07-30T00:32:58Z

Run Python PreCommit

aaltay · 2021-07-30T01:47:37Z

Run Python PreCommit

rohdesamuel · 2021-07-30T19:55:24Z

Run Python PreCommit

rohdesamuel · 2021-07-30T22:31:31Z

Run Python PreCommit

rohdesamuel · 2021-07-31T00:47:13Z

Run Python PreCommit

rohdesamuel · 2021-07-31T07:17:31Z

Run Python PreCommit

rohdesamuel · 2021-08-02T06:20:06Z

Run Python PreCommit

* Fills correct partitioning for fillna * Add version number to lib * Condition on incompatible version features * Due to BEAM-12495 value_counts with dropna=False does not work on frames * Added NotImplementedError when detecting BEAM-12495

rohdesamuel · 2021-08-03T18:05:38Z

Run Python PreCommit

rohdesamuel · 2021-08-03T23:56:09Z

Run Python PreCommit

rohdesamuel · 2021-08-09T17:12:13Z

Run Python PreCommit

rohdesamuel · 2021-08-11T17:04:29Z

Run Python PreCommit

pabloem · 2021-08-12T17:43:44Z

failures are from a test broken on master

lukecwik · 2021-08-16T22:01:01Z

I believe merging this broke testing: https://issues.apache.org/jira/browse/BEAM-12764

…] Verify DataFrame API on pandas 1.3.0" This reverts commit faac725.

[BEAM-12764] Revert "Merge pull request #15165 from [BEAM-12593] Veri…

…] Veri…"

…I on pandas 1.3.0 * [BEAM-12593] Verify DataFrame API on pandas 1.3.0 * Fills correct partitioning for fillna * Add version number to lib * Condition on incompatible version features * Due to BEAM-12495 value_counts with dropna=False does not work on frames * Added NotImplementedError when detecting BEAM-12495 * fix insert/value_counts doc test * update Pandas doctests for groupby for 1.3.x Pandas * test dropna&normalize combinations Co-authored-by: Brian Hulette <[email protected]>

…e) (#15525) * Merge pull request #15165 from [BEAM-12593] Verify DataFrame API on pandas 1.3.0 * [BEAM-12593] Verify DataFrame API on pandas 1.3.0 * Fills correct partitioning for fillna * Add version number to lib * Condition on incompatible version features * Due to BEAM-12495 value_counts with dropna=False does not work on frames * Added NotImplementedError when detecting BEAM-12495 * fix insert/value_counts doc test * update Pandas doctests for groupby for 1.3.x Pandas * test dropna&normalize combinations Co-authored-by: Brian Hulette <[email protected]> * Upgrade to pandas==1.3.3 in Python container Co-authored-by: Sam sam <[email protected]>

…e) (apache#15525) * Merge pull request apache#15165 from [BEAM-12593] Verify DataFrame API on pandas 1.3.0 * [BEAM-12593] Verify DataFrame API on pandas 1.3.0 * Fills correct partitioning for fillna * Add version number to lib * Condition on incompatible version features * Due to BEAM-12495 value_counts with dropna=False does not work on frames * Added NotImplementedError when detecting BEAM-12495 * fix insert/value_counts doc test * update Pandas doctests for groupby for 1.3.x Pandas * test dropna&normalize combinations Co-authored-by: Brian Hulette <[email protected]> * Upgrade to pandas==1.3.3 in Python container Co-authored-by: Sam sam <[email protected]>

rohdesamuel force-pushed the dataframe-1.3.0rc1 branch from b771712 to f7b6574 Compare July 13, 2021 17:51

rohdesamuel marked this pull request as ready for review July 13, 2021 17:58

aaltay reviewed Jul 13, 2021

View reviewed changes

aaltay approved these changes Jul 21, 2021

View reviewed changes

aaltay mentioned this pull request Jul 23, 2021

[WIP][BEAM-12593] Verify DataFrame API on pandas 1.3.0 #15008

Closed

rohdesamuel force-pushed the dataframe-1.3.0rc1 branch from 4a4ceb7 to f1b88ae Compare July 28, 2021 21:12

rohdesamuel force-pushed the dataframe-1.3.0rc1 branch from f1b88ae to 98cee78 Compare July 28, 2021 23:40

rohdesamuel force-pushed the dataframe-1.3.0rc1 branch from 98cee78 to 86ca490 Compare July 29, 2021 22:22

TheNeuralBit and others added 4 commits August 2, 2021 12:50

fix insert/value_counts doc test

74f498b

update Pandas doctests for groupby for 1.3.x Pandas

8b0f299

test dropna&normalize combinations

2ebdf65

rohdesamuel force-pushed the dataframe-1.3.0rc1 branch from 86ca490 to 2ebdf65 Compare August 2, 2021 19:50

pabloem merged commit faac725 into apache:master Aug 12, 2021

ibzib pushed a commit to ibzib/beam that referenced this pull request Aug 24, 2021

[BEAM-12764] Revert "Merge pull request apache#15165 from [BEAM-12593…

fc7df4b

…] Verify DataFrame API on pandas 1.3.0" This reverts commit faac725.

ibzib pushed a commit that referenced this pull request Aug 25, 2021

Merge pull request #15385 from ibzib/rollback-dataframes

aa1c8e5

[BEAM-12764] Revert "Merge pull request #15165 from [BEAM-12593] Veri…

chamikaramj added a commit that referenced this pull request Aug 26, 2021

Revert "[BEAM-12764] Revert "Merge pull request #15165 from [BEAM-12593…

0766b5c

…] Veri…"

TheNeuralBit mentioned this pull request Sep 16, 2021

[BEAM-12593] Verify DataFrame API on pandas 1.3 (with container update) #15525

Merged

Conversation

rohdesamuel commented Jul 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

rohdesamuel commented Jul 13, 2021

Uh oh!

rohdesamuel commented Jul 13, 2021

Uh oh!

rohdesamuel commented Jul 13, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jul 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

aaltay left a comment

Choose a reason for hiding this comment

Uh oh!

rohdesamuel commented Jul 28, 2021

Uh oh!

aaltay commented Jul 29, 2021

Uh oh!

aaltay commented Jul 29, 2021

Uh oh!

rohdesamuel commented Jul 30, 2021

Uh oh!

aaltay commented Jul 30, 2021

Uh oh!

rohdesamuel commented Jul 30, 2021

Uh oh!

rohdesamuel commented Jul 30, 2021

Uh oh!

rohdesamuel commented Jul 31, 2021

Uh oh!

rohdesamuel commented Jul 31, 2021

Uh oh!

rohdesamuel commented Aug 2, 2021

Uh oh!

rohdesamuel commented Aug 3, 2021

Uh oh!

rohdesamuel commented Aug 3, 2021

Uh oh!

rohdesamuel commented Aug 9, 2021

Uh oh!

rohdesamuel commented Aug 11, 2021

Uh oh!

pabloem commented Aug 12, 2021

Uh oh!

lukecwik commented Aug 16, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rohdesamuel commented Jul 13, 2021 •

edited

Loading

`ValidatesRunner` compliance status (on master branch)

codecov Bot commented Jul 16, 2021 •

edited

Loading