Skip to content

Commit 1872d87

Browse files
author
Tobiasz Kędzierski
authored
Add Apache Beam operators (#12814)
1 parent 14805cc commit 1872d87

File tree

33 files changed

+3248
-683
lines changed

33 files changed

+3248
-683
lines changed

‎CONTRIBUTING.rst‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -641,12 +641,13 @@ Here is the list of packages and their extras:
641641
Package Extras
642642
========================== ===========================
643643
amazon apache.hive,exasol,ftp,google,imap,mongo,mysql,postgres,ssh
644+
apache.beam google
644645
apache.druid apache.hive
645646
apache.hive amazon,microsoft.mssql,mysql,presto,samba,vertica
646647
apache.livy http
647648
dingding http
648649
discord http
649-
google amazon,apache.cassandra,cncf.kubernetes,facebook,microsoft.azure,microsoft.mssql,mysql,oracle,postgres,presto,salesforce,sftp,ssh
650+
google amazon,apache.beam,apache.cassandra,cncf.kubernetes,facebook,microsoft.azure,microsoft.mssql,mysql,oracle,postgres,presto,salesforce,sftp,ssh
650651
hashicorp google
651652
microsoft.azure google,oracle
652653
microsoft.mssql odbc

‎airflow/providers/apache/beam/BACKPORT_PROVIDER_README.md‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
21+
# Package apache-airflow-backport-providers-apache-beam
22+
23+
Release:
24+
25+
**Table of contents**
26+
27+
- [Backport package](#backport-package)
28+
- [Installation](#installation)
29+
- [PIP requirements](#pip-requirements)
30+
- [Cross provider package dependencies](#cross-provider-package-dependencies)
31+
- [Provider class summary](#provider-classes-summary)
32+
- [Operators](#operators)
33+
- [Moved operators](#moved-operators)
34+
- [Transfer operators](#transfer-operators)
35+
- [Moved transfer operators](#moved-transfer-operators)
36+
- [Hooks](#hooks)
37+
- [Moved hooks](#moved-hooks)
38+
- [Releases](#releases)
39+
- [Release](#release)
40+
41+
## Backport package
42+
43+
This is a backport providers package for `apache.beam` provider. All classes for this provider package
44+
are in `airflow.providers.apache.beam` python package.
45+
46+
**Only Python 3.6+ is supported for this backport package.**
47+
48+
While Airflow 1.10.* continues to support Python 2.7+ - you need to upgrade python to 3.6+ if you
49+
want to use this backport package.
50+
51+
52+
## Installation
53+
54+
You can install this package on top of an existing airflow 1.10.* installation via
55+
`pip install apache-airflow-backport-providers-apache-beam`
56+
57+
## Cross provider package dependencies
58+
59+
Those are dependencies that might be needed in order to use all the features of the package.
60+
You need to install the specified backport providers package in order to use them.
61+
62+
You can install such cross-provider dependencies when installing from PyPI. For example:
63+
64+
```bash
65+
pip install apache-airflow-beckport-providers-apache-beam[google]
66+
```
67+
68+
| Dependent package | Extra |
69+
|:----------------------------------------------------------------------------------------------------------|:------------|
70+
| [apache-airflow-providers-apache-google](https://pypi.org/project/apache-airflow-providers-apache-google) | google |
71+
72+
73+
# Provider classes summary
74+
75+
In Airflow 2.0, all operators, transfers, hooks, sensors, secrets for the `apache.beam` provider
76+
are in the `airflow.providers.apache.beam` package. You can read more about the naming conventions used
77+
in [Naming conventions for provider packages](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#naming-conventions-for-provider-packages)
78+
79+
80+
## Operators
81+
82+
### New operators
83+
84+
| New Airflow 2.0 operators: `airflow.providers.apache.beam` package |
85+
|:-----------------------------------------------------------------------------------------------------------------------------------------------|
86+
| [operators.beam.BeamRunJavaPipelineOperator](https://github.com/apache/airflow/blob/master/airflow/providers/apache/beam/operators/beam.py) |
87+
| [operators.beam.BeamRunPythonPipelineOperator](https://github.com/apache/airflow/blob/master/airflow/providers/apache/beam/operators/beam.py) |
88+
89+
90+
## Hooks
91+
92+
### New hooks
93+
94+
| New Airflow 2.0 hooks: `airflow.providers.apache.beam` package |
95+
|:-----------------------------------------------------------------------------------------------------------------|
96+
| [hooks.beam.BeamHook](https://github.com/apache/airflow/blob/master/airflow/providers/apache/beam/hooks/beam.py) |
97+
98+
99+
## Releases

‎airflow/providers/apache/beam/CHANGELOG.rst‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one
2+
or more contributor license agreements. See the NOTICE file
3+
distributed with this work for additional information
4+
regarding copyright ownership. The ASF licenses this file
5+
to you under the Apache License, Version 2.0 (the
6+
"License"); you may not use this file except in compliance
7+
with the License. You may obtain a copy of the License at
8+
9+
.. http://www.apache.org/licenses/LICENSE-2.0
10+
11+
.. Unless required by applicable law or agreed to in writing,
12+
software distributed under the License is distributed on an
13+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
KIND, either express or implied. See the License for the
15+
specific language governing permissions and limitations
16+
under the License.
17+
18+
19+
Changelog
20+
---------
21+
22+
1.0.0
23+
.....
24+
25+
Initial version of the provider.

‎airflow/providers/apache/beam/README.md‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
21+
# Package apache-airflow-providers-apache-beam
22+
23+
Release: 0.0.1
24+
25+
**Table of contents**
26+
27+
- [Provider package](#provider-package)
28+
- [Installation](#installation)
29+
- [PIP requirements](#pip-requirements)
30+
- [Cross provider package dependencies](#cross-provider-package-dependencies)
31+
- [Provider class summary](#provider-classes-summary)
32+
- [Operators](#operators)
33+
- [Transfer operators](#transfer-operators)
34+
- [Hooks](#hooks)
35+
- [Releases](#releases)
36+
37+
## Provider package
38+
39+
This is a provider package for `apache.beam` provider. All classes for this provider package
40+
are in `airflow.providers.apache.beam` python package.
41+
42+
## Installation
43+
44+
NOTE!
45+
46+
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
47+
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
48+
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
49+
`pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3, you need to add option
50+
`--use-deprecated legacy-resolver` to your pip install command.
51+
52+
You can install this package on top of an existing airflow 2.* installation via
53+
`pip install apache-airflow-providers-apache-beam`
54+
55+
## Cross provider package dependencies
56+
57+
Those are dependencies that might be needed in order to use all the features of the package.
58+
You need to install the specified backport providers package in order to use them.
59+
60+
You can install such cross-provider dependencies when installing from PyPI. For example:
61+
62+
```bash
63+
pip install apache-airflow-providers-apache-beam[google]
64+
```
65+
66+
| Dependent package | Extra |
67+
|:--------------------------------------------------------------------------------------------|:------------|
68+
| [apache-airflow-providers-google](https://pypi.org/project/apache-airflow-providers-google) | google |
69+
70+
71+
# Provider classes summary
72+
73+
In Airflow 2.0, all operators, transfers, hooks, sensors, secrets for the `apache.beam` provider
74+
are in the `airflow.providers.apache.beam` package. You can read more about the naming conventions used
75+
in [Naming conventions for provider packages](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#naming-conventions-for-provider-packages)
76+
77+
78+
## Operators
79+
80+
### New operators
81+
82+
| New Airflow 2.0 operators: `airflow.providers.apache.beam` package |
83+
|:-----------------------------------------------------------------------------------------------------------------------------------------------|
84+
| [operators.beam.BeamRunJavaPipelineOperator](https://github.com/apache/airflow/blob/master/airflow/providers/apache/beam/operators/beam.py) |
85+
| [operators.beam.BeamRunPythonPipelineOperator](https://github.com/apache/airflow/blob/master/airflow/providers/apache/beam/operators/beam.py) |
86+
87+
88+
## Hooks
89+
90+
### New hooks
91+
92+
| New Airflow 2.0 hooks: `airflow.providers.apache.beam` package |
93+
|:-----------------------------------------------------------------------------------------------------------------|
94+
| [hooks.beam.BeamHook](https://github.com/apache/airflow/blob/master/airflow/providers/apache/beam/hooks/beam.py) |
95+
96+
97+
## Releases

‎airflow/providers/apache/beam/__init__.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.

‎airflow/providers/apache/beam/example_dags/__init__.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.

0 commit comments

Comments
 (0)