🎉 New source: Babelforce by mohamagdy · Pull Request #12700 · airbytehq/airbyte

Mohamed Magdy (mohamagdy) · 2022-05-09T12:45:59Z

What

Introduce Babelforce Source

🚨 User Impact 🚨

This is a new source. No user impact is expected.

Pre-merge Checklist

Community member or Airbyter

Tests

Unit

Test session starts (platform: darwin, Python 3.9.7, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/mohamedmagdy/projects/airbyte/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_call_stream.py::test_request_params ✓                                                                                                                         6% ▋
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_call_stream.py::test_parse_response ✓                                                                                                                        12% █▍
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_incremental_streams.py::test_cursor_field ✓                                                                                                                  19% █▉
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_incremental_streams.py::test_get_updated_state ✓                                                                                                             25% ██▌
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_incremental_streams.py::test_supports_incremental ✓                                                                                                          31% ███▎
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_incremental_streams.py::test_source_defined_cursor ✓                                                                                                         38% ███▊
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_source.py::test_check_connection ✓                                                                                                                           44% ████▍
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_source.py::test_streams ✓                                                                                                                                    50% █████
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_streams.py::test_next_page_token ✓                                                                                                                           56% █████▋
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_streams.py::test_request_headers ✓                                                                                                                           62% ██████▍
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_streams.py::test_http_method ✓                                                                                                                               69% ██████▉
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_streams.py::test_should_retry[HTTPStatus.OK-False] ✓                                                                                                         75% ███████▌
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_streams.py::test_should_retry[HTTPStatus.BAD_REQUEST-False] ✓                                                                                                81% ████████▎
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_streams.py::test_should_retry[HTTPStatus.TOO_MANY_REQUESTS-True] ✓                                                                                           88% ████████▊
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_streams.py::test_should_retry[HTTPStatus.INTERNAL_SERVER_ERROR-True] ✓                                                                                       94% █████████▍
 airbyte-integrations/connectors/source-babelforce/unit_tests/test_streams.py::test_backoff_time ✓                                                                                                                             100% ██████████
============================================================================================================== warnings summary ==============================================================================================================
../../../../../../.pyenv/versions/3.9.7/envs/source-babelforce/lib/python3.9/site-packages/airbyte_cdk/sources/utils/transform.py:12
  /Users/mohamedmagdy/.pyenv/versions/3.9.7/envs/source-babelforce/lib/python3.9/site-packages/airbyte_cdk/sources/utils/transform.py:12: DeprecationWarning: Call to deprecated class AirbyteLogger. (Use logging.getLogger('airbyte') instead) -- Deprecated since version 0.1.47.
    logger = AirbyteLogger()

../../../../../../.pyenv/versions/3.9.7/envs/source-babelforce/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/rate_limiting.py:19
  /Users/mohamedmagdy/.pyenv/versions/3.9.7/envs/source-babelforce/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/rate_limiting.py:19: DeprecationWarning: Call to deprecated class AirbyteLogger. (Use logging.getLogger('airbyte') instead) -- Deprecated since version 0.1.47.
    logger = AirbyteLogger()

../../../../../../.pyenv/versions/3.9.7/envs/source-babelforce/lib/python3.9/site-packages/airbyte_cdk/utils/event_timing.py:13
  /Users/mohamedmagdy/.pyenv/versions/3.9.7/envs/source-babelforce/lib/python3.9/site-packages/airbyte_cdk/utils/event_timing.py:13: DeprecationWarning: Call to deprecated class AirbyteLogger. (Use logging.getLogger('airbyte') instead) -- Deprecated since version 0.1.47.
    logger = AirbyteLogger()

airbyte-integrations/connectors/source-babelforce/unit_tests/test_call_stream.py: 2 warnings
airbyte-integrations/connectors/source-babelforce/unit_tests/test_incremental_streams.py: 4 warnings
airbyte-integrations/connectors/source-babelforce/unit_tests/test_source.py: 2 warnings
airbyte-integrations/connectors/source-babelforce/unit_tests/test_streams.py: 8 warnings
  /Users/mohamedmagdy/.pyenv/versions/3.9.7/envs/source-babelforce/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:43: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator: HttpAuthenticator = NoAuth()

airbyte-integrations/connectors/source-babelforce/unit_tests/test_call_stream.py: 2 warnings
airbyte-integrations/connectors/source-babelforce/unit_tests/test_incremental_streams.py: 4 warnings
airbyte-integrations/connectors/source-babelforce/unit_tests/test_source.py: 2 warnings
airbyte-integrations/connectors/source-babelforce/unit_tests/test_streams.py: 8 warnings
  /Users/mohamedmagdy/.pyenv/versions/3.9.7/envs/source-babelforce/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

airbyte-integrations/connectors/source-babelforce/unit_tests/test_source.py::test_check_connection
airbyte-integrations/connectors/source-babelforce/unit_tests/test_source.py::test_streams
  /Users/mohamedmagdy/projects/airbyte/airbyte/airbyte-integrations/connectors/source-babelforce/source_babelforce/source.py:148: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return BabelforceAuthenticator(access_key_id=access_key_id, access_token=access_token)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.92s):
      16 passed

Integration

cachedir: .pytest_cache
rootdir: /Users/mohamedmagdy/projects/airbyte/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...

Results (0.03s):

Acceptance

cachedir: .pytest_cache
rootdir: /Users/mohamedmagdy/projects/airbyte/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_config_match_spec[inputs0] ✓                                                                    5% ▌
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓                                                                       9% ▉
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_docker_env[inputs0] ✓                                                                          14% █▍
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oneof_usage[inputs0] ✓                                                                         18% █▊
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓                                                                            23% ██▍
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓                                                                            27% ██▊
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓                                                                          32% ███▎
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓                                                          36% ███▋
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_defined_refs_exist_in_json_spec_file[inputs0] ✓                                                41% ████▏
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓                                                               45% ████▋
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓                                                                         50% █████
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓                                                                         55% █████▌
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓                                                                       59% █████▉
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓                                                64% ██████▍
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓                                                   68% ██████▊
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-allOf] ✓                                          73% ███████▍
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-not] ✓                                            77% ███████▊
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_primary_keys_exist_in_schema[inputs0] ✓                                                   82% ████████▎
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓                                                                           86% ████████▋
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs0] ✓                                                     91% █████████▏
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs0] ✓                                                  95% █████████▋
 ../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs0] s                                   100% ██████████
{"type": "LOG", "log": {"level": "INFO", "message": "/Users/mohamedmagdy/projects/airbyte/airbyte/airbyte-integrations/connectors/source-babelforce - SAT run - e3de55e402e6f79cd0837fc7f43116cc5ea72cd2 - PASSED"}}

========================================================================================================== short test summary info ===========================================================================================================
SKIPPED [1] ../../../../../canary/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py:21: `future_state_path` not specified, skipping

Results (51.61s):
      21 passed
       1 skipped

Augustin (alafanechere) · 2022-05-16T21:02:15Z

Thank you for this contribution Mohamed Magdy (@mohamagdy) , we need a test account for Babelforce to test your code. I created an issue here for this and our team will review your contribution asap.

Mohamed Magdy (mohamagdy) · 2022-05-16T21:34:07Z

Thank you for this contribution Mohamed Magdy (@mohamagdy) , we need a test account for Babelforce to test your code. I created an issue here for this and our team will review your contribution asap.

Thanks Augustin (@alafanechere). I just asked Babelforce if they can share a test/sandbox account to test the changes. I will keep you posted.

Augustin (alafanechere)

I made a quick high-level review, I have minor comments. Thanks for the quality code ;) We will come back with a final review once we have sandbox credentials.

airbyte-integrations/connectors/source-babelforce/source_babelforce/spec.json

airbyte-integrations/connectors/source-babelforce/source_babelforce/schemas/calls.json

airbyte-integrations/connectors/source-babelforce/source_babelforce/source.py

Mohamed Magdy (mohamagdy) · 2022-05-18T13:32:36Z

Augustin (@alafanechere) thanks for your review. I will keep you posted once I get a sandbox credentials for testing.

Augustin (alafanechere) · 2022-05-19T18:59:59Z

Thank you Mohamed Magdy (@mohamagdy) , we might ask ourselves for a sandbox account, but feel free to share any inputs you might receive from Babelforce (DM on our Slack is better).

Augustin (alafanechere) · 2022-06-02T08:50:17Z

Mohamed Magdy (@mohamagdy) we did not hear back from Babelforce yet, we try to organize a call with them to create a sandbox account (https://github.com/airbytehq/airbyte-internal-issues/issues/620)

Mohamed Magdy (mohamagdy) · 2022-06-02T10:12:01Z

Mohamed Magdy (@mohamagdy) we did not hear back from Babelforce yet, we try to organize a call with them to create a sandbox account (#12890)

Thanks Augustin (@alafanechere) for following up. I also haven't heard from Babelforce. I emailed them twice with no response yet 😕 .

ycherniaiev · 2022-07-14T13:24:27Z

Mohamed Magdy (@mohamagdy) do you use babelforce.If yes name please your organisation.it is needed to receive sandbox

Mohamed Magdy (mohamagdy) · 2022-07-14T14:07:12Z

ycherniaiev yes we use Babelforce and the organization is: Canary

Marcos Marx (marcosmarxm) · 2022-10-06T21:19:36Z

Mohamed Magdy (@mohamagdy) I saw some left commeents made by Augustin, do you want to continue this implementation?

Mohamed Magdy (mohamagdy) · 2022-10-06T21:54:09Z

Marcos Marx (@marcosmarxm) as far as I recall, the sandbox credentials from Babelforce is the blocker, no?

Marcos Marx (marcosmarxm) · 2022-11-22T20:22:27Z

Hello 👋, first thank you for this amazing contribution.

We really appreciate the effort you've made to improve the project.
We ask you patience for the code review. Last month our team was focused on Hacktoberfest event and that probably left some PR without the proper feedback. And this week, due to the Thanksgiving US Holiday, most our team is out of office with their families. Another important piece of information why code won't be merge this week is: as a safety measure the core team has decided to freeze merging code to main branch to keep the release stable. Next week we'll return to you with the proper code review and update the status of your contribution.

If you have any questions feel free to send me a message in Slack!
Thanks!

…orce source spec

In Babelforce source

For Babelforce source

In Babelforce source

Marcos Marx (marcosmarxm) · 2022-12-02T22:13:30Z

/publish connector=connectors/source-babelforce run-tests=false

🕑 Publishing the following connectors:
connectors/source-babelforce
https://github.com/airbytehq/airbyte/actions/runs/3605459252

Connector	Did it publish?	Were definitions generated?
connectors/source-babelforce	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Marcos Marx (marcosmarxm)

Thanks for the contribution Mohamed Magdy (@mohamagdy)

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels May 9, 2022

Octavia Squidington III (octavia-squidington-iii) added the community label May 9, 2022

Mohamed Magdy (mohamagdy) force-pushed the source-babelforce branch from 282c1ad to 4ef315e Compare May 9, 2022 13:59

Augustin (alafanechere) assigned Augustin (alafanechere) and John Wasserman (johnwasserman) May 16, 2022

Augustin (alafanechere) changed the title ~~🎉 Introduce Babelforce source~~ 🎉 New source: Babelforce May 16, 2022

Augustin (alafanechere) added the new-connector label May 16, 2022

Augustin (alafanechere) reviewed May 17, 2022

View reviewed changes

Augustin (alafanechere) added the waiting-for-cred Waiting for CI credentials label May 20, 2022

Brian Leonard (bleonard) added bounty connectors/source/babelforce labels Jun 2, 2022

Augustin (alafanechere) added internal and removed bounty labels Jun 7, 2022

Augustin (alafanechere) removed their assignment Jul 8, 2022

Marcos Marx (marcosmarxm) unassigned John Wasserman (johnwasserman) Aug 11, 2022

Marcos Marx (marcosmarxm) and others added 3 commits December 2, 2022 17:39

solve deleted file

f61c757

Set Babelforce version to 0.1.0

9b6fd7e

Update Babelforce source documentaiton with the PR link

0638a6b

Mohamed Magdy (mohamagdy) added 10 commits December 2, 2022 17:39

Remove .python-version file in Babelforce source from Git

160e363

Fix typo in Babelfoce spec description field

011f095

Remove redaundant title field from JSON schema in Babelforce source

52b4e96

Add examples for the date_created_from and date_created_to Babelf…

43c179e

…orce source spec

Remove redaundant static authenticator method in Babelforce source

29302a3

Check if date_created_from less than date_created_to

46003cc

In Babelforce source

Add parsing details to docstring

04f555f

For Babelforce source

Fix fetching next page token

991b542

In Babelforce source

Remove parent class docstring

0724877

In Babelforce source

Use pendulum to parse date/time

5a4b073

In Babelforce source

Marcos Marx (marcosmarxm) force-pushed the source-babelforce branch from 165edc7 to 5a4b073 Compare December 2, 2022 20:39

Marcos Marx (marcosmarxm) added 3 commits December 2, 2022 17:41

add babelforce to source index

dfbcc6d

remove icon

53d304b

format files

40b6c91

Octavia Squidington III (octavia-squidington-iii) and others added 2 commits December 2, 2022 22:25

auto-bump connector version

eaed4e2

Merge branch 'master' into source-babelforce

4eda154

Marcos Marx (marcosmarxm) approved these changes Dec 2, 2022

View reviewed changes

Marcos Marx (marcosmarxm) merged commit a7f32c0 into airbytehq:master Dec 2, 2022

Mohamed Magdy (mohamagdy) deleted the source-babelforce branch March 14, 2023 14:58

Conversation

Mohamed Magdy (mohamagdy) commented May 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Tests

Uh oh!

Augustin (alafanechere) commented May 16, 2022

Uh oh!

Mohamed Magdy (mohamagdy) commented May 16, 2022

Uh oh!

Augustin (alafanechere) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mohamed Magdy (mohamagdy) commented May 18, 2022

Uh oh!

Augustin (alafanechere) commented May 19, 2022

Uh oh!

Augustin (alafanechere) commented Jun 2, 2022

Uh oh!

Mohamed Magdy (mohamagdy) commented Jun 2, 2022

Uh oh!

ycherniaiev commented Jul 14, 2022

Uh oh!

Mohamed Magdy (mohamagdy) commented Jul 14, 2022

Uh oh!

Marcos Marx (marcosmarxm) commented Oct 6, 2022

Uh oh!

Mohamed Magdy (mohamagdy) commented Oct 6, 2022

Uh oh!

Marcos Marx (marcosmarxm) commented Nov 22, 2022

Uh oh!

Marcos Marx (marcosmarxm) commented Dec 2, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Marcos Marx (marcosmarxm) left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Mohamed Magdy (mohamagdy) commented May 9, 2022 •

edited

Loading

Marcos Marx (marcosmarxm) commented Dec 2, 2022 •

edited by github-actions bot

Loading