🎉 New Destination: Timeplus by jovezhong · Pull Request #20391 · airbytehq/airbyte

Jove Zhong (jovezhong) · 2022-12-12T21:50:13Z

What

A new destination connector for Timeplus (works for any cloud deployment or onprem deployment of Timeplus)

close #20195

How

This python based destination connector will send data to Timeplus. It will create streams in Timeplus if necessary, with the exactly same schema from the source (kind of normalization, but not based on DBT). For example, if there are 4 columns in the CSV, then there will be 4 columns in the Timeplus stream.

🚨 User Impact 🚨

no breaking change

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

To test this connector, you can create a free account at https://beta.timeplus.cloud, sign in with Google/MS SSO, then create a workspace, create an API key, then set them in the secret/UI to push data to it (tested with CSV and db)

Tests

Unit

pytest -s unit_tests
================================================== test session starts ==================================================
platform darwin -- Python 3.9.16, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /Users/jove/Dev/timeplus-io/airbyte/airbyte-integrations/connectors/destination-timeplus/.venv/bin/python3.9
cachedir: .pytest_cache
rootdir: /Users/jove/Dev/timeplus-io/airbyte, configfile: pytest.ini
collected 1 item

unit_tests/unit_test.py::test_type_mapping PASSED

=================================================== 1 passed in 0.71s ===================================================

Integration

================================================== test session starts ==================================================
platform darwin -- Python 3.9.16, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /Users/jove/Dev/timeplus-io/airbyte/airbyte-integrations/connectors/destination-timeplus/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/jove/Dev/timeplus-io/airbyte, configfile: pytest.ini
collected 0 items

================================================= no tests ran in 0.01s =================================================

Acceptance

Put your acceptance tests output here.

CLAassistant · 2022-12-12T21:50:18Z

All committers have signed the CLA.

Jove Zhong (jovezhong) · 2022-12-15T20:29:40Z

Marcos Marx (@marcosmarxm) , would you please share your review feedbacks? Thanks

Ivica Taseski (itaseskii) · 2022-12-20T08:32:19Z

Jove Zhong (@jovezhong) Is this work in progress? I don't see any test coverage in the PR.

Jove Zhong (jovezhong) · 2022-12-20T16:40:17Z

Jove Zhong (@jovezhong) Is this work in progress? I don't see any test coverage in the PR.

Hi Ivica Taseski (@itaseskii) , thanks for the comments (during holiday break :))
The PR is ready to review. There is one unit test and I checked in the sample messages and run command line to push them to remote Timeplus. Sorry that part is not automated in the integration test, since I am now sure how the airbyte CI will handle the sensitive information such as api token on Timeplus Cloud (feel free to create a free account on timeplus.cloud then create an api key)

If you can guide me how to configure CI to store/fetch the secrets and other integration test in other sinks, I will be more than happy to add the integration test in this week.

Again, this is my first PR to airbyte community. Sorry if I miss anything. Thanks for the review and happy holiday.

Marcos Marx (marcosmarxm) · 2022-12-20T19:53:00Z

Hello 👋:skin-tone-2: and thank you for your contribution!

Airbyte has instituted a code freeze between 19 and 30 December, to make sure there are no disruptions during the holidays.
Because of this, reviewing and merging your contribution may take longer than usual.
We apologize for the delay, but we want everyone to have a quiet and happy holiday.

If you have any questions or need further clarification, please don't hesitate to ping via Slack.

Jove Zhong (jovezhong) · 2022-12-21T05:51:47Z

Sure, no problem.

In my previous organization, we also had similar rules that no deployment on Friday afternoon or right before the holiday. Otherwise we may just bring more risk/trouble to ourselves, without a strong business need/urgency.

Peace~

Jove Zhong (jovezhong) · 2023-01-05T17:53:00Z

Happy New Year, everyone!

Hi Ivica Taseski (@itaseskii) and Marcos Marx (@marcosmarxm), not sure whether you have bandwidth to review this PR. I made some refinements and added more tests.

Ganesh R (geekwhocodes) · 2023-01-08T11:15:15Z

Sajarin (@sajarin) I can review this PR.

Ivica Taseski (itaseskii) · 2023-01-09T20:46:51Z

Hey Jove Zhong (@jovezhong) have you allowed edits and access to secrets by maintainers on the PR? :)

Jove Zhong (jovezhong) · 2023-01-09T22:25:03Z

Hey Jove Zhong (@jovezhong) have you allowed edits and access to secrets by maintainers on the PR? :)

Hi Ivica Taseski (@itaseskii) , can you guide me where to enable this? You can go https://timeplus.cloud to sign up a free account and generate the API token.

Ivica Taseski (itaseskii) · 2023-01-09T22:46:16Z

Hey Jove Zhong (@jovezhong) have you allowed edits and access to secrets by maintainers on the PR? :)

Hi Ivica Taseski (@itaseskii) , can you guide me where to enable this? You can go https://timeplus.cloud to sign up a free account and generate the API token.

Jove Zhong (@jovezhong) its on the right hand side on the current page. example from a different PR

Jove Zhong (jovezhong) · 2023-01-09T22:50:58Z

Oops, there is no such option "Lock conversation" in my github page

with 1.0 python SDK

Jove Zhong (jovezhong) · 2023-01-10T18:53:54Z

Is that because no code reviewers are assigned to the PR? I am happy to share the secrets for testing

Ivica Taseski (itaseskii) · 2023-01-10T19:35:49Z

Is that because no code reviewers are assigned to the PR? I am happy to share the secrets for testing

Jove Zhong (@jovezhong) no, it has nothing to do with the PR not having a reviewer nor the secrets for testing. I think that you have the similar issue mentioned here #20368 (comment). Most likely you will need to open the PR from a personal repo.

Jove Zhong (jovezhong) · 2023-01-10T19:38:00Z

I see. I will do the same

"I guess the problem is the "personal account", because .. is an org. I will fork the repo into my personal account and create another PR."

It should be ready within half an hour

Jove Zhong (jovezhong) · 2023-01-10T19:53:41Z

Can I close this PR? A new PR is ready from my own account #21226 Pls review

Jove Zhong (jovezhong) · 2023-01-16T18:37:01Z

I am about to close this PR in 48 hours if no one objects. Please review the PR sent from my personal account #21226

Jove Zhong (jovezhong) · 2023-01-17T18:26:20Z

Close this PR. Pls review PR #21226

Octavia Squidington IV (octavia-squidington-iv) added area/connectors Connector related issues bounty community connectors/destination/timeplus labels Dec 12, 2022

Sajarin (sajarin) added the bounty-XL Maintainer program: claimable extra large bounty PR label Dec 19, 2022

Jove Zhong added 9 commits January 10, 2023 10:53

Initial code for Timeplus destination connector

f19d4ab

with 1.0 python SDK

fix SDK call

20c3358

IngestData with string[] for columns

5b59bcf

major enhancement to normalize the data, create multiple columns

fad3266

Update unit_test.py

9a684eb

Update README.md

b46aa65

upgrade to 1.1.0 timeplus python sdk. pin to py 3.9

691b6ee

Update integration_test.py

389a2b2

upgrade to 1.1.1 timeplus sdk

8d41fac

Jove Zhong (jovezhong) mentioned this pull request Jan 10, 2023

🎉 New Destination: Timeplus #21226

Merged

37 tasks

Jove Zhong (jovezhong) closed this Jan 17, 2023

Jove Zhong (jovezhong) deleted the feature/timeplus-destination branch January 17, 2023 18:26

Marcos Marx (marcosmarxm) removed the community label Apr 25, 2023

Conversation

Jove Zhong (jovezhong) commented Dec 12, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

Uh oh!

CLAassistant commented Dec 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jove Zhong (jovezhong) commented Dec 15, 2022

Uh oh!

Ivica Taseski (itaseskii) commented Dec 20, 2022

Uh oh!

Jove Zhong (jovezhong) commented Dec 20, 2022

Uh oh!

Marcos Marx (marcosmarxm) commented Dec 20, 2022

Uh oh!

Jove Zhong (jovezhong) commented Dec 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jove Zhong (jovezhong) commented Jan 5, 2023

Uh oh!

Ganesh R (geekwhocodes) commented Jan 8, 2023

Uh oh!

Ivica Taseski (itaseskii) commented Jan 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jove Zhong (jovezhong) commented Jan 9, 2023

Uh oh!

Ivica Taseski (itaseskii) commented Jan 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jove Zhong (jovezhong) commented Jan 9, 2023

Uh oh!

Jove Zhong (jovezhong) commented Jan 10, 2023

Uh oh!

Ivica Taseski (itaseskii) commented Jan 10, 2023

Uh oh!

Jove Zhong (jovezhong) commented Jan 10, 2023

Uh oh!

Jove Zhong (jovezhong) commented Jan 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jove Zhong (jovezhong) commented Jan 16, 2023

Uh oh!

Jove Zhong (jovezhong) commented Jan 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

CLAassistant commented Dec 12, 2022 •

edited

Loading

Jove Zhong (jovezhong) commented Dec 21, 2022 •

edited

Loading

Ivica Taseski (itaseskii) commented Jan 9, 2023 •

edited

Loading

Ivica Taseski (itaseskii) commented Jan 9, 2023 •

edited

Loading

Jove Zhong (jovezhong) commented Jan 10, 2023 •

edited

Loading