🎉 New Destination: Timeplus by jovezhong · Pull Request #21226 · airbytehq/airbyte

Jove Zhong (jovezhong) · 2023-01-10T19:52:59Z

(this is to replace #20391 Cannot share secrets to maintainer if the PR is sent from a github org not personal account)

What

A new destination connector for Timeplus (works for any cloud deployment or onprem deployment of Timeplus)

close #20195

How

This python based destination connector will send data to Timeplus. It will create streams in Timeplus if necessary, with the exactly same schema from the source (kind of normalization, but not based on DBT). For example, if there are 4 columns in the CSV, then there will be 4 columns in the Timeplus stream.

🚨 User Impact 🚨

no breaking change

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

To test this connector, you can create a free account at https://us.timeplus.cloud, sign in with Google/MS SSO, then create a workspace, create an API key, then set them in the secret/UI to push data to it (tested with CSV and db)

Tests

Unit

pytest -s unit_tests
=================================================== test session starts ===================================================
platform darwin -- Python 3.9.16, pytest-7.2.0, pluggy-1.0.0 -- /Users/jove/Dev/jovezhong/airbyte/airbyte-integrations/connectors/destination-timeplus/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/jove/Dev/jovezhong/airbyte, configfile: pytest.ini
collected 1 item

unit_tests/unit_test.py::test_type_mapping PASSED

=================================================== 1 passed in 0.71s ===================================================

Integration

=================================================== test session starts ===================================================
platform darwin -- Python 3.9.16, pytest-7.2.0, pluggy-1.0.0 -- /Users/jove/Dev/jovezhong/airbyte/airbyte-integrations/connectors/destination-timeplus/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/jove/Dev/jovezhong/airbyte, configfile: pytest.ini
collected 2 items

integration_tests/integration_test.py::test_check_valid_config {"type": "LOG", "log": {"level": "INFO", "message": "Successfully connected to https://dev.timeplus.cloud/latest"}}
PASSED
integration_tests/integration_test.py::test_check_invalid_config PASSED

Acceptance

Put your acceptance tests output here.

Jove Zhong (jovezhong) · 2023-02-07T19:46:48Z

Any maintainer look into this PR? It has been quite a while

Jove Zhong (jovezhong) · 2023-02-18T17:47:04Z

still waiting for a reviewer

Jove Zhong (jovezhong) · 2023-02-27T19:58:49Z

Again, waiting for reviewers. What is the process to assign such review tickets?

CLAassistant · 2023-02-27T21:38:44Z

All committers have signed the CLA.

airbyte-integrations/connectors/destination-timeplus/destination_timeplus/spec.json

airbyte-integrations/connectors/destination-timeplus/destination_timeplus/destination.py

Jove Zhong (jovezhong) · 2023-03-01T01:52:38Z

Hi Marcos Marx (@marcosmarxm) , thanks for the reviews. I am not quite sure about the process. I resolved the review comments. Feel free to reopen them if they need more changes. I added 3 commits to address your review comments one by one. Hopefully it should make it a bit easier to check the code change.

Jove Zhong (jovezhong) · 2023-03-07T19:37:08Z

Hi Marcos Marx (@marcosmarxm), could you please take the other look? Thanks

Jove Zhong (jovezhong) · 2023-03-15T22:58:55Z

I don't want to ask this every week, but please please give some feedback Marcos Marx (@marcosmarxm)

Marcos Marx (marcosmarxm) · 2023-03-16T01:12:39Z

Sorry the long delay Jove Zhong (@jovezhong) I added this to our team backlog. Hope to have a slot this week.

Marcos Marx (marcosmarxm) · 2023-06-08T16:17:23Z

Jove Zhong (@jovezhong) can you fix:
❌ - check_documentation_file_exists
❌ - check_changelog_entry_is_updated
❌ - check_connector_icon_is_available

Jove Zhong (jovezhong) · 2023-06-08T16:21:23Z

Sure, Marcos Marx (@marcosmarxm) , happy to add those files. I checked the log and cannot find the exact file they are looking for? Can you share any example or docs about those 3 checks? Thanks.

❌ - check_documentation_file_exists
❌ - check_changelog_entry_is_updated
❌ - check_connector_icon_is_available

Jove Zhong (jovezhong) · 2023-06-08T16:49:03Z

Never mind. I got the checker code tools/ci_connector_ops/ci_connector_ops/qa_checks.py
I will add commits soon

boolean and pokemon

Marcos Marx (marcosmarxm) · 2023-06-14T20:10:03Z

/test connector=connectors/destination-timeplus

⚠️ The test slash command is now deprecated.

The connector tests are automatically triggered as CI checks.

Please use /legacy-test if you need to test CDK or CAT changes.

Please join post to #pipeline-complaint-hotline slack channel if something is not working as expected.

Jove Zhong (jovezhong) · 2023-06-14T20:56:01Z

Thank you Marcos Marx (@marcosmarxm) for the review and merge!

Octavia Squidington IV (octavia-squidington-iv) added area/connectors Connector related issues bounty community connectors/destination/timeplus labels Jan 10, 2023

Jove Zhong (jovezhong) mentioned this pull request Jan 10, 2023

🎉 New Destination: Timeplus #20391

Closed

37 tasks

Jove Zhong (jovezhong) force-pushed the feature/timeplus-destination branch 2 times, most recently from 176efde to b74d546 Compare January 17, 2023 22:07

Jove Zhong (jovezhong) force-pushed the feature/timeplus-destination branch from 79301fd to eed848b Compare January 31, 2023 01:36

Jove Zhong (jovezhong) force-pushed the feature/timeplus-destination branch from eed848b to d3decf2 Compare February 7, 2023 19:46

Jove Zhong (jovezhong) force-pushed the feature/timeplus-destination branch from d3decf2 to d72ccd8 Compare February 18, 2023 17:47

Jove Zhong (jovezhong) force-pushed the feature/timeplus-destination branch from 49f155c to bdcb91d Compare February 28, 2023 01:00

Marcos Marx (marcosmarxm) suggested changes Feb 28, 2023

View reviewed changes

Marcos Marx (marcosmarxm) self-assigned this Feb 28, 2023

Jove Zhong (jovezhong) force-pushed the feature/timeplus-destination branch 2 times, most recently from 6c13262 to 438df17 Compare March 1, 2023 01:50

Jove Zhong (jovezhong) force-pushed the feature/timeplus-destination branch from 438df17 to ee47e7c Compare March 1, 2023 16:17

Jove Zhong (jovezhong) requested a review from Marcos Marx (marcosmarxm) March 1, 2023 16:32

Jove Zhong (jovezhong) force-pushed the feature/timeplus-destination branch 3 times, most recently from 9e60a78 to 37fb4a4 Compare March 7, 2023 19:36

Jove Zhong (jovezhong) force-pushed the feature/timeplus-destination branch from 37fb4a4 to 37a1803 Compare March 15, 2023 22:58

Marcos Marx (marcosmarxm) requested a review from a team March 16, 2023 01:11

Jove Zhong (jovezhong) force-pushed the feature/timeplus-destination branch from fc6edb6 to d8cbfa1 Compare June 8, 2023 16:21

Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label Jun 8, 2023

Joan Rojas (jrolom) removed the bounty label Jun 9, 2023

Jove Zhong and others added 14 commits June 12, 2023 13:54

timeplus destination connector to push data to Timeplus via REST API/SDK

15e3182

0.2.0, support nullable columns, add more test data

280781b

boolean and pokemon

mark the apikey as secret

428ede5

us.timeplus.cloud as sample URL

c635388

removed the unnecessary pass

9f77064

rewrite the logic of when to delete/create stream

1d140f5

format code, following the PEP-8

812ee97

add write integration test

eee0b9b

replacing AirbyteLogger() with logging.getLogger('airbyte')

0295d96

add logs to destination.py, refine integration test

1d5e784

add docs/integrations/destinations/timeplus.md

8519cbe

add icon.svg

7669e94

set to 0.4.0 version

01c4d89

upgrade timeplus sdk from 1.1.1 to 1.2.1

b4dbe37

Jove Zhong (jovezhong) force-pushed the feature/timeplus-destination branch from 3e41a0c to b4dbe37 Compare June 12, 2023 20:54

Jove Zhong (jovezhong) requested a review from Marcos Marx (marcosmarxm) June 12, 2023 20:55

Marcos Marx (marcosmarxm) and others added 4 commits June 14, 2023 17:11

Merge branch 'master' into feature/timeplus-destination

c12dab6

add metadata file

e90ef53

Merge branch 'master' into feature/timeplus-destination

efbd650

fix docker version

9aa8851

Marcos Marx (marcosmarxm) approved these changes Jun 14, 2023

View reviewed changes

Marcos Marx (marcosmarxm) merged commit e63d371 into airbytehq:master Jun 14, 2023

Conversation

Jove Zhong (jovezhong) commented Jan 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

Uh oh!

Jove Zhong (jovezhong) commented Feb 7, 2023

Uh oh!

Jove Zhong (jovezhong) commented Feb 18, 2023

Uh oh!

Jove Zhong (jovezhong) commented Feb 27, 2023

Uh oh!

CLAassistant commented Feb 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jove Zhong (jovezhong) commented Mar 1, 2023

Uh oh!

Jove Zhong (jovezhong) commented Mar 7, 2023

Uh oh!

Jove Zhong (jovezhong) commented Mar 15, 2023

Uh oh!

Marcos Marx (marcosmarxm) commented Mar 16, 2023

Uh oh!

Marcos Marx (marcosmarxm) commented Jun 8, 2023

Uh oh!

Jove Zhong (jovezhong) commented Jun 8, 2023

Uh oh!

Jove Zhong (jovezhong) commented Jun 8, 2023

Uh oh!

Marcos Marx (marcosmarxm) commented Jun 14, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jove Zhong (jovezhong) commented Jun 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Jove Zhong (jovezhong) commented Jan 10, 2023 •

edited

Loading

CLAassistant commented Feb 27, 2023 •

edited

Loading

Marcos Marx (marcosmarxm) commented Jun 14, 2023 •

edited by github-actions bot

Loading