🎉 New Source: Convex.dev [python cdk]#18403
🎉 New Source: Convex.dev [python cdk]#18403Marcos Marx (marcosmarxm) merged 51 commits intoairbytehq:masterfrom
Conversation
…urce.py Co-authored-by: Sujay Jayakar <[email protected]>
| @@ -0,0 +1,4 @@ | |||
| { | |||
| "deployment_name": "descriptive-vulture-260", | |||
| "access_key": "Your access token" | |||
There was a problem hiding this comment.
how do i give the access key used in acceptance tests? it's stored in secrets/config.json locally which is gitignored.
Marcos Marx (marcosmarxm)
left a comment
Marcos Marx (marcosmarxm)
left a comment
There was a problem hiding this comment.
Some comments.
| - name: Convex | ||
| sourceDefinitionId: c332628c-f55c-4017-8222-378cfafda9b2 | ||
| dockerRepository: airbyte/source-convex | ||
| dockerImageTag: dev |
There was a problem hiding this comment.
The image is generated during the review process. Someone will trigger CI actions to test and publish the docker image to Dockerhub.
Yes. the image must be tagged 0.1.0
| | Chartmogul | [](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-chartmogul/) | | ||
| | Cart.com | [](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-cart/) | | ||
| | Close.com | [](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-close-com/) | | ||
| | Convex | [](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-convex/) | |
There was a problem hiding this comment.
During the review process you're going to see the CI results.
airbyte-integrations/connectors/source-convex/integration_tests/catalog.json
Outdated
Show resolved
Hide resolved
| @@ -0,0 +1,19 @@ | |||
| documentationUrl: https://docs.airbyte.com/integrations/sources/convex | |||
airbyte-integrations/connectors/source-convex/source_convex/spec.yaml
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-convex/source_convex/source.py
Outdated
Show resolved
Hide resolved
| if self._snapshot_has_more: | ||
| return "/api/list_snapshot" | ||
| else: | ||
| return "/api/document_deltas" |
There was a problem hiding this comment.
can you link those endpoints to convex.dev api docs?
There was a problem hiding this comment.
we don't have docs for these endpoints yet, because i just created them recently. i'll work on exposing documentation for them now
There was a problem hiding this comment.
linked to documentation here and in bootstrap.md
|
Lee Danilek (@ldanilek) please ask my review again after all changes are resolved. |
|
Marcos Marx (@marcosmarxm) i've updated the PR so it should be ready for re-review. thanks! |
|
Awesome, Lee Danilek (@ldanilek) do you have test credentials to share? |
here they are (link expires in a week) |
Marcos Marx (marcosmarxm)
left a comment
Marcos Marx (marcosmarxm)
left a comment
There was a problem hiding this comment.
Thanks Lee Danilek (@ldanilek) I'll publish the connector later today!
|
Hello! Your PR is approved but didn't have the time to publish and merge it this week. As you can check in Chris' comment all PRs submitted before 2-nov are eligible to win the prize. I'll be out of the office on Friday and return Monday to start publishing your contribution. Any question you can send a message in Have a good weekend and thank you for this amazing contribution for Hacktoberfest 🎉 |
|
Hi Marcos Marx (@marcosmarxm) . Do you have a timeline for when this connector can be deployed? I'm not worried about the hacktober prize. I'm trying to launch this feature for customers of Convex, where I work. |
|
/test connector=connectors/source-convex
Build PassedTest summary info: |
|
/publish connector=connectors/source-convex
if you have connectors that successfully published but failed definition generation, follow step 4 here |
Chris "Sean" Dabatos (RealChrisSean)
added
the
python
Marcos Marx (marcosmarxm)
changed the title
* convex source * fixed unit tests * use the smaller test instance * fix integration tests and pagination * fix tableName casing in api call * fix logo * more shapes -- thinking about moving this to an internal api * use new /json_schemas api * use new APIs * remove unused start_date * doc skeleton * fix deletes with incremental sync * fix test of json_schema * fix expected records * merge * version and docs * fix test * fix docs url * mention that you need to ask convex to enable it for your account * docs * two stage pagination for initial sync. some unit tests not working * fixed all the unit tests, but we need to push backends before acceptance tests work * fix snapshot pagination * fix integration tests * some checkboxes from the PR description * Update airbyte-integrations/connectors/source-convex/source_convex/source.py Co-authored-by: Sujay Jayakar <[email protected]> * Update airbyte-integrations/connectors/source-convex/source_convex/source.py Co-authored-by: Sujay Jayakar <[email protected]> * stop storing the delta_has_more in state * fix mypy * use UTC timestamps * rename instance_name to deployment_name * include data format request param * remove dev docker image * fix unit test * documentation * . * code review * documentation and change deployment_name to deployment_url * fix pip dependency * regenerate the spec file Co-authored-by: Sujay Jayakar <[email protected]>
6 participants
What
Hacktoberfest: airbytehq/connector-contest#225
I work at Convex, where we're building a new backend as a service platform, and we'd like to have a connector between Convex's database and Airbyte.
Create a new source connector to export data from Convex (https://convex.dev) into analytics tools.
Screenshots of the connector running locally:
How
/json_schemas,/list_snapshot,/document_deltascheck,discover, andreadworkRecommended reading order
source_convex/spec.yamlsource_convex/source.pydocs/integrations/sources/convex.mdsource-convex/bootstrap.mdresources/seed/source_definitions.yaml🚨 User Impact 🚨
New source connector in release stage "alpha".
Pre-merge Checklist
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampledocs/integrations/README.mdairbyte-integrations/builds.mdAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing/publishcommand described hereTests
Unit
Put your unit tests output here.
Integration
Put your integration tests output here.
no custom integration tests
Acceptance
from pytest:
I don't understand that
ConfiguredAirbyteCatalogerror. it doesn't say which value it doesn't like, andsource-convex/integration_tests/configured_catalog.jsonhas valid values, as far as i can tell. also this error doesn't show up when running the same tests in gradle.From gradle: