[ISSUE #4618] Add HTTP source connector #4634

Fungx · 2023-12-10T07:09:45Z

Fixes #4618.

Motivation

Add Http source connector.

Modifications

Add Http source connector
Add unit tests for Http source connector

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs. Please refer to http-protocol-binding
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

github-actions

Welcome to the Apache EventMesh community!!
This is your first PR in our project. We're very excited to have you onboard contributing. Your contributions are greatly appreciated!

Please make sure that the changes are covered by tests.
We will be here shortly.
Let us know if you need any help!

Want to get closer to the community?

WeChat Assistant	WeChat Public Account	Slack
		Join Slack Chat

Mailing Lists:

Name	Description	Subscribe	Unsubscribe	Archive
Users	User support and questions mailing list	Subscribe	Unsubscribe	Mail Archives
Development	Development related discussions	Subscribe	Unsubscribe	Mail Archives
Commits	All commits to repositories	Subscribe	Unsubscribe	Mail Archives
Issues	Issues or PRs comments and reviews	Subscribe	Unsubscribe	Mail Archives

pandaapo · 2023-12-10T07:36:07Z

tools/dependency-check/known-dependencies.txt

 checker-qual-3.12.0.jar
 cloudevents-api-2.4.2.jar
 cloudevents-core-2.4.2.jar
+cloudevents-http-vertx-2.3.0.jar


Apart from cloudevents-http-vertx and vertx-web, are there any unnecessary dependencies in the newly added dependencies?

Many newly added dependencies are indirectly imported by vertx. I can't pass the License Check of CI/CD after removing them.

Actually, I'm confused about known-dependencies.txt. Why do we need to manually add dependencies here? Since we are already using a build tool.

Actually, I'm confused about known-dependencies.txt. Why do we need to manually add dependencies here? Since we are already using a build tool.

I don't fully understand either, I only know that the script check-dependencies.sh for checking dependencies will use this file to check. I also want to know the purpose of using this method of checking @ruanwenjun , and whether there is a better way.

We need to use the known-dependencies to recognize which packages are new dependencies and make sure these new dependencies must under the ASF 3rd party license policy. @Fungx @pandaapo

I have manually comfirmed that these newly added dependencies(vertx , netty & cloudevents-http-vertx) are under the ASF 3rd party license policy, since they are all using Apache License 2.0. Is this correct? @xwm1992

.../src/main/java/org/apache/eventmesh/connector/http/source/connector/HttpSourceConnector.java

pandaapo

Would you be willing to write a document(#4601 (comment)) for this connector? If you are willing, they can be written in this PR or in a new PR.

Fungx · 2023-12-10T11:10:31Z

Would you be willing to write a document(#4601 (comment)) for this connector? If you are willing, they can be written in this PR or in a new PR.

Of course. I'll write it in a new PR.

codecov · 2023-12-11T02:51:28Z

Codecov Report

Attention: 28 lines in your changes are missing coverage. Please review.

Comparison is base (ee9ee4c) 16.93% compared to head (2e8ac4a) 17.38%.
Report is 3 commits behind head on master.

Files	Patch %	Lines
...tor/http/source/connector/HttpSourceConnector.java	74.07%	11 Missing and 3 partials ⚠️
...tmesh/connector/http/server/HttpConnectServer.java	0.00%	7 Missing ⚠️
...ntmesh/connector/http/config/HttpServerConfig.java	0.00%	4 Missing ⚠️
...ctor/http/source/config/SourceConnectorConfig.java	60.00%	2 Missing ⚠️
...connector/http/source/config/HttpSourceConfig.java	66.66%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #4634      +/-   ##
============================================
+ Coverage     16.93%   17.38%   +0.45%     
- Complexity     1708     1757      +49     
============================================
  Files           790      797       +7     
  Lines         29526    29770     +244     
  Branches       2550     2573      +23     
============================================
+ Hits           5001     5177     +176     
- Misses        24061    24115      +54     
- Partials        464      478      +14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pandaapo

LGTM. Welcome!

hhuang1231 · 2023-12-11T09:39:19Z

Maybe this question shouldn't be asked here, but I still want to know why most sourceConnectors don't implement the commit() method.

pandaapo · 2023-12-11T09:54:20Z

Maybe this question shouldn't be asked here, but I still want to know why most sourceConnectors don't implement the commit() method.

@hhuang1231
commit() is mainly used to record the offset of messages that need to be persisted, and to avoid duplicate transmission of such messages as much as possible. However, not all connectors need to implement it, such as this Connector and the Lark Connector you refactored. Connectors such as RocketMQ, RabbitMQ, Kafka, MySQL, etc. should implement to make them more reliable. Of course, there are currently some connectors that need to implement it but have not implemented. The community wants to gradually improve the functionality of a Connector and does not require contributors to implement all functions in one PR.

commit()是主要用于记录需要持久化的消息的位移，尽量避免这类消息的重复传输，但不是所有的Connector都需要实现，像该Connector、您重构的Lark Connector等不需要实现。像RocketMQ、RabbitMQ、Kafka、MySQL等这类的Connector，应该实现，这样Connector会更加可靠。当然，目前有一些需要实现的Connector并没有实现，社区是想逐步完善一个Connector的功能，不强制要求贡献者这在一次PR中实现所有功能。

Pil0tXia

Good job.

Pil0tXia · 2023-12-11T11:25:44Z

.../src/main/java/org/apache/eventmesh/connector/http/source/connector/HttpSourceConnector.java

+
+    private void doInit() throws Exception {
+        this.queue = new LinkedBlockingQueue<>(1000);


Redundant throws Exception at doInit(), start() and stop() can be removed.

Fixed. And now the trace stack will be logged by the outside SourceWorker.

Pil0tXia

LGTM

pandaapo · 2023-12-27T04:42:49Z

Would you be willing to write a document(#4601 (comment)) for this connector? If you are willing, they can be written in this PR or in a new PR.

Of course. I'll write it in a new PR.

@Fungx In the future, community will mainly maintain documents in eventmesh-site. Therefore, if it is convenient for you, please write the new English documents in this directory of eventmesh-site, and Chinese documents in the directory.

Fungx · 2023-12-28T05:35:22Z

Would you be willing to write a document(#4601 (comment)) for this connector? If you are willing, they can be written in this PR or in a new PR.

Of course. I'll write it in a new PR.

@Fungx In the future, community will mainly maintain documents in eventmesh-site. Therefore, if it is convenient for you, please write the new English documents in this directory of eventmesh-site, and Chinese documents in the directory.

Sure. Leave it to me.

* [ISSUE apache#4618] Add HTTP source connector * fix exception & config

[ISSUE apache#4618] Add HTTP source connector

115e599

github-actions bot reviewed Dec 10, 2023

View reviewed changes

pandaapo reviewed Dec 10, 2023

View reviewed changes

pandaapo previously approved these changes Dec 11, 2023

View reviewed changes

Pil0tXia requested changes Dec 11, 2023

View reviewed changes

Fungx dismissed pandaapo’s stale review via 7233d26 December 11, 2023 13:07

Fungx force-pushed the feat_http_source branch 3 times, most recently from 2589415 to cf49f64 Compare December 11, 2023 13:36

fix exception & config

2e8ac4a

Fungx force-pushed the feat_http_source branch from cf49f64 to 2e8ac4a Compare December 11, 2023 13:47

Pil0tXia approved these changes Dec 11, 2023

View reviewed changes

pandaapo approved these changes Dec 11, 2023

View reviewed changes

xwm1992 approved these changes Dec 12, 2023

View reviewed changes

xwm1992 merged commit a531968 into apache:master Dec 12, 2023

xwm1992 added this to the 1.10 milestone Dec 12, 2023

xuhongjia pushed a commit to Deckers-Ohana/eventmesh that referenced this pull request Mar 13, 2025

[ISSUE apache#4618] Add HTTP source connector (apache#4634)

97e69d0

* [ISSUE apache#4618] Add HTTP source connector * fix exception & config


		private void doInit() throws Exception {
		this.queue = new LinkedBlockingQueue<>(1000);

[ISSUE #4618] Add HTTP source connector #4634

[ISSUE #4618] Add HTTP source connector #4634

Uh oh!

Conversation

Fungx commented Dec 10, 2023

Motivation

Modifications

Documentation

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

pandaapo Dec 10, 2023

Choose a reason for hiding this comment

Uh oh!

Fungx Dec 10, 2023

Choose a reason for hiding this comment

Uh oh!

pandaapo Dec 10, 2023

Choose a reason for hiding this comment

Uh oh!

xwm1992 Dec 12, 2023

Choose a reason for hiding this comment

Uh oh!

Fungx Dec 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pandaapo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fungx commented Dec 10, 2023

Uh oh!

codecov bot commented Dec 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pandaapo left a comment

Choose a reason for hiding this comment

Uh oh!

hhuang1231 commented Dec 11, 2023

Uh oh!

pandaapo commented Dec 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pil0tXia left a comment

Choose a reason for hiding this comment

Uh oh!

Pil0tXia Dec 11, 2023

Choose a reason for hiding this comment

Uh oh!

Fungx Dec 11, 2023

Choose a reason for hiding this comment

Uh oh!

Pil0tXia left a comment

Choose a reason for hiding this comment

Uh oh!

pandaapo commented Dec 27, 2023

Uh oh!

Fungx commented Dec 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Fungx Dec 12, 2023 •

edited

Loading

pandaapo left a comment •

edited

Loading

codecov bot commented Dec 11, 2023 •

edited

Loading

pandaapo commented Dec 11, 2023 •

edited

Loading