Skip to content

fix: etcd sync data checker should work#11430

Closed
ronething wants to merge 4 commits into
apache:masterfrom
ronething:fix/etcd_sync_data_checker
Closed

fix: etcd sync data checker should work#11430
ronething wants to merge 4 commits into
apache:masterfrom
ronething:fix/etcd_sync_data_checker

Conversation

@ronething

@ronething ronething commented Jul 24, 2024

Copy link
Copy Markdown
Member

Description

Fixes # (issue)

Background

the load_full_data function in the file config_etcd.lua(load all exist data from etcd when apisix start), checker does not depend on item_schema to have a value

if data_valid and self.checker then
data_valid, err = self.checker(item.value)
if not data_valid then
log.error("failed to check item data of [", self.key,
"] err:", err, " ,val: ", json.delay_encode(item.value))
end
end

But in the sync_data function (watch new events from etcd when apisix running), checker relies on item_schema should have value.

if data_valid and res.value and self.item_schema then

As a result, the plugin metadata resource cannot reach the checker logic when etcd watch.

apisix/apisix/plugin.lua

Lines 816 to 821 in 507df12

local plugin_metadatas, err = core.config.new("/plugin_metadata",
{
automatic = true,
checker = check_plugin_metadata
}
)

So we need to fix it.

about the test case:

The admin api is not used for the operation, because the admin api will directly fail to verify the schema and cannot go to the etcd checker logic.

error-log-logger-clickhouse.t: The request header sent to clickhouse should be decrypted to be correct.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

moonming
moonming previously approved these changes Jul 24, 2024
@ronething
ronething requested a review from moonming July 31, 2024 02:57
this is a warning message for test2
clickhouse body: INSERT INTO t FORMAT JSONEachRow
clickhouse headers: x-clickhouse-key:dpwomMlEsHH2L7wSUi6YiQ==
clickhouse headers: x-clickhouse-key:a

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why change this?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#11076
The modification of error-log-logger-clickhouse.t in this PR is wrong. The header sent to clickhouse should be decrypted.

Comment thread t/admin/plugin-metadata.t
}
--- response_body
{"key":"/apisix/plugin_metadata/example-plugin","value":{"ikey":1,"skey":"val"}}
{"key":"/apisix/plugin_metadata/example-plugin","value":{"id":"example-plugin","ikey":1,"skey":"val"}}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why change this?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the admin api, we have added an id field to the plugin metadata, so the response here needs to be added.

Comment on lines +66 to +70
properties = {
log_format = {
type = "object"
}
},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this change have to do with etcd?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Originally sls-logger didn't add the metadata_schema, and then because the etcd checker didn't take effect before, it couldn't detect the problem.
As for the metadata schema, I refer to file-logger.

local metadata_schema = {
type = "object",
properties = {
log_format = {
type = "object"
}
}
}

@moonming

Copy link
Copy Markdown
Member

One PR is only for one task.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants