Skip to content

feat(xds): using data written by xds to control dp behavior #6759

Merged
spacewander merged 16 commits into
apache:masterfrom
tzssangglass:xds2
Apr 7, 2022
Merged

feat(xds): using data written by xds to control dp behavior #6759
spacewander merged 16 commits into
apache:masterfrom
tzssangglass:xds2

Conversation

@tzssangglass

@tzssangglass tzssangglass commented Mar 31, 2022

Copy link
Copy Markdown
Member

Description

This PR is designed to mock the behavior of xds writing data and reading it from ngx.shared.DICT to complete the request proxy.

It needs to accomplish the following functions:

  1. fetch the latest data version number every 1s
  2. update the data in full from ngx.shared.DICT if a new version is available
  3. handle route and upstream data correctly, as if it were subscribed to them from etcd

Fixes # (issue)

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

@tzssangglass
tzssangglass marked this pull request as draft March 31, 2022 02:51
@tzssangglass

Copy link
Copy Markdown
Member Author

I want to write some test cases, such as without any keys, json with wrong format, json that does not conform to the route's json scheme, etc., but I don't know how to control the data passed with libxds.so in the test cases.

But I can write data to shdict directly in the test case, which seems more convenient than writing data with libxds.so, is this acceptable way of testing.

cc @spacewander @membphis

@tzssangglass
tzssangglass marked this pull request as ready for review April 1, 2022 10:29
Comment thread t/xds-library/main.go Outdated
Comment thread t/xds-library/config_xds_2.t Outdated
}
--- error_code: 404
--- no_error_log
[alert]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why should we check the alert log? Is there any alert log in the path?

@tzssangglass tzssangglass Apr 2, 2022

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because this test case throws an error level log, but the above defines

    if (!$block->no_error_log) {
        $block->set_value("no_error_log", "[error]\n[alert]");
    }

so I need to override the above definition with the --- no_error_log statement, and I need to verify that the error level log.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tzssangglass
We can check like this?

if ((!defined $block->error_log) && (!defined $block->no_error_log)) {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can safely remove it, right?

Comment thread apisix/cli/ngx_tpl.lua Outdated
Comment thread t/APISIX.pm Outdated
Comment thread t/xds-library/config_xds_2.t
Comment thread t/xds-library/config_xds_2.t
membphis
membphis previously approved these changes Apr 3, 2022

@membphis membphis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread apisix/init.lua Outdated

require("apisix.debug").init_worker()

if core.config == require("apisix.core.config_xds") then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just put all the init_worker together?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For config_xds, I need it to run before route, service is initialised. So do I need to advance config_yaml to the same level as config_xds?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix

Comment thread t/xds-library/config_xds_2.t Outdated
}
--- error_code: 404
--- no_error_log
[alert]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can safely remove it, right?

Comment thread apisix/core/config_xds.lua Outdated

if not keys or #keys <= 0 then
-- xds did not write any data to shdict
return true, "no keys"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return true, "no keys"
return false, "no keys"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix

Comment thread apisix/core/config_xds.lua Outdated
end

-- v1 version we only support route/upstream
local capacity = math_ceil(#keys / 2)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why divide 2?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ngx.shared["xds-config"] stores all routes, upstreams, here I estimate the capacity of the route by using the total capacity / 2.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Route and upstream are not 1:1.
Maybe we can use #keys directly

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix


-- blocking until xds completes initial configuration
while true do
fetch_version()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need a sleep here?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ngx.sleep can not used in init_worker_by_lua, is there another way?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use another sleep?

int usleep(useconds_t usec);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix

Comment thread apisix/core/config_xds.lua Outdated


local function sync_data(self)
if not latest_version then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we already block for the version, we can skip this check?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix

Comment thread apisix/core/config_xds.lua Outdated

local sync_data
local latest_version
sync_data = function(self)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use local function xxx?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix


-- blocking until xds completes initial configuration
while true do
fetch_version()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use another sleep?

int usleep(useconds_t usec);

@tzssangglass
tzssangglass requested a review from membphis April 7, 2022 02:32
@spacewander
spacewander merged commit fc3a1c9 into apache:master Apr 7, 2022
Liu-Junlin pushed a commit to Liu-Junlin/apisix that referenced this pull request May 20, 2022
@tzssangglass
tzssangglass deleted the xds2 branch May 24, 2022 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants