Skip to content

Commit b23022d

Browse files
matt-borismatt-boris
committed
feat(db): provide worker counts and capacities by state for worker pools
1 parent 92f0d55 commit b23022d

File tree

16 files changed

+601
-42
lines changed

16 files changed

+601
-42
lines changed

‎changelog/issue-4942.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
audience: developers
2+
level: minor
3+
reference: issue 4942
4+
---
5+
Addresses #4942. Add `get_worker_pool_with_capacity_and_counts_by_state`, `get_worker_pools_with_capacity_and_counts_by_state`, and `update_worker_pool_with_capacity_and_counts_by_state` functions to get worker counts and capacity by state for worker pools.
6+
7+
Deprecates `get_worker_pool_with_capacity`, `get_worker_pools_with_capacity`, and `update_worker_pool_with_capacity`.

‎clients/client-go/tcworkermanager/types.go‎

Lines changed: 40 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎db/fns.md‎

Lines changed: 42 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -150,13 +150,13 @@
150150
* [`get_worker_2`](#get_worker_2)
151151
* [`get_worker_pool_error`](#get_worker_pool_error)
152152
* [`get_worker_pool_errors_for_worker_pool`](#get_worker_pool_errors_for_worker_pool)
153-
* [`get_worker_pool_with_capacity`](#get_worker_pool_with_capacity)
154-
* [`get_worker_pools_with_capacity`](#get_worker_pools_with_capacity)
153+
* [`get_worker_pool_with_capacity_and_counts_by_state`](#get_worker_pool_with_capacity_and_counts_by_state)
154+
* [`get_worker_pools_with_capacity_and_counts_by_state`](#get_worker_pools_with_capacity_and_counts_by_state)
155155
* [`get_workers_without_provider_data`](#get_workers_without_provider_data)
156156
* [`remove_worker_pool_previous_provider_id`](#remove_worker_pool_previous_provider_id)
157157
* [`update_worker_2`](#update_worker_2)
158158
* [`update_worker_pool_provider_data`](#update_worker_pool_provider_data)
159-
* [`update_worker_pool_with_capacity`](#update_worker_pool_with_capacity)
159+
* [`update_worker_pool_with_capacity_and_counts_by_state`](#update_worker_pool_with_capacity_and_counts_by_state)
160160

161161
## auth
162162

@@ -2363,13 +2363,13 @@ If the hashed session id does not exist, then an error code `P0002` will be thro
23632363
* [`get_worker_2`](#get_worker_2)
23642364
* [`get_worker_pool_error`](#get_worker_pool_error)
23652365
* [`get_worker_pool_errors_for_worker_pool`](#get_worker_pool_errors_for_worker_pool)
2366-
* [`get_worker_pool_with_capacity`](#get_worker_pool_with_capacity)
2367-
* [`get_worker_pools_with_capacity`](#get_worker_pools_with_capacity)
2366+
* [`get_worker_pool_with_capacity_and_counts_by_state`](#get_worker_pool_with_capacity_and_counts_by_state)
2367+
* [`get_worker_pools_with_capacity_and_counts_by_state`](#get_worker_pools_with_capacity_and_counts_by_state)
23682368
* [`get_workers_without_provider_data`](#get_workers_without_provider_data)
23692369
* [`remove_worker_pool_previous_provider_id`](#remove_worker_pool_previous_provider_id)
23702370
* [`update_worker_2`](#update_worker_2)
23712371
* [`update_worker_pool_provider_data`](#update_worker_pool_provider_data)
2372-
* [`update_worker_pool_with_capacity`](#update_worker_pool_with_capacity)
2372+
* [`update_worker_pool_with_capacity_and_counts_by_state`](#update_worker_pool_with_capacity_and_counts_by_state)
23732373

23742374
### create_worker
23752375

@@ -2593,7 +2593,7 @@ ordered by `reported`.
25932593
If the pagination arguments are both NULL, all rows are returned.
25942594
Otherwise, page_size rows are returned at offset page_offset.
25952595

2596-
### get_worker_pool_with_capacity
2596+
### get_worker_pool_with_capacity_and_counts_by_state
25972597

25982598
* *Mode*: read
25992599
* *Arguments*:
@@ -2610,11 +2610,19 @@ Otherwise, page_size rows are returned at offset page_offset.
26102610
* `email_on_error boolean`
26112611
* `provider_data jsonb`
26122612
* `current_capacity integer`
2613-
* *Last defined on version*: 13
2613+
* `requested_count integer`
2614+
* `running_count integer`
2615+
* `stopping_count integer`
2616+
* `stopped_count integer`
2617+
* `requested_capacity integer`
2618+
* `running_capacity integer`
2619+
* `stopping_capacity integer`
2620+
* `stopped_capacity integer`
2621+
* *Last defined on version*: 70
26142622

26152623
Get an existing worker pool. The returned table will have one or (if no such worker pool is defined) zero rows.
26162624

2617-
### get_worker_pools_with_capacity
2625+
### get_worker_pools_with_capacity_and_counts_by_state
26182626

26192627
* *Mode*: read
26202628
* *Arguments*:
@@ -2632,7 +2640,15 @@ Get an existing worker pool. The returned table will have one or (if no such wo
26322640
* `email_on_error boolean`
26332641
* `provider_data jsonb`
26342642
* `current_capacity integer`
2635-
* *Last defined on version*: 13
2643+
* `requested_count integer`
2644+
* `running_count integer`
2645+
* `stopping_count integer`
2646+
* `stopped_count integer`
2647+
* `requested_capacity integer`
2648+
* `running_capacity integer`
2649+
* `stopping_capacity integer`
2650+
* `stopped_capacity integer`
2651+
* *Last defined on version*: 70
26362652

26372653
Get existing worker pools, ordered by `worker_pool_id`. If the pagination arguments are both NULL, all rows are returned.
26382654
Otherwise, page_size rows are returned at offset page_offset.
@@ -2731,7 +2747,7 @@ this sets the provider_data property unconditionally, and it is up to the servic
27312747
to ensure that concurrent modifications do not occur. It is not an error if the
27322748
worker pool does not exist.
27332749

2734-
### update_worker_pool_with_capacity
2750+
### update_worker_pool_with_capacity_and_counts_by_state
27352751

27362752
* *Mode*: write
27372753
* *Arguments*:
@@ -2753,11 +2769,25 @@ worker pool does not exist.
27532769
* `email_on_error boolean`
27542770
* `previous_provider_id text`
27552771
* `current_capacity integer`
2756-
* *Last defined on version*: 13
2772+
* `requested_count integer`
2773+
* `running_count integer`
2774+
* `stopping_count integer`
2775+
* `stopped_count integer`
2776+
* `requested_capacity integer`
2777+
* `running_capacity integer`
2778+
* `stopping_capacity integer`
2779+
* `stopped_capacity integer`
2780+
* *Last defined on version*: 70
27572781

27582782
Update API-accessible columns on an existig worker pool. All fields are
27592783
overridden, but if the provider_id changes, then the existing provider_id
27602784
is added to previous_provider_ids. The return value contains values
27612785
required for an API response and previous_provider_id (singular) containing
27622786
the provider_id found before the update. If no such worker pool exists,
27632787
the return value is an empty set.
2788+
2789+
### deprecated methods
2790+
2791+
* `get_worker_pool_with_capacity(worker_pool_id_in text)` (compatibility guaranteed until v46.0.0)
2792+
* `get_worker_pools_with_capacity(page_size_in integer, page_offset_in integer)` (compatibility guaranteed until v46.0.0)
2793+
* `update_worker_pool_with_capacity(worker_pool_id_in text, provider_id_in text, description_in text, config_in jsonb, last_modified_in timestamptz, owner_in text, email_on_error_in boolean)` (compatibility guaranteed until v46.0.0)

‎db/test/fns/worker_manager_test.js‎

Lines changed: 117 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -43,12 +43,26 @@ suite(testing.suiteName(), function() {
4343
wp.owner || '[email protected]',
4444
wp.email_on_error || false,
4545
];
46-
const with_cap = await db.fns.update_worker_pool_with_capacity(...input);
46+
const with_cap = await db.fns.update_worker_pool_with_capacity_and_counts_by_state(...input);
4747
for (const wp of with_cap) {
48-
assert(wp.current_capacity !== undefined);
49-
delete wp.current_capacity;
48+
assert(wp.requested_count !== undefined);
49+
assert(wp.running_count !== undefined);
50+
assert(wp.stopping_count !== undefined);
51+
assert(wp.stopped_count !== undefined);
52+
assert(wp.requested_capacity !== undefined);
53+
assert(wp.running_capacity !== undefined);
54+
assert(wp.stopping_capacity !== undefined);
55+
assert(wp.stopped_capacity !== undefined);
56+
delete wp.requested_count;
57+
delete wp.running_count;
58+
delete wp.stopping_count;
59+
delete wp.stopped_count;
60+
delete wp.requested_capacity;
61+
delete wp.running_capacity;
62+
delete wp.stopping_capacity;
63+
delete wp.stopped_capacity;
5064
}
51-
const old = await db.deprecatedFns.update_worker_pool(...input);
65+
const old = await db.deprecatedFns.update_worker_pool_with_capacity(...input);
5266
// We override previous_provider_id in this comparison because a side-effect
5367
// of calling this function is updating that value so we can't compare here
5468
assert.deepEqual({ ...with_cap[0], previous_provider_id: '' }, { ...old[0], previous_provider_id: '' });
@@ -115,23 +129,51 @@ suite(testing.suiteName(), function() {
115129
};
116130

117131
const get_worker_pool = async (db, worker_pool_id) => {
118-
const with_cap = await db.fns.get_worker_pool_with_capacity(worker_pool_id);
132+
const with_cap = await db.fns.get_worker_pool_with_capacity_and_counts_by_state(worker_pool_id);
119133
for (const wp of with_cap) {
120-
assert(wp.current_capacity !== undefined);
121-
delete wp.current_capacity;
134+
assert(wp.requested_count !== undefined);
135+
assert(wp.running_count !== undefined);
136+
assert(wp.stopping_count !== undefined);
137+
assert(wp.stopped_count !== undefined);
138+
assert(wp.requested_capacity !== undefined);
139+
assert(wp.running_capacity !== undefined);
140+
assert(wp.stopping_capacity !== undefined);
141+
assert(wp.stopped_capacity !== undefined);
142+
delete wp.requested_count;
143+
delete wp.running_count;
144+
delete wp.stopping_count;
145+
delete wp.stopped_count;
146+
delete wp.requested_capacity;
147+
delete wp.running_capacity;
148+
delete wp.stopping_capacity;
149+
delete wp.stopped_capacity;
122150
}
123-
const old = await db.deprecatedFns.get_worker_pool(worker_pool_id);
151+
const old = await db.deprecatedFns.get_worker_pool_with_capacity(worker_pool_id);
124152
assert.deepEqual(with_cap, old);
125153
return with_cap;
126154
};
127155

128156
const get_worker_pools = async (db, page_size, page_offset) => {
129-
const with_cap = await db.fns.get_worker_pools_with_capacity(page_size, page_offset);
157+
const with_cap = await db.fns.get_worker_pools_with_capacity_and_counts_by_state(page_size, page_offset);
130158
for (const wp of with_cap) {
131-
assert(wp.current_capacity !== undefined);
132-
delete wp.current_capacity;
159+
assert(wp.requested_count !== undefined);
160+
assert(wp.running_count !== undefined);
161+
assert(wp.stopping_count !== undefined);
162+
assert(wp.stopped_count !== undefined);
163+
assert(wp.requested_capacity !== undefined);
164+
assert(wp.running_capacity !== undefined);
165+
assert(wp.stopping_capacity !== undefined);
166+
assert(wp.stopped_capacity !== undefined);
167+
delete wp.requested_count;
168+
delete wp.running_count;
169+
delete wp.stopping_count;
170+
delete wp.stopped_count;
171+
delete wp.requested_capacity;
172+
delete wp.running_capacity;
173+
delete wp.stopping_capacity;
174+
delete wp.stopped_capacity;
133175
}
134-
const old = await db.deprecatedFns.get_worker_pools(page_size, page_offset);
176+
const old = await db.deprecatedFns.get_worker_pools_with_capacity(page_size, page_offset);
135177
assert.deepEqual(with_cap, old);
136178
return with_cap;
137179
};
@@ -901,54 +943,108 @@ suite(testing.suiteName(), function() {
901943
suite('existing capacity', function() {
902944
helper.dbTest('no workers', async function(db) {
903945
await create_worker_pool(db);
904-
assert.equal((await db.fns.get_worker_pool_with_capacity('wp/id'))[0].current_capacity, 0);
946+
const row = (await db.fns.get_worker_pool_with_capacity_and_counts_by_state('wp/id'))[0];
947+
assert.equal(row.current_capacity, 0);
948+
assert.equal(row.requested_count, 0);
949+
assert.equal(row.running_count, 0);
950+
assert.equal(row.stopping_count, 0);
951+
assert.equal(row.stopped_count, 0);
952+
assert.equal(row.requested_capacity, 0);
953+
assert.equal(row.running_capacity, 0);
954+
assert.equal(row.stopping_capacity, 0);
955+
assert.equal(row.stopped_capacity, 0);
905956
});
906957
helper.dbTest('single worker, capacity 1', async function(db) {
907958
await create_worker_pool(db);
908959
await create_worker(db, { capacity: 1, state: 'running' });
909-
assert.equal((await db.fns.get_worker_pool_with_capacity('wp/id'))[0].current_capacity, 1);
960+
const row = (await db.fns.get_worker_pool_with_capacity_and_counts_by_state('wp/id'))[0];
961+
assert.equal(row.current_capacity, 1);
962+
assert.equal(row.requested_count, 0);
963+
assert.equal(row.running_count, 1);
964+
assert.equal(row.stopping_count, 0);
965+
assert.equal(row.stopped_count, 0);
966+
assert.equal(row.requested_capacity, 0);
967+
assert.equal(row.running_capacity, 1);
968+
assert.equal(row.stopping_capacity, 0);
969+
assert.equal(row.stopped_capacity, 0);
910970
});
911971
helper.dbTest('single worker, capacity > 1', async function(db) {
912972
await create_worker_pool(db);
913973
await create_worker(db, { capacity: 64, state: 'running' });
914-
assert.equal((await db.fns.get_worker_pool_with_capacity('wp/id'))[0].current_capacity, 64);
974+
const row = (await db.fns.get_worker_pool_with_capacity_and_counts_by_state('wp/id'))[0];
975+
assert.equal(row.current_capacity, 64);
976+
assert.equal(row.requested_count, 0);
977+
assert.equal(row.running_count, 1);
978+
assert.equal(row.stopping_count, 0);
979+
assert.equal(row.stopped_count, 0);
980+
assert.equal(row.requested_capacity, 0);
981+
assert.equal(row.running_capacity, 64);
982+
assert.equal(row.stopping_capacity, 0);
983+
assert.equal(row.stopped_capacity, 0);
915984
});
916985
helper.dbTest('multiple workers, capacity 1', async function(db) {
917986
await create_worker_pool(db);
918987
await create_worker(db, { worker_id: 'foo1', capacity: 1, state: 'running' });
919988
await create_worker(db, { worker_id: 'foo2', capacity: 1, state: 'running' });
920989
await create_worker(db, { worker_id: 'foo3', capacity: 1, state: 'running' });
921990
await create_worker(db, { worker_id: 'foo4', capacity: 1, state: 'running' });
922-
assert.equal((await db.fns.get_worker_pool_with_capacity('wp/id'))[0].current_capacity, 4);
991+
const row = (await db.fns.get_worker_pool_with_capacity_and_counts_by_state('wp/id'))[0];
992+
assert.equal(row.current_capacity, 4);
993+
assert.equal(row.requested_count, 0);
994+
assert.equal(row.running_count, 4);
995+
assert.equal(row.stopping_count, 0);
996+
assert.equal(row.stopped_count, 0);
997+
assert.equal(row.requested_capacity, 0);
998+
assert.equal(row.running_capacity, 4);
999+
assert.equal(row.stopping_capacity, 0);
1000+
assert.equal(row.stopped_capacity, 0);
9231001
});
9241002
helper.dbTest('multiple workers, capacity > 1', async function(db) {
9251003
await create_worker_pool(db);
9261004
await create_worker(db, { worker_id: 'foo1', capacity: 32, state: 'running' });
9271005
await create_worker(db, { worker_id: 'foo2', capacity: 64, state: 'running' });
9281006
await create_worker(db, { worker_id: 'foo3', capacity: 64, state: 'running' });
9291007
await create_worker(db, { worker_id: 'foo4', capacity: 1, state: 'running' });
930-
assert.equal((await db.fns.get_worker_pool_with_capacity('wp/id'))[0].current_capacity, 161);
1008+
const row = (await db.fns.get_worker_pool_with_capacity_and_counts_by_state('wp/id'))[0];
1009+
assert.equal(row.current_capacity, 161);
1010+
assert.equal(row.requested_count, 0);
1011+
assert.equal(row.running_count, 4);
1012+
assert.equal(row.stopping_count, 0);
1013+
assert.equal(row.stopped_count, 0);
1014+
assert.equal(row.requested_capacity, 0);
1015+
assert.equal(row.running_capacity, 161);
1016+
assert.equal(row.stopping_capacity, 0);
1017+
assert.equal(row.stopped_capacity, 0);
9311018
});
9321019
helper.dbTest('multiple workers, multiple states', async function(db) {
9331020
await create_worker_pool(db);
9341021
await create_worker(db, { worker_id: 'foo1', capacity: 32, state: 'running' });
9351022
await create_worker(db, { worker_id: 'foo2', capacity: 64, state: 'stopped' });
9361023
await create_worker(db, { worker_id: 'foo3', capacity: 64, state: 'running' });
9371024
await create_worker(db, { worker_id: 'foo4', capacity: 1, state: 'requested' });
938-
assert.equal((await db.fns.get_worker_pool_with_capacity('wp/id'))[0].current_capacity, 97);
1025+
const row = (await db.fns.get_worker_pool_with_capacity_and_counts_by_state('wp/id'))[0];
1026+
assert.equal(row.current_capacity, 97);
1027+
assert.equal(row.requested_count, 1);
1028+
assert.equal(row.running_count, 2);
1029+
assert.equal(row.stopping_count, 0);
1030+
assert.equal(row.stopped_count, 1);
1031+
assert.equal(row.requested_capacity, 1);
1032+
assert.equal(row.running_capacity, 96);
1033+
assert.equal(row.stopping_capacity, 0);
1034+
assert.equal(row.stopped_capacity, 64);
9391035
});
9401036
helper.dbTest('no workers (multiple pools)', async function(db) {
9411037
await create_worker_pool(db);
9421038
await create_worker_pool(db, { worker_pool_id: 'ff/tt' });
943-
const pools = (await db.fns.get_worker_pools_with_capacity(null, null)).sort();
1039+
const pools = (await db.fns.get_worker_pools_with_capacity_and_counts_by_state(null, null)).sort();
9441040
assert.equal(pools[0].current_capacity, 0);
9451041
assert.equal(pools[1].current_capacity, 0);
9461042
});
9471043
helper.dbTest('single worker (multiple pools)', async function(db) {
9481044
await create_worker_pool(db);
9491045
await create_worker_pool(db, { worker_pool_id: 'ff/tt' });
9501046
await create_worker(db, { capacity: 4, state: 'running' });
951-
const pools = (await db.fns.get_worker_pools_with_capacity(null, null)).sort();
1047+
const pools = (await db.fns.get_worker_pools_with_capacity_and_counts_by_state(null, null)).sort();
9521048
assert.equal(pools[0].worker_pool_id, 'ff/tt');
9531049
assert.equal(pools[1].worker_pool_id, 'wp/id');
9541050
assert.equal(pools[0].current_capacity, 0);
@@ -962,7 +1058,7 @@ suite(testing.suiteName(), function() {
9621058
await create_worker(db, { worker_id: 'foo3', capacity: 10, state: 'running', worker_pool_id: 'ff/tt' });
9631059
await create_worker(db, { worker_id: 'foo4', capacity: 3, state: 'stopped' });
9641060
await create_worker(db, { worker_id: 'foo5', capacity: 7, state: 'stopped', worker_pool_id: 'ff/tt' });
965-
const pools = (await db.fns.get_worker_pools_with_capacity(null, null)).sort();
1061+
const pools = (await db.fns.get_worker_pools_with_capacity_and_counts_by_state(null, null)).sort();
9661062
assert.equal(pools[0].worker_pool_id, 'ff/tt');
9671063
assert.equal(pools[1].worker_pool_id, 'wp/id');
9681064
assert.equal(pools[0].current_capacity, 10);

‎db/test/versions/0070_test.js‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
const testing = require('taskcluster-lib-testing');
2+
3+
suite(testing.suiteName(), function() {
4+
// this version only updates method,
5+
// to trigger index usage, so no tests are needed
6+
});

0 commit comments

Comments
 (0)