Add controller API for reload segment task status #8828

saurabhd336 · 2022-06-03T12:00:00Z

This PR adds a controller API that can be used to get the current status of a 'reload segments' task. When a segment reload task is submitted, an identifier will be returned, and the task details will be stored into ZK. Users can then call the newly added API with the task ID, to get the status details of the reload task.
Another API has been added to get all historical reload task ids.

Design document: https://docs.google.com/document/d/1Eqn2FDDIhCr8G2JFlifs5FjT0LsVPfpPTpdJIJvorwI/edit?usp=sharing

Jackie-Jiang

The overall approach looks good!

Jackie-Jiang · 2022-06-08T21:28:22Z

...server/src/main/java/org/apache/pinot/server/starter/helix/SegmentReloadTaskStatusCache.java

We can consider adding a size limit to the cache, and once the entry count reaches the limit, we remove the earliest entries

Should I replace this map with a LRU map instead then? I think that might achieve something close to this and we can specify max size when initialising the map

pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java

Jackie-Jiang · 2022-06-08T21:30:48Z

pinot-server/src/main/java/org/apache/pinot/server/api/resources/TaskStatusResource.java

Will this handle null status properly?

Yes. It returns "null" string

...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java

Jackie-Jiang · 2022-06-08T21:32:19Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

null response need to be properly handled

Jackie-Jiang · 2022-06-08T21:33:22Z

pinot-common/src/main/java/org/apache/pinot/common/messages/SegmentReloadMessage.java

Interesting. The msgId should just be the ZNode id

Yeah the getMsgId does fetch the record's zk id but for some reason I was seeing this difference on server and controller side. I'll look into this more and update the PR accordingly if we are able to use msgId itself.

Any update on this?

So helix, when sending a PARTICIPANT message, generates one message per participant to store into ZK. When doing that, it overrides the original message's id with a new random one
https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/messaging/DefaultMessagingService.java#L199

It only retains the fields from the original message

Jackie-Jiang · 2022-06-08T21:38:09Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

We should consider storing the task id and reload info (table name / segment name) into a ZK node, and add another API to read the reload tasks in the cluster in case user loses the response of the reload call. We can also put other useful info there, e.g. number of messages sent, which can be included in the status check

So that API just returns the list of active reload task ids and description like # of messages sent, table names etc?
Agree that might be useful. But should that list be cleaned up by some sort of periodic task? Or no need?

Yes. We may put some default retention to keep the reload info (e.g. 1 day). I don't think we need a periodic task to clean up the ZK node. Instead, for each reload call, when we add new entry, we can clean up the expired ones.

Have added the API, anc the changes in realod API to store taskMetadata into ZK

Although I could not find "retention" specification when creating ZNodes. Is there any example already in place where we might be doing that?

I've handled retention via explicitly cleaning up old tasks when new ones are created.

saurabhd336 · 2022-06-14T09:43:01Z

I've updated the PR to

Add taskMeta to ZK.
Add an API to return all tasks for a table.

Yet to add

Retention on task ZNode
Track status for single segment reload

I was hoping to get the approach validated first, for all segments API. Once we are aligned on this, I'll update the PR to include changes for single segment reload.

cc: @Jackie-Jiang @npawar

codecov-commenter · 2022-06-15T11:06:06Z

Codecov Report

Merging #8828 (0b3fc36) into master (cc8362f) will decrease coverage by 43.94%.
The diff coverage is 67.85%.

❗ Current head 0b3fc36 differs from pull request most recent head 966e46f. Consider uploading reports for the commit 966e46f to get more accurate results

@@              Coverage Diff              @@
##             master    #8828       +/-   ##
=============================================
- Coverage     68.80%   24.86%   -43.95%     
+ Complexity     4747       51     -4696     
=============================================
  Files          1842     1834        -8     
  Lines         97579    97437      -142     
  Branches      14716    14698       -18     
=============================================
- Hits          67141    24228    -42913     
- Misses        25705    70827    +45122     
+ Partials       4733     2382     -2351

Flag	Coverage Δ
integration1	`?`
integration2	`24.86% <67.85%> (?)`
unittests1	`?`
unittests2	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...oller/api/resources/PinotTableRestletResource.java	`27.72% <0.00%> (-36.82%)`	⬇️
...segment/local/data/manager/SegmentDataManager.java	`0.00% <0.00%> (-83.34%)`	⬇️
...va/org/apache/pinot/spi/utils/CommonConstants.java	`0.00% <0.00%> (-28.58%)`	⬇️
...spi/utils/builder/ControllerRequestURLBuilder.java	`0.00% <0.00%> (ø)`
...ver/api/resources/ControllerJobStatusResource.java	`60.00% <60.00%> (ø)`
...ntroller/helix/core/PinotHelixResourceManager.java	`31.75% <66.66%> (-35.55%)`	⬇️
...ler/api/resources/PinotSegmentRestletResource.java	`31.19% <70.10%> (+6.29%)`	⬆️
...ache/pinot/common/metadata/ZKMetadataProvider.java	`60.11% <100.00%> (-5.43%)`	⬇️
...mmon/metadata/controllerjob/ControllerJobType.java	`100.00% <100.00%> (ø)`
...urces/ServerReloadControllerJobStatusResponse.java	`100.00% <100.00%> (ø)`
... and 1407 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

Jackie-Jiang · 2022-06-16T00:41:47Z

pinot-common/src/main/java/org/apache/pinot/common/messages/SegmentReloadMessage.java

Any update on this?

Jackie-Jiang · 2022-06-16T00:47:18Z

pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java

Suggest not keeping per-task level ZK metadata. It might create too many ZNode, and can be easily left over if not handled properly

Jackie-Jiang · 2022-06-16T00:47:42Z

pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java

Suggested change

public static String constructPropertyStorePathForTaskResource(String resourceName) {

public static String constructPropertyStorePathForTask(String resourceName) {

Jackie-Jiang · 2022-06-16T00:48:26Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

(nit) revert the empty lines

Jackie-Jiang · 2022-06-16T00:52:49Z

...server/src/main/java/org/apache/pinot/server/starter/helix/SegmentReloadTaskStatusCache.java

Will this create a LRU cache?

npawar · 2022-06-17T01:51:20Z

pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java

could we name this something other than TASKS? Task is already a concept in helix task framework and as a result in minion, so I can see this getting very confusing.

CONTROLLER_JOB?

npawar · 2022-06-17T01:53:06Z

pinot-common/src/main/java/org/apache/pinot/common/metadata/task/TaskType.java

same here, there's already a string association between TaskType and minion tasks like MergeRollup, R2O etc.

npawar · 2022-06-17T02:10:30Z

...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java

extract this block into a common util to share across this and the above method?

npawar · 2022-06-17T02:17:53Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

s/tableName/tableNameWithType ?

npawar · 2022-06-17T02:28:50Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

in case of a hybrid table, these 2 lines would run twice and always be redundant for one of the tables. Curious, why you chose to not just have tableNameWithType in the params?

The reload API accepts tableName and type is optional. Hence thought of keeping it same.

But even the reload API, returns 1 jobId per tableType for hybrid table. I think expecting tableNameWithType for this API makes sense. Have changed

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

mcvsubbu

Still reviewing.

mcvsubbu · 2022-06-17T17:00:46Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

Add segment name & tablename in the log message.

Will not repeat for other log messages

mcvsubbu · 2022-06-17T17:02:45Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

I believe most other APIs have table name and table type as independent parameters. We should keep all APIs that way, and move towards this model if that is not already the case

mcvsubbu · 2022-06-17T17:21:20Z

Please change the description. This PR does much more than just get reload status. Earlier design for reload was to "fire and forget" (send a reload helix message and let servers handle it as they please). This one (if I understand right) changes the reload command to start a helix task.

have you considered what happens when controller is upgraded to the new version but the servers are running old version, and the reload command is issued? Please evaluate and add that to the PR description, and also mark this PR for release notes

saurabhd336 · 2022-06-18T01:24:10Z

@mcvsubbu So nothing about the core functionality wrt reloading segments has been changed here. It's still largely a fire and forget operation, using helix messages that are handled by servers. This PR just associates each reload op with a trackable id, writes some additional meta about the task into ZK, and provides APIs which can be used to 1) Get the status of a given realod op task id 2) Get a historical list of reload tasks.

mcvsubbu · 2022-06-18T01:38:33Z

@mcvsubbu So nothing about the core functionality wrt reloading segments has been changed here. It's still largely a fire and forget operation, using helix messages that are handled by servers. This PR just associates each reload op with a trackable id, writes some additional meta about the task into ZK, and provides APIs which can be used to 1) Get the status of a given realod op task id 2) Get a historical list of reload tasks.

When you start using new znodes, then we need to worry about upgrades, cleaning up of znodes, etc. Please write up a design document on this before proceeding.

saurabhd336 · 2022-06-18T01:52:19Z

When you start using new znodes, then we need to worry about upgrades, cleaning up of znodes, etc. Please write up a design document on this before proceeding.

Sounds good. I'll share the design doc for this.

saurabhd336 · 2022-06-21T06:35:45Z

@mcvsubbu @npawar @Jackie-Jiang I've updated the description with the design doc link. Please do review 👍
https://docs.google.com/document/d/1Eqn2FDDIhCr8G2JFlifs5FjT0LsVPfpPTpdJIJvorwI/edit?usp=sharing

npawar

The changed approach is clever and very light weight. It's just a little odd to think of this as "status of the exact reload id" because there's no correlation anymore between the exact fired reload and the status we're returning.
Are there any other operations that would set that load time? Like a resetSegment? (I think not, but just confirming). Or any other gotchas that need to be called out?

npawar · 2022-06-23T18:45:08Z

.../java/org/apache/pinot/controller/api/resources/ServerReloadControllerJobStatusResponse.java

jobMetadata or just metadata?

npawar · 2022-06-23T18:48:02Z

pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java

rename all "Task" in this file to "Job" (and also in PinotHelixResourceManager)

saurabhd336 · 2022-06-24T04:53:18Z

The changed approach is clever and very light weight. It's just a little odd to think of this as "status of the exact reload id" because there's no correlation anymore between the exact fired reload and the status we're returning. Are there any other operations that would set that load time? Like a resetSegment? (I think not, but just confirming). Or any other gotchas that need to be called out?

@npawar Yes and that's a idea with this change. Scenarios like server restarting in the middle of a reload op, a rebalance, etc can lead to this. Earlier with the tight coupling of task id -> status map at the server, there were issues with server restarts etc leading to missing status data at the server. With this approach, success of a reload op has simply been defined as the load time being > reload job submission time.

Jackie-Jiang

In the reload message, we should also carry the reload submission time so that we don't need to reload the same segment multiple times when multiple reload is submitted

Jackie-Jiang · 2022-06-30T11:19:39Z

...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java

No need to check if the record exists. We can directly get() and check if the return is null. It can also prevent race condition.
When the record exist, we should track the record version (passing a Stat into the get()), and check the expected version when setting the record to prevent race condition

Ack. Should we be retrying in case of a version mismatch? For now I've not added retries, in line with other version based _propertyStore updates.

Jackie-Jiang · 2022-06-30T11:20:07Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

Suggested change

return new SuccessResponse("Sent " + msgInfo + " reload messages");

return new SuccessResponse("Sent " + msgInfo.getLeft() + " reload messages");

Wanted to return the job id as part of the response.

Then let's change the response message to include both job id and the messages sent, something like "Submitted reload job: %s, sent %d reload messages"

Ack. Added both the job id as well as success / failure status of ZK write

Jackie-Jiang · 2022-06-30T11:20:52Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

Suggest reflecting this back to the user, instead of just logging an error and return success. User won't be able to read the reload status

Ack. Modified the response for this API

Jackie-Jiang · 2022-06-30T11:22:31Z

...ment-local/src/main/java/org/apache/pinot/segment/local/data/manager/SegmentDataManager.java

(minor) _loadTimeMs for simplicity, same for the getter and setter

Jackie-Jiang · 2022-07-11T20:55:48Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

Then let's change the response message to include both job id and the messages sent, something like "Submitted reload job: %s, sent %d reload messages"

Jackie-Jiang · 2022-07-11T20:55:57Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

Jackie-Jiang · 2022-07-11T21:10:35Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

Let's make the response more clear to the user

Ack. Changed this to return both job id as well as zk meta write success / failure status

Jackie-Jiang · 2022-07-11T21:23:27Z

pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java

This will create one ZNode per table, which is too much IMO. We are just storing the recent controller jobs in the ZNode, and there shouldn't be a lot of them running in parallel, so we can have one single ZNode for all the tables. Keeping one single node will help track all the running jobs, and also clean up the old jobs.
We can also consider having one node per job type to isolate the metadata for different jobs. All the entries for the same job type should have the same format, which is easier to manage

Ack. I suppose since the tableName is part of the meta being stored itself and the jobId is guaranteed to be unique, we need not have to keep tableName as a key? I've flattened it out

Jackie-Jiang

The overall logic looks good.
We can consider adding a test into the OfflineClusterIntegrationTest where we have several places reloading the table

Jackie-Jiang · 2022-07-26T00:46:01Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

(nit) We usually put : for easier searching purpose, same for other places

Suggested change

LOGGER.error("Failed to add reload segment job meta into zookeeper for table {}, segment {}",

LOGGER.error("Failed to add reload segment job meta into zookeeper for table: {}, segment: {}",

Jackie-Jiang · 2022-07-26T18:06:48Z

...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java

We can save one redundant ZK access and also avoid race condition, same for other places

Suggested change

if (_propertyStore.exists(controllerJobResourcePath, AccessOption.PERSISTENT)) {

ZNRecord taskResourceZnRecord = _propertyStore.get(controllerJobResourcePath, null, -1);

ZNRecord controllerJobRecord = _propertyStore.get(controllerJobResourcePath, null, -1);

if (controllerJobRecord != null) {

Helix may throw a ZkNoNodeException exception if options = -1. So I've handled that

Jackie-Jiang · 2022-07-26T18:15:54Z

...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java

Annotate with @Nullable.
Let's add some javadoc for the public method. Same for other public methods

Added javadoc. jobId must not be be null?

Jackie-Jiang · 2022-07-26T18:22:27Z

...ntroller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java

Move this line into addNewReloadSegmentJob() and addNewReloadAllSegmentsJob() to be more readable. We want to keep the metadata creation part together

Jackie-Jiang · 2022-07-26T18:39:12Z

pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java

(nit)

Suggested change

* Task ZK props

* Controller job ZK props

Jackie-Jiang · 2022-07-26T18:40:23Z

pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java

This one applies to all jobs, instead of just the reload jobs

Jackie-Jiang · 2022-07-26T18:43:49Z

pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java

We usually use camel case as the key name in the ZNode. We may also simplify the key as it is always under the context of controller job. E.g. jobType, jobId, tableName, submissionTimeMs, messagesCount, reloadSegmentName

Jackie-Jiang · 2022-07-26T18:50:55Z

...-server/src/main/java/org/apache/pinot/server/api/resources/ControllerJobStatusResource.java

Can we handle this properly on the controller side? Should we return 0/0 instead?

With controller now filtering servers for the segment being queried using idealstate, this should be far less frequent.
And yes the controller makes the API calls using CompletionServiceHelper which handles non 200 responses correctly.

I feel returning 0/0 might be preferred so that we can differentiate server not responding vs server not having the segment

Changed this to return 0 / 0

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

Jackie-Jiang · 2022-08-02T17:19:47Z

Let's add a release-notes section in the PR description to list down the new rest APIs added

walterddr · 2023-10-14T02:17:39Z

...ler/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java

+        ServerReloadControllerJobStatusResponse response =
+            JsonUtils.stringToObject(responseString, ServerReloadControllerJobStatusResponse.class);


why are we using ServerReloadControllerJobStatusResponse.class in JsonUtil instead of SegmentReloadStatusValue.class ? does it guarantee to be always compatible? this seems like a risky assumption consider controller and server are upgraded separately during rolling upgrade

Agreed that this is risky assumption. We can change this to SegmentReloadStatusValue.class
But, I think since the agreed upon upgrade sequence is controller -> broker -> servers, if new fields are added to SegmentReloadStatusValue, in the controller we'd always need to ensure the newer fields are null checked before accessing. Since during upgrade server can be on older version.

saurabhd336 marked this pull request as draft June 3, 2022 12:00

saurabhd336 force-pushed the reloadStatus branch from ea4ff75 to 5bd1a20 Compare June 8, 2022 06:52

Jackie-Jiang reviewed Jun 8, 2022

View reviewed changes

saurabhd336 force-pushed the reloadStatus branch from 9e12366 to e2664c3 Compare June 14, 2022 09:35

saurabhd336 changed the title ~~(WIP) Add server API for task status~~ Add controller API for reload segment task status Jun 15, 2022

saurabhd336 marked this pull request as ready for review June 15, 2022 05:57

Jackie-Jiang reviewed Jun 16, 2022

View reviewed changes

saurabhd336 force-pushed the reloadStatus branch 2 times, most recently from 4823c44 to 1047e19 Compare June 16, 2022 09:03

npawar reviewed Jun 17, 2022

View reviewed changes

saurabhd336 force-pushed the reloadStatus branch from 7959a4a to 3e6f5a0 Compare June 17, 2022 10:27

mcvsubbu reviewed Jun 17, 2022

View reviewed changes

saurabhd336 force-pushed the reloadStatus branch from 1b74f18 to e65acbc Compare June 21, 2022 04:14

saurabhd336 force-pushed the reloadStatus branch from d65b2bc to cdbc371 Compare June 23, 2022 06:50

npawar reviewed Jun 23, 2022

View reviewed changes

Jackie-Jiang reviewed Jun 30, 2022

View reviewed changes

saurabhd336 force-pushed the reloadStatus branch from d0e0300 to 3bbe9bc Compare July 4, 2022 04:21

Jackie-Jiang reviewed Jul 11, 2022

View reviewed changes

saurabhd336 force-pushed the reloadStatus branch from 3bbe9bc to 214c76f Compare July 25, 2022 07:07

Jackie-Jiang reviewed Jul 26, 2022

View reviewed changes

saurabhd336 force-pushed the reloadStatus branch 2 times, most recently from e0ca1a4 to 5f0e293 Compare July 27, 2022 06:34

Saurabh Dubey added 23 commits August 2, 2022 13:18

Add derived fields to response

38a0d0d

Lint

1252a1c

Build fix

a9bb169

Review comments

9c68c6d

Review comments

a23755e

Fix npe

a992c0f

Review comments

77f1c6f

Use segment loadTime for server response

51069f0

Cleanup

9ee7b86

Enhance response

7dbfe3f

Review comments + lint

00f0170

Review comments

02d421c

Review comments

4287096

Add segmentName to server status API

ca6dcd9

Review comments

5404a06

Change response type

9483cc4

Change response type

2eff6ce

change tests

f03d7bf

Lint fix

ba386fb

Add test

9759922

Review comments

ae6ca35

Fix total segments count logic

c1a94fe

Review comments

966e46f

saurabhd336 force-pushed the reloadStatus branch from 892bdea to 966e46f Compare August 2, 2022 07:52

Jackie-Jiang added feature release-notes Referenced by PRs that need attention when compiling the next release notes labels Aug 2, 2022

Jackie-Jiang merged commit 3d2b6f3 into apache:master Aug 2, 2022

Jackie-Jiang mentioned this pull request Nov 21, 2022

Provide an indication that segments reload is complete #5391

Closed

walterddr reviewed Oct 14, 2023

View reviewed changes

	public static String constructPropertyStorePathForTaskResource(String resourceName) {
	public static String constructPropertyStorePathForTask(String resourceName) {

	return new SuccessResponse("Sent " + msgInfo + " reload messages");
	return new SuccessResponse("Sent " + msgInfo.getLeft() + " reload messages");

	LOGGER.error("Failed to add reload segment job meta into zookeeper for table {}, segment {}",
	LOGGER.error("Failed to add reload segment job meta into zookeeper for table: {}, segment: {}",

-    if (_propertyStore.exists(controllerJobResourcePath, AccessOption.PERSISTENT)) {
-      ZNRecord taskResourceZnRecord = _propertyStore.get(controllerJobResourcePath, null, -1);
+    ZNRecord controllerJobRecord = _propertyStore.get(controllerJobResourcePath, null, -1);
+    if (controllerJobRecord != null) {

		ServerReloadControllerJobStatusResponse response =
		JsonUtils.stringToObject(responseString, ServerReloadControllerJobStatusResponse.class);

Add controller API for reload segment task status #8828

Add controller API for reload segment task status #8828

Uh oh!

Conversation

saurabhd336 commented Jun 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jackie-Jiang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saurabhd336 Jun 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saurabhd336 commented Jun 14, 2022

Uh oh!

codecov-commenter commented Jun 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saurabhd336 commented Jun 3, 2022 •

edited

Loading

saurabhd336 Jun 16, 2022 •

edited

Loading

codecov-commenter commented Jun 15, 2022 •

edited

Loading