Skip to content

[release/0.9] Backport hostprocess fixes #1465

Merged
dcantah merged 8 commits intomicrosoft:release/0.9from
dcantah:hpc-storagefix
Jul 22, 2022
Merged

[release/0.9] Backport hostprocess fixes #1465
dcantah merged 8 commits intomicrosoft:release/0.9from
dcantah:hpc-storagefix

Conversation

@dcantah
Copy link
Copy Markdown
Contributor

@dcantah dcantah commented Jul 21, 2022

This change brings a fix to make storage stats work for hostprocess containers, as well as some additional fixes to jobobjects/hpc along the way in the commit history.

Commits cped:

  1. 1154e47
  2. 52a4ea9
  3. 7fbb79f
  4. 566e3b0
  5. f29964f
  6. 014849a
  7. 9d05b5b

And one additional commit to vendor everything in test as release/0.9 still has the test vendor directory.

dcantah and others added 6 commits July 21, 2022 13:19
* Rework /internal/queue package

Given our use cases for this package, we don't need methods that don't block
on reads if there's no value to be read. Due to this, I've removed the
ReadOrWait function and did a small redesign of the methods to be more
in line with standard queue method naming.

* Change Read/Write/IsEmpty to Dequeue/Enqueue/Size and remove ReadOrWait.
Now there is no version of Read/Dequeue that doesn't block if the queue
is empty.
* Fix up tests to be in line with this removal of the non-blocking read
and simplified most of the tests.

Signed-off-by: Daniel Canter <[email protected]>
(cherry picked from commit 12d4cd8)
Signed-off-by: Daniel Canter <[email protected]>
* Add IO tracking option for job objects

HCS enables this to get more in depth IO stats for the silo of the
container. I'd swapped hostprocess containers to querying for
these stats but without the prerequisite of actually enabling them :)

This change adds a new option on jobobject.Options{} to enable this
functionality and adds a new test to ensure we can actually call
StorageStats now.

Signed-off-by: Daniel Canter <[email protected]>
(cherry picked from commit 598ea47)
Signed-off-by: Daniel Canter <[email protected]>
Per rule 4 of unsafe.Pointer usage, conversion of a unsafe.Pointer
to a uinptr to pass to the syscall.Syscall family should only be done in
the argument list. We had a couple spots where we were passing it in as
an argument to a small wrapper function until it reached the underlying
syscall.Syscall*.

https://pkg.go.dev/unsafe#Pointer

Signed-off-by: Daniel Canter <[email protected]>
(cherry picked from commit 1ec8cad)
Signed-off-by: Daniel Canter <[email protected]>
Two tests added recently by yours truly don't need to be running as
silos. Remove the silo field set to true. Was running into this when
trying to backport a fix to a branch that doesn't have the silo work.

Signed-off-by: Daniel Canter <[email protected]>
(cherry picked from commit 41d8f5a)
Signed-off-by: Daniel Canter <[email protected]>
Additionally fix the queue tests logic.

Signed-off-by: Maksim An <[email protected]>
(cherry picked from commit 873b8e1)
Signed-off-by: Daniel Canter <[email protected]>
Signed-off-by: Daniel Canter <[email protected]>
Signed-off-by: James Sturtevant <[email protected]>
(cherry picked from commit 9d05b5b)
Signed-off-by: Daniel Canter <[email protected]>
When installing tools into container it is sometimes desirable to have
those tools on the path for the current process.  Since the Windows
doesn't support variable expansion on the PATH variable and the
job mount path as something that isn't know at runtime we need to expand
it for the passer.  This could go away if the mount path is no longer
used in the future.

Signed-off-by: James Sturtevant <[email protected]>
(cherry picked from commit 014849a)
Signed-off-by: Daniel Canter <[email protected]>
@dcantah dcantah changed the title [release/0.9] Backport hostprocess fixes for storage stats [release/0.9] Backport hostprocess fixes Jul 22, 2022
@dcantah
Copy link
Copy Markdown
Contributor Author

dcantah commented Jul 22, 2022

fyi @jsturtevant

@dcantah dcantah merged commit e6107b7 into microsoft:release/0.9 Jul 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants