Validate container improvements with .NET 6
We've made various changes in .NET 6 to improve aspects of container resource limits governance, particularly for Windows containers. This issue demonstrates the new capabilities and their current behavior using the dotnetapp sample app.
Issues:
Changes:
The last change is to a sample, which will be used to demonstrate the others changes. It is a .NET 5 app. As a result, various roll-forward settings are used to coerce the app to run on .NET 6.
At the time of writing, the relevant changes have not been released in a public .NET 6 preview. As a result, I updated and locally rebuilt various Dockerfiles with daily builds from dotnet/installer.
Relevant docs:
Linux -- CPU and memory limits
The following test demonstrates the .NET runtime honoring CPU and resource limits.
C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 mcr.microsoft.com/dotnet/runtime:6.0 dotnet dotnetapp.dll
42
42 ,d ,d
42 42 42
,adPPYb,42 ,adPPYba, MM42MMM 8b,dPPYba, ,adPPYba, MM42MMM
a8" `Y42 a8" "8a 42 42P' `"8a a8P_____42 42
8b 42 8b d8 42 42 42 8PP""""""" 42
"8a, ,d42 "8a, ,a8" 42, 42 42 "8b, ,aa 42,
`"8bbdP"Y8 `"YbbdP"' "Y428 42 42 `"Ybbd8"' "Y428
.NET 6.0.0-preview.2.21154.6
Debian GNU/Linux bullseye/sid
OSArchitecture: X64
ProcessorCount: 3
TotalAvailableMemoryBytes: 75.00 MiB
cfs_quota_us: 300000
usage_in_bytes: 25821184 24.63 MiB
limit_in_bytes: 104857600 100.00 MiBSimilar, but with CPU sets (AKA CPU affinity) is used instead.
C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpuset-cpus "1-3,5" -m 100mb -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 mcr.microsoft.com/dotnet/runtime:6.0 dotnet dotnetapp.dll
42
42 ,d ,d
42 42 42
,adPPYb,42 ,adPPYba, MM42MMM 8b,dPPYba, ,adPPYba, MM42MMM
a8" `Y42 a8" "8a 42 42P' `"8a a8P_____42 42
8b 42 8b d8 42 42 42 8PP""""""" 42
"8a, ,d42 "8a, ,a8" 42, 42 42 "8b, ,aa 42,
`"8bbdP"Y8 `"YbbdP"' "Y428 42 42 `"Ybbd8"' "Y428
.NET 6.0.0-preview.2.21154.6
Debian GNU/Linux bullseye/sid
OSArchitecture: X64
ProcessorCount: 4
TotalAvailableMemoryBytes: 75.00 MiB
usage_in_bytes: 6459392 6.16 MiB
limit_in_bytes: 104857600 100.00 MiBThe latest .NET 6 Preview is used for these examples, since these capabilities are all pre .NET 6.
Linux -- custom processor count
The following test demonstrates the .NET runtime honoring the new DOTNET_PROCESSOR_COUNT ENV, which provides a different value to Environment.ProcessorCount and the equivalent setting in the native runtime. It doesn't directly related to and does not affect the docker run --cpus mechanism, but is intended as a higher-level signal for scaling algorithms.
C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb -e DOTNET_PROCESSOR_COUNT=10 -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 dotnet6runtime dotnet dotnetapp.dll
42
42 ,d ,d
42 42 42
,adPPYb,42 ,adPPYba, MM42MMM 8b,dPPYba, ,adPPYba, MM42MMM
a8" `Y42 a8" "8a 42 42P' `"8a a8P_____42 42
8b 42 8b d8 42 42 42 8PP""""""" 42
"8a, ,d42 "8a, ,a8" 42, 42 42 "8b, ,aa 42,
`"8bbdP"Y8 `"YbbdP"' "Y428 42 42 `"Ybbd8"' "Y428
.NET 6.0.0-preview.6.21276.13
Debian GNU/Linux bullseye/sid
OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiB
cfs_quota_us: 300000
usage_in_bytes: 6885376 6.57 MiB
limit_in_bytes: 104857600 100.00 MiBI used a .NET 6 Preview 6 build to demonstrate this change. It works as expected.
Let's double check that we get the right behavior if no CPU limits are set.
C:\git\dotnet-docker\samples\dotnetapp>docker run --rm -m 100mb -e DOTNET_PROCESSOR_COUNT=10 -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 dotnet6runtime dotnet dotnetapp.dll
42
42 ,d ,d
42 42 42
,adPPYb,42 ,adPPYba, MM42MMM 8b,dPPYba, ,adPPYba, MM42MMM
a8" `Y42 a8" "8a 42 42P' `"8a a8P_____42 42
8b 42 8b d8 42 42 42 8PP""""""" 42
"8a, ,d42 "8a, ,a8" 42, 42 42 "8b, ,aa 42,
`"8bbdP"Y8 `"YbbdP"' "Y428 42 42 `"Ybbd8"' "Y428
.NET 6.0.0-preview.6.21276.13
Debian GNU/Linux bullseye/sid
OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiB
usage_in_bytes: 6713344 6.40 MiB
limit_in_bytes: 104857600 100.00 MiBWe do.
Let's check if we get the correct value with CPU affinity.
C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpuset-cpus "1-3,5" -m 100mb -e DOTNET_PROCESSOR_COUNT=10 -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 dotnet6runtime dotnet dotnetapp.dll
42
42 ,d ,d
42 42 42
,adPPYb,42 ,adPPYba, MM42MMM 8b,dPPYba, ,adPPYba, MM42MMM
a8" `Y42 a8" "8a 42 42P' `"8a a8P_____42 42
8b 42 8b d8 42 42 42 8PP""""""" 42
"8a, ,d42 "8a, ,a8" 42, 42 42 "8b, ,aa 42,
`"8bbdP"Y8 `"YbbdP"' "Y428 42 42 `"Ybbd8"' "Y428
.NET 6.0.0-preview.6.21276.13
Debian GNU/Linux bullseye/sid
OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiB
usage_in_bytes: 5898240 5.63 MiB
limit_in_bytes: 104857600 100.00 MiBWe do.
Windows -- CPU and memory limits
The major container improvement changes were made on Windows in .NET 6. Let's try with the .NET 6 preview 3. We're expecting that CPU limits are not honored with process isolated containers.
C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor mcr.microsoft.com/dotnet/runtime:6.0 dotnet dotnetapp.dll
42
42 ,d ,d
42 42 42
,adPPYb,42 ,adPPYba, MM42MMM 8b,dPPYba, ,adPPYba, MM42MMM
a8" `Y42 a8" "8a 42 42P' `"8a a8P_____42 42
8b 42 8b d8 42 42 42 8PP""""""" 42
"8a, ,d42 "8a, ,a8" 42, 42 42 "8b, ,aa 42,
`"8bbdP"Y8 `"YbbdP"' "Y428 42 42 `"Ybbd8"' "Y428
.NET 6.0.0-preview.3.21201.4
Microsoft Windows 10.0.19042
OSArchitecture: X64
ProcessorCount: 16
TotalAvailableMemoryBytes: 75.00 MiBAs expected, processor count is incorrect.
Note: You will see the correct behavior with Hyper-V isolated containers. The functionality gap is with process-isolated containers.
Let's try .NET 6 Preview 6.
C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor dotnet6runtime dotnet dotnetapp.dll
42
42 ,d ,d
42 42 42
,adPPYb,42 ,adPPYba, MM42MMM 8b,dPPYba, ,adPPYba, MM42MMM
a8" `Y42 a8" "8a 42 42P' `"8a a8P_____42 42
8b 42 8b d8 42 42 42 8PP""""""" 42
"8a, ,d42 "8a, ,a8" 42, 42 42 "8b, ,aa 42,
`"8bbdP"Y8 `"YbbdP"' "Y428 42 42 `"Ybbd8"' "Y428
.NET 6.0.0-preview.6.21272.4
Microsoft Windows 10.0.19042
OSArchitecture: X64
ProcessorCount: 3
TotalAvailableMemoryBytes: 75.00 MiBPerfect.
Now let's try CPU affinity, like we did with Linux.
C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpuset-cpus="1-3" -m 100mb --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_PROCESSOR_COUNT=10 dotnet6runtime dotnet dotnetapp.dll
docker: Error response from daemon: invalid option: Windows does not support CpusetCpus.
See 'docker run --help'.
It isn't supported. I didn't know that.
Windows -- custom processor count
Just like with Linux, let's try setting a custom processor count.
C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb -e DOTNET_PROCESSOR_COUNT=10 --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor dotnet6runtime dotnet dotnetapp.dll
42
42 ,d ,d
42 42 42
,adPPYb,42 ,adPPYba, MM42MMM 8b,dPPYba, ,adPPYba, MM42MMM
a8" `Y42 a8" "8a 42 42P' `"8a a8P_____42 42
8b 42 8b d8 42 42 42 8PP""""""" 42
"8a, ,d42 "8a, ,a8" 42, 42 42 "8b, ,aa 42,
`"8bbdP"Y8 `"YbbdP"' "Y428 42 42 `"Ybbd8"' "Y428
.NET 6.0.0-preview.6.21276.13
Microsoft Windows 10.0.19042
OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiBAs expected, it behaves just like demonstrated with Linux.
Validating isolation mode
Here's what I did to validate the isolation mode.
C:\git\dotnet-docker\samples\dotnetapp>docker run --rm -it --isolation=process --name dotnet6runtime dotnet6runtime
And then in another command prompt.
C:\>docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ee11156d0488 dotnet6runtime "c:\\windows\\system32…" 15 seconds ago Up 14 seconds dotnet6runtime
C:\>docker inspect dotnet6runtime | findstr Isolation
"Isolation": "process",
Validate container improvements with .NET 6
We've made various changes in .NET 6 to improve aspects of container resource limits governance, particularly for Windows containers. This issue demonstrates the new capabilities and their current behavior using the dotnetapp sample app.
Issues:
Changes:
The last change is to a sample, which will be used to demonstrate the others changes. It is a .NET 5 app. As a result, various roll-forward settings are used to coerce the app to run on .NET 6.
At the time of writing, the relevant changes have not been released in a public .NET 6 preview. As a result, I updated and locally rebuilt various Dockerfiles with daily builds from dotnet/installer.
Relevant docs:
docker run-- runtime constraints on resourcesLinux -- CPU and memory limits
The following test demonstrates the .NET runtime honoring CPU and resource limits.
Similar, but with CPU sets (AKA CPU affinity) is used instead.
The latest .NET 6 Preview is used for these examples, since these capabilities are all pre .NET 6.
Linux -- custom processor count
The following test demonstrates the .NET runtime honoring the new
DOTNET_PROCESSOR_COUNTENV, which provides a different value toEnvironment.ProcessorCountand the equivalent setting in the native runtime. It doesn't directly related to and does not affect thedocker run--cpusmechanism, but is intended as a higher-level signal for scaling algorithms.I used a .NET 6 Preview 6 build to demonstrate this change. It works as expected.
Let's double check that we get the right behavior if no CPU limits are set.
We do.
Let's check if we get the correct value with CPU affinity.
We do.
Windows -- CPU and memory limits
The major container improvement changes were made on Windows in .NET 6. Let's try with the .NET 6 preview 3. We're expecting that CPU limits are not honored with process isolated containers.
As expected, processor count is incorrect.
Note: You will see the correct behavior with Hyper-V isolated containers. The functionality gap is with process-isolated containers.
Let's try .NET 6 Preview 6.
Perfect.
Now let's try CPU affinity, like we did with Linux.
It isn't supported. I didn't know that.
Windows -- custom processor count
Just like with Linux, let's try setting a custom processor count.
As expected, it behaves just like demonstrated with Linux.
Validating isolation mode
Here's what I did to validate the isolation mode.
C:\git\dotnet-docker\samples\dotnetapp>docker run --rm -it --isolation=process --name dotnet6runtime dotnet6runtimeAnd then in another command prompt.