Tested on two systems with different OS, both failed with same error during hcttest.
Repro steps: (following the build document at: https://github.com/Microsoft/DirectXShaderCompiler)
- On a new system, installed Windows 10 Pro version 10.0.16299 or 10.0.17134 (RS4 Windows 10 version 1803: https://changewindows.org/milestone/redstone4)
- Installed Git for Windows from https://git-scm.com/download/win (64bit Git-2.18.0-64-bit.exe)
- Installed Visual Studio 2017 community edition from https://visualstudio.microsoft.com/downloads/
- Installed Python 2.7.15 from https://www.python.org/downloads/release/python-2715/
- On the system, did: git clone https://github.com/Microsoft/DirectXShaderCompiler.git
- After clone, set up a build environment shortcut by double-clicking the utils\hct\hctshortcut.js file, which created a shortcut on my desktop with a default configuration.
- click that shortcut on my desktop, it started a HLSL console
- From the HLSL console, ran the script at utils\hct\hctgettaef.py as python hctgettaef.py, it works fine
- From the HLSL console, ran: hctbuild, build successful
- From the HLSL console, ran: hcttest, encountered following error:

- Tried clicking "Retry" or "Ignore", there were no effects, the build continues, and here is the output:

- Here is the failed test error:
FileName: C:\Users\admin\AppData\Local\Temp\Tfm-13068-0\te.processhost.exe-20180729-084148.dmp
Error: TAEF: [HRESULT 0x800706BE] A failure occurred while running a test operation: 'CompilerTest::CompileWhenNoMemThenOOM'. (The process hosting the test code dropped the connection with the TAEF engine while invoking a test operation. It terminated with an exit code of 0x80000003.)
Command Line Usage Warnings:
The /labMode switch is being ignored as this setting is no longer supported.
EndGroup: CompilerTest::CompileWhenNoMemThenOOM [Failed]
TestSkipped: TAEF: The cleanup method 'TestModuleCleanup' will not be run as TAEF has stopped communicating with the test host process due to a previous failure.
Non-passing Tests:
CompilerTest::CompileWhenNoMemThenOOM [Failed]
Summary: Total=1203, Passed=1202, Failed=1, Blocked=0, Not Run=0, Skipped=0
You can debug the test with the following command line.
start devenv /debugexe TE.exe /inproc C:\github\hlsl.bin\Debug\test\clang-hlsl-tests.dll /p:"HlslDataDir=C:\github\DirectXShaderCompiler\tools\clang\test\HLSL" /select: "Peter G (@priority)<1"
Use this te.exe for out-of-proc, or pick the correct one for the target arch, currently x64.
C:\Program Files (x86)\windows kits\10\Testing\Runtimes\TAEF\TE.exe
C:\github\DirectXShaderCompiler\external\taef\build\Binaries\amd64\TE.exe
Use /name:TestClass* or /name:TestClass::MethodName to filter and /breakOnError to catch the failure.
1 file(s) copied.
Testing command line programs at C:\github\hlsl.bin\Debug\test ...
Smoke test for dxr command line program ...
warning: DXIL.dll not found. Resulting DXIL will not be signed for use in release environments.

- Tried using above repro command to debug, changed little bit according to my system settings:
"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\devenv.exe" /debugexe "C:\github\DirectXShaderCompiler\external\taef\build\Binaries\amd64\TE.exe" /inproc "C:\github\hlsl.bin\Debug\test\clang-hlsl-tests.dll" /p:"HlslDataDir=C:\github\DirectXShaderCompiler\tools\clang\test\HLSL" /select: "Peter G (@priority)<1"

- Te.exe triggers a breakpoint, and it is because of ptr is null pointer (looks like allocation error, my dev system has 16GB memory, so it should not be an memory issue, but DXC issue).
void * ptr = DxcGetThreadMallocNoRef()->Alloc(size);

This is a blocking bug,
Tested on two systems with different OS, both failed with same error during hcttest.
Repro steps: (following the build document at: https://github.com/Microsoft/DirectXShaderCompiler)
FileName: C:\Users\admin\AppData\Local\Temp\Tfm-13068-0\te.processhost.exe-20180729-084148.dmp
Error: TAEF: [HRESULT 0x800706BE] A failure occurred while running a test operation: 'CompilerTest::CompileWhenNoMemThenOOM'. (The process hosting the test code dropped the connection with the TAEF engine while invoking a test operation. It terminated with an exit code of 0x80000003.)
Command Line Usage Warnings:
EndGroup: CompilerTest::CompileWhenNoMemThenOOM [Failed]
TestSkipped: TAEF: The cleanup method 'TestModuleCleanup' will not be run as TAEF has stopped communicating with the test host process due to a previous failure.
Non-passing Tests:
Summary: Total=1203, Passed=1202, Failed=1, Blocked=0, Not Run=0, Skipped=0
You can debug the test with the following command line.
start devenv /debugexe TE.exe /inproc C:\github\hlsl.bin\Debug\test\clang-hlsl-tests.dll /p:"HlslDataDir=C:\github\DirectXShaderCompiler\tools\clang\test\HLSL" /select: "Peter G (@priority)<1"
Use this te.exe for out-of-proc, or pick the correct one for the target arch, currently x64.
C:\Program Files (x86)\windows kits\10\Testing\Runtimes\TAEF\TE.exe
C:\github\DirectXShaderCompiler\external\taef\build\Binaries\amd64\TE.exe
Use /name:TestClass* or /name:TestClass::MethodName to filter and /breakOnError to catch the failure.
1 file(s) copied.
Testing command line programs at C:\github\hlsl.bin\Debug\test ...
Smoke test for dxr command line program ...
warning: DXIL.dll not found. Resulting DXIL will not be signed for use in release environments.
"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\devenv.exe" /debugexe "C:\github\DirectXShaderCompiler\external\taef\build\Binaries\amd64\TE.exe" /inproc "C:\github\hlsl.bin\Debug\test\clang-hlsl-tests.dll" /p:"HlslDataDir=C:\github\DirectXShaderCompiler\tools\clang\test\HLSL" /select: "Peter G (@priority)<1"
void * ptr = DxcGetThreadMallocNoRef()->Alloc(size);
This is a blocking bug,