In some games, you can't set the borderless mode for vulcan, the settings in game not work.

"Valheim" work on exclusive fullscreen - (X)
"Tribes of Midgard" work on borderless for game settings- (V)

Not entirely sure what you mean? The problem that the dialogue mode option solves does not really exist in D3D11.

This command allows you to run the application in bordeless fullscreen instead of an exclusive full screen on dx9.
In dx11 applications, vulkan is launched in exclusive full screen mode in most cases. If it is possible to launch a vulkan borderless mode, I would like to use it.

Not entirely sure what you mean? The problem that the dialogue mode option solves does not really exist in D3D11.

How about making "VK_EXT_full_screen_exclusive" controllable from a config file? Name it dxvk.AllowExclusiveFullscreen, (Auto/True/False).

The extension is a Windows only thing fwiw which is not officially a target platform.

On Windows, Nvidia & AMD drivers use DXGI Swapchain for Vulkan apps.
This effectively is Borderless Fullscreen.

That means even if you select Exclusive/Fullscreen, as long as it uses Vulkan and you don't see any window borders, it will be borderless fullscreen.

tldr: Don't use "Borderless" settings in game options.

lextra2
No.
Vulkan ignores the game settings and always uses the exclusive full screen in some games (black screen when Alt+Tab), in others it uses the game settings and runs in exclusive and Borderless mode (good games)

Sounds more like an EDID issue.
Game picks a different resolution/refresh rate then what your desktop uses > mode change on alt-tab.

You can fix that really easily with CRU
Delete all resolutions but the one you are using.
The resolution you are using should have only one entry.
Example: 2560x1440@165Hz

Alternatively you could use SpecialK
Enable multitasking setting+continues rendering