Certain combinations of [EXPERIMENTAL] llama_context_params are not always supported:

    struct llama_context_params {
        ...

        enum ggml_type type_k; // data type for K cache [EXPERIMENTAL]
        enum ggml_type type_v; // data type for V cache [EXPERIMENTAL]

        bool flash_attn;  // whether to use flash attention [EXPERIMENTAL]

        ...
    };

Here is a list of known incompatibilities (we can try to update it in the future):

• flash_attn == true && type_k == F16 && type_v == F16

  • CPU
    • Generally slower compared to no-FA, mostly used for testing purposes
  • CUDA
  • Metal
    • Slow or can't build for models with head size = 256 (Metal (iOS): Compute function exceeds available temporary registers #7261)
  • Vulkan
  • SYCL

• flash_attn == true && (type_k != F16 || type_v != F16)

  • CPU
  • CUDA - partial support (CUDA: quantized KV support for FA vec #7527)
  • Metal
  • Vulkan
  • SYCL

• flash_attn == false && type_v != F16

  • Not supported because the V cache is stored transposed, which prevents quantization