First of all, thank you for the amazing work: transformers-cfg really makes it easy to build ebnf-based constrained decoding 🚀

While trying to do so with DeepSeek-Coder (e.g. of the 1.3b model), I have realised that even though it is LLaMa-based, transformers-cfg does not support it.

After deep-diving into the code of the package I found the root: in the ByteProxyMapping, the decode_proxytoken2bytes has ad hoc patches for BPE. The problem is that the space before tokens, in DeepSeek, is not encoded using _, but using Ġ instead. In addition, there are other special cases, e.g., \t is encoded as ĉ, and \n as Ċ.

To patch it, I am overriding the decode_proxytoken2bytes with the bellow:

def decode_proxytoken2bytes(self: ByteProxyMapping, proxy_token: str) -> bytes:
    """Hammered function to replace the deepseek non ascii tokens."""
    if proxy_token.startswith("<0x"):
        hex_value: str = proxy_token[3:-1]
        return bytes.fromhex(hex_value)
    else:
        # ad hoc fix for BPE
        if "Ċ" in proxy_token:
            proxy_token = proxy_token.replace("Ċ", "\n")
        if "ĉ" in proxy_token:
            proxy_token = proxy_token.replace("ĉ", "\t")
        if proxy_token.startswith("Ġ"):
            proxy_token = proxy_token.replace("Ġ", " ")
        return proxy_token.encode("utf-8")

But certainly, a cleaner and more generalisable approach, that is tokenizer independent exists 🤔