Skip to content

Internal server error: unhashable type: 'dict' #26405

New issue
New issue
Closed as not planned
Closed as not planned
Internal server error: unhashable type: 'dict'#26405
Assignees
rekram1-node

Description

@Mestoph
Mestoph
opened on May 8, 2026

Description

When i use some model like kimi k2.6 i get all time the message :

Internal server error: unhashable type: 'dict'

Plugins

none

OpenCode version

lasted

Steps to reproduce

All time with kimi k2.6 provider.

End of the error message :

Part of the log with error message :

"responseBody":"{\"error\":{\"object\":\"error\",\"message\":\"Internal server error: unhashable type: 'dict'\",\"type\":\"InternalServerError\",\"param\":null,\"code\":500}}","isRetryable":true,"data":{"error":{"message":"Internal server error: unhashable type: 'dict'","type":"InternalServerError","param":null,"code":500}}}} stream error

Screenshot and/or share link

No response

Operating System

Windows 11

Terminal

Windows Terminal

Activity

  1. github-actions
    assigned
    rekram1-node
    on May 8, 2026

  2. Mestoph commented on May 23, 2026

    @Mestoph
    Mestoph
    on May 23, 2026
    Author

    Hi everyone :)

    Tool handling is different with Kimi K2.6, and the default opencode protocol is no longer compatible with it.

    🐛 NVIDIA NIM Tooling Bug Report – unhashable type: 'dict' (Kimi K2.6)

    🧩 Overview

    When using OpenAI-compatible tool schemas with NVIDIA NIM, especially with Kimi K2.6, requests containing structured parameters.properties may fail with a 500 Internal Server Error.

    Internal server error: unhashable type: 'dict'

    This breaks standard tool calling workflows (shell, git, filesystem, etc.).

    📦 Environment

    ❌ PROBLEM

    When sending tools with structured JSON schemas:

    {
  "type": "function",
  "function": {
    "name": "git_status",
    "description": "Get git status of a repository",
    "parameters": {
      "type": "object",
      "properties": {
        "repo_path": {
          "type": "string",
          "description": "Path to repository"
        }
      },
      "required": ["repo_path"]
    }
  }
}

    ❌ Error returned:

    500 Internal Server Error
unhashable type: 'dict'

    🧠 WHY IT HAPPENS (Root Cause)

    The backend likely processes tool schemas in Python and uses dictionaries in hash-based structures.

    Problematic structure:

    {"repo_path": {"type": "string"}}

    Python limitation:

    hash({"type": "string"})  # TypeError

    ➡️ Result: crash during schema processing.

    🎯 WHY IT MAINLY AFFECTS KIMI K2.6

    Kimi K2.6 appears to:

    • Strictly validate tool schemas
    • Reprocess parameters internally
    • Reject complex nested OpenAI schemas

    🔁 REPRODUCTION EXAMPLE (Git Tool)

    Tool definition

    {
  "type": "function",
  "function": {
    "name": "git_status",
    "description": "Get git status of repository",
    "parameters": {
      "type": "object",
      "properties": {
        "repo_path": {
          "type": "string"
        }
      },
      "required": ["repo_path"]
    }
  }
}

    User prompt

    Check git status of my project

    ❌ Result

    500 Internal Server Error
unhashable type: 'dict'

    🛠️ SOLUTION (WORKAROUND)

    Since the issue is server-side, the fix is preprocessing.

    🔧 STEP 1 — Strip schema

    BEFORE ❌

    "parameters": {
  "type": "object",
  "properties": {
    "repo_path": {
      "type": "string"
    }
  }
}

    AFTER ✅

    "parameters": {
  "type": "object"
}

    🔧 STEP 2 — Move schema to description

    Get git status of repository

Parameters:
- repo_path: string (required)

    🔧 STEP 3 — Safe final tool

    {
  "type": "function",
  "function": {
    "name": "git_status",
    "description": "Get git status of repository\n\nParameters:\n- repo_path: string (required)",
    "parameters": {
      "type": "object"
    }
  }
}

    ⚙️ OPTIONAL PROXY FIX

    A middleware proxy can automatically:

    • Remove properties
    • Flatten schema
    • Inject parameters into description
    • Prevent 500 errors

    🧱 AFFECTED TOOLS

    • git_status
    • shell_exec
    • list_files
    • any tool using JSON schema properties

    📌 IMPACT

    • Breaks OpenAI tool compatibility
    • Blocks structured function calling
    • Requires preprocessing layer
    • Affects multi-tool systems

    🚀 RESULT AFTER FIX

    Before After
    500 error 200 OK
    crash stable
    schema used flattened
    tools broken working

    🧾 SUMMARY

    • Problem: NIM crashes on nested tool schemas
    • Cause: Python dict hashing limitation
    • Affected model: Kimi K2.6
    • Fix: strip schema + move to description
    • Best solution: proxy middleware

    SOLUTION

    The only solution I found without adding specific conditions directly into the opencode source code was to create a small proxy.

    #!/usr/bin/env node
/**
 * FR: kimi-proxy-k2.6.mjs — Proxy local pour Kimi K2.6 (NVIDIA NIM)
 * ------------------------------------------------------------------
 * Ce proxy s'interpose entre un client OpenAI-compatible (opencode)
 * et l'API NVIDIA NIM qui héberge le modèle Moonshot Kimi K2.6.
 *
 * Problèmes adressés :
 *   1. Bug 500 NIM ("unhashable type: 'dict'") : certains schémas
 *      JSON complexes dans les outils font planter le backend Python.
 *      Le proxy strippe les paramètres et les injecte dans la description.
 *   2. Rate-limiting strict (40 RPM / TPM opaque) : sérialisation des
 *      requêtes, fenêtre glissante TPM et backoff exponentiel 429.
 *
 * EN: Local proxy layer for Kimi K2.6 on NVIDIA NIM.
 * ------------------------------------------------------------------
 * Intermediates between an OpenAI-compatible client and the NIM API.
 * Addresses: 1) NIM 500 crash on tool schemas, 2) strict rate limits.
 *
 * Target model : Moonshot Kimi K2.6 (NVIDIA NIM endpoint)
 * Upstream     : https://integrate.api.nvidia.com/v1
 * Local proxy  : http://localhost:8787/v1
 *
 * ┌──────────┐  HTTP/1.1   ┌──────────────────────┐  HTTPS/TLS      ┌─────────────────┐
 * │  Client  │ ───────────►│ Proxy (localhost)    │ ───────────────►│  NVIDIA NIM API │
 * │ (OpenAI) │ (req JSON)  │ kim-proxy-k2.6.mjs   │ (req modifiée)  │ integrate.api...│
 * └──────────┘             │                      │                 └─────────────────┘
 *      ▲                   │ • Patch Tools        │                          │
 *      │                   │ • Scheduling RPM     │                          │
 *      │                   │ • TPM Window         │                          │
 *      └───────────────────┘ • Retry 429          │                          │
 *         (streaming SSE)  │ • Stream pass-through│                          │
 *                          └──────────────────────┘                          │
 *                                     ▲                                      │
 *                                     └────────────←←←←←←←←←←←←←←←←←←←←←←←←←←┘
 */

/* ------------------------------------------------------------------ */
/*  FR: Imports des modules Node.js natifs.                           */
/*  EN: Node.js built-in module imports.                              */
/* ------------------------------------------------------------------ */
import http from "http";   // FR: Serveur HTTP local / EN: Local HTTP server
import https from "https"; // FR: Client HTTPS vers NIM / EN: HTTPS client to NIM
import fs   from "fs";     // FR: Écriture du fichier de debug / EN: Debug file writing
import util from "util";   // FR: Inspection profonde d'objets / EN: Deep object inspection

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: CONSTANTES & VARIABLES D'ENVIRONNEMENT                       */
/*  EN: ENVIRONMENT CONSTANTS & CONFIGURATION                        */
/* ═════════════════════════════════════════════════════════════════ */

/**
 * FR: Niveau de débogage. 1 (par défaut) active l'écriture dans un fichier.
 * EN: Debug level. 1 (default) enables file logging.
 * @type {number}
 */
const DEBUG = parseInt(process.env.DEBUG        || "1",   10);

/**
 * FR: Port d'écoute du proxy local.
 * EN: Local listening port.
 * @type {number}
 */
const PORT            = parseInt(process.env.PORT        || "8787",   10);

/**
 * FR: Nombre max de tentatives avant abandon sur 429.
 * EN: Max retry attempts before giving up on 429.
 * @type {number}
 */
const MAX_RETRIES     = parseInt(process.env.MAX_RETRIES || "5",      10);

/**
 * FR: Budget Tokens Per Minute (TPM) supposé pour éviter les rejets
 *     silencieux. Adaptez cette valeur si vous disposez d'un compte payant.
 * EN: Assumed TPM budget to avoid silent rejects. Increase for paid tiers.
 * @type {number}
 */
const ASSUMED_TPM     = parseInt(process.env.ASSUMED_TPM || "500000", 10);

/**
 * FR: Durée avant rupture de connexion vers NVIDIA (ms).
 * EN: Upstream request timeout in milliseconds.
 * @type {number}
 */
const REQUEST_TIMEOUT = parseInt(process.env.TIMEOUT     || "300000", 10);

/**
 * FR: Cible NVIDIA NIM (hôte et préfixe d'API OpenAI-compatible).
 * EN: NVIDIA NIM host and OpenAI-compatible base path.
 */
const NVIDIA_HOST         = "integrate.api.nvidia.com";
const NVIDIA_BASE = "/v1";

/* ------------------------------------------------------------------ */
/*  FR: Limites de débit (RPM) et gestion du backoff                  */
/*  EN: Rate limits (RPM) and backoff configuration                   */
/* ------------------------------------------------------------------ */
/**
 * FR: Requêtes/min sécurisées. L'intervalle inter-requête est calculé
 *     à partir de cette constante et de RPM_WINDOW.
 * EN: Safe requests per minute. Inter-request delay is derived from this.
 */
const MAX_RPM             = 30;       // FR: Requêtes/min sécurisées / EN: Safe req/min

/**
 * FR: Fenêtre utilisée pour le calcul du délai inter-requête et pour le
 *     reset du compteur TPM (en ms). Valeur par défaut : 1 heure (3600000).
 * EN: Window used for inter-request delay and TPM counter reset (ms).
 */
const RPM_WINDOW          = 3600_000;   // FR: Fenêtre RPM/TPM en ms / EN: RPM/TPM window (ms)

const INTER_REQUEST_DELAY = RPM_WINDOW / MAX_RPM; // FR: Pause fixe inter-requête / EN: Fixed inter-request delay
const MAX_COOLDOWN        = 300_000;  // FR: Plafond backoff 429 (5 min) / EN: 429 backoff cap (5 min)

/* ------------------------------------------------------------------ */
/*  FR: MODE DEBUG - Journalisation fichier                            */
/*  EN: DEBUG MODE - File logging                                      */
/* ------------------------------------------------------------------ */
let   debugStream = null;

if (DEBUG) {
  debugStream = fs.createWriteStream("kimi-proxy-debug.log", { flags: "a" });
  // FR: Capture des crashs nonattrapés pour ne rien perdre au fichier.
  // EN: Capture uncaught exceptions so nothing is lost to the log.
  process.on("uncaughtException", (err) => {
    const line = `[${new Date().toISOString()}] [FATAL] ${err.stack}\n`;
    debugStream.write(line, () => process.exit(1));
  });
}

/**
 * FR: Tente de parser une chaîne JSON sans jamais lever d'exception.
 * EN: Safely parse JSON without ever throwing.
 *
 * @param {string} str - Chaîne candidate.
 * @returns {any|string} - Arbre parsé ou la chaîne brute si impossible.
 */
function tryParseJson(str) {
  try {
    return JSON.parse(str);
  } catch {
    return str;
  }
}

/**
 * FR: Formate une chaîne JSON de manière lisible si possible.
 * EN: Pretty-print a string as JSON if possible.
 *
 * @param {string} str - Chaîne source.
 * @returns {string} - JSON indenté ou la chaîne d'origine.
 */
function tryPrettyJson(str) {
  try {
    return JSON.stringify(tryParseJson(str), null, 2);
  } catch {
    return str;
  }
}

/**
 * FR: Écrit une section structurée dans le fichier de débogage.
 *     Accepte Buffers, chaînes et objets complexes.
 * EN: Write a structured section into the debug log file.
 *     Handles Buffers, strings and complex objects.
 *
 * @param {string} section - Nom de la section journalisée.
 * @param {any} data - Payload à enregistrer.
 */
function debugFile(section, data) {
  if (!debugStream) return;

  const time = new Date().toISOString();
  let payload;

  if (Buffer.isBuffer(data)) {
    payload = tryPrettyJson(data.toString("utf8"));
  } else if (typeof data === "string") {
    payload = tryPrettyJson(data);
  } else if (typeof data === "object" && data !== null) {
    try {
      payload = JSON.stringify(data, null, 2);
    } catch {
      payload = util.inspect(data, {
        depth: null,
        colors: false,
        compact: false,
        breakLength: 120,
        maxArrayLength: null,
      });
    }
  } else {
    payload = String(data);
  }

  debugStream.write(`[${time}] [${section}]\n${payload}\n${"─".repeat(80)}\n`);
}

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: AGENT HTTPS (Keep-Alive)                                     */
/*  EN: HTTPS KEEP-ALIVE AGENT                                      */
/* ═════════════════════════════════════════════════════════════════ */

/**
 * FR: Agent HTTPS avec connexions persistantes.
 *     Évite de recréer un handshake TLS à chaque requête.
 *     maxSockets: 2 limite la concurrence pour respecter le RPM.
 * EN: Reusable HTTPS agent with persistent connections.
 *     maxSockets: 2 enforces low concurrency to respect RPM limits.
 */
const httpsAgent = new https.Agent({
  keepAlive:  true,
  maxSockets: 2,
  timeout:    REQUEST_TIMEOUT,
});

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: PALETTE ANSI & UTILITAIRES D'AFFICHAGE                       */
/*  EN: ANSI COLOR PALETTE & DISPLAY UTILITIES                       */
/* ═════════════════════════════════════════════════════════════════ */

/** FR: Codes de style pour le terminal. EN: Terminal styling codes. */
const C = {
  reset:   "\x1b[0m",   // Annulation de style
  bold:    "\x1b[1m",   // Gras
  dim:     "\x1b[2m",   // Gris / texte secondaire
  white:   "\x1b[97m",  // Blanc vif (évite le 'undefined' si oublié)
  green:   "\x1b[92m",  // Vert (succès)
  yellow:  "\x1b[93m",  // Jaune (avertissement, retry)
  red:     "\x1b[91m",  // Rouge (erreur critique)
  cyan:    "\x1b[96m",  // Cyan (données sortantes / tokens)
  blue:    "\x1b[94m",  // Bleu (données entrantes / réponse NIM)
  magenta: "\x1b[95m",  // Magenta (patch tools, infos TPM)
};

/**
 * FR: Timestamp localement formaté en gris.
 * EN: Grey formatted local timestamp.
 * @returns {string}
 */
const ts    = () => `${C.dim}${new Date().toLocaleTimeString("fr-FR")}${C.reset}`;

/**
 * FR: Pause asynchrone.
 * EN: Async sleep utility.
 * @param {number} ms - Millisecondes.
 */
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

/**
 * FR: Barre de progression ASCII colorée.
 * EN: Colored ASCII progress bar.
 *
 * @param {number} pct - Pourcentage entre 0 et 1.
 * @param {number} width - Largeur en colonnes.
 * @returns {string}
 */
function buildBar(pct, width) {
  const filled = Math.round(Math.max(0, Math.min(pct, 1)) * width);
  const color  = pct > 0.9 ? C.red : pct > 0.6 ? C.yellow : C.green;
  return `${color}[${"█".repeat(filled)}${"░".repeat(width - filled)}]${C.reset}`;
}

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: CENTRALISATION DES LOGS                                      */
/*  EN: CENTRALIZED LOGGING                                          */
/* ═════════════════════════════════════════════════════════════════ */

/**
 * FR: Objets de log par catégorie, chacun préfixé par une couleur
 *     et un tag explicite pour repérage immédiat dans le terminal.
 * EN: Categorized loggers with color-coded prefixes.
 */
const log = {
  ok    : (...a) => console.log (`${C.green}${C.bold}[proxy]${C.reset}`,   ...a),
  warn  : (...a) => console.warn(`${C.yellow}${C.bold}[warn]${C.reset}`,   ...a),
  error : (...a) => console.error(`${C.red}${C.bold}[error]${C.reset}`,    ...a),
  patch : (...a) => console.log (`${C.magenta}${C.bold}[patch]${C.reset}`, ...a),
  req   : (...a) => console.log (`${C.cyan}${C.bold}[→ nim]${C.reset}`,    ...a),
  nim   : (...a) => console.log (`${C.blue}${C.bold}[← nim]${C.reset}`,    ...a),
  retry : (...a) => console.log (`${C.yellow}${C.bold}[retry]${C.reset}`,  ...a),
  tpm   : (...a) => console.log (`${C.magenta}${C.bold}[tpm]${C.reset}`,   ...a),
};

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: ESTIMATION DE CONTEXTE (TOKENS)                              */
/*  EN: TOKEN CONTEXT ESTIMATION                                     */
/* ═════════════════════════════════════════════════════════════════ */

/**
 * FR: Dernier total de tokens réellement retourné par NIM.
 *     Attention : variable globale mutable, utilisée comme heuristique
 *     de contexte pour la prochaine requête.
 * EN: Last known token total returned by NIM. Warning: mutable global,
 *     used as heuristic context for the next request.
 */
let lastKnownTotal = 0;

/**
 * FR: Estime le nombre de tokens d'après une chaîne ou un objet.
 *     Heuristique : ~3,5 caractères = 1 token ; caractères multi-octets comptent double.
 * EN: Quick-and-dirty token estimator.
 *
 * @param {string|any} text
 * @returns {number}
 */
function estimateTokens(text) {
  const str = typeof text === "string" ? text : JSON.stringify(text);
  const multiByte = (str.match(/[\u0080-\uFFFF]/g) || []).length;
  return Math.ceil((str.length + multiByte) / 3.5);
}

/**
 * FR: Retourne le meilleur chiffre de contexte dispo :
 *     total réel NIM si déjà reçu, sinon estimation locale.
 * EN: Returns the best available token count: real NIM total or local estimate.
 *
 * @param {string} rawBody
 * @returns {number}
 */
function getEffectiveTokens(rawBody) {
  return lastKnownTotal > 0 ? lastKnownTotal : estimateTokens(rawBody);
}

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: GESTION DU BUDGET TPM (SLIDING WINDOW)                       */
/*  EN: TPM BUDGET MANAGEMENT (SLIDING WINDOW)                       */
/* ═════════════════════════════════════════════════════════════════ */

let tpmUsed = 0;                 // FR: Tokens réservés dans la fenêtre / EN: Reserved tokens in window
let tpmWindowStart = Date.now(); // FR: Début de la fenêtre courante / EN: Current window start

/**
 * FR: Réinitialise la fenêtre glissante si les 60 s sont écoulées.
 * EN: Reset sliding window once 60 seconds have elapsed.
 */
function resetWindowIfNeeded() {
  if (Date.now() - tpmWindowStart >= RPM_WINDOW) {
    tpmUsed        = 0;
    tpmWindowStart = Date.now();
  }
}

/**
 * FR: Réserve un budget de tokens avant d'envoyer.
 *     Bloque si le plafond ASSUMED_TPM est dépassé jusqu'à la fin de la fenêtre.
 * EN: Acquire a token budget before sending. Blocks if ASSUMED_TPM ceiling is hit.
 *
 * @param {number} tokens
 */
async function acquireTokens(tokens) {
  resetWindowIfNeeded();

  if (tpmUsed + tokens > ASSUMED_TPM) {
    const waitMs = Math.max(0, RPM_WINDOW - (Date.now() - tpmWindowStart)) + 200;
    log.tpm(`budget ~${Math.round(tpmUsed / 1000)}k/${Math.round(ASSUMED_TPM / 1000)}k — freeze ${Math.round(waitMs / 1000)}s`);
    await sleep(waitMs);
    tpmUsed        = 0;
    tpmWindowStart = Date.now();
  }
  tpmUsed += tokens;
}

/**
 * FR: Libère les tokens précédemment réservés (échec ou correction après usage réel).
 * EN: Release previously reserved tokens on failure or after real usage correction.
 *
 * @param {number} tokens
 */
function releaseTokens(tokens) {
  tpmUsed = Math.max(0, tpmUsed - tokens);
}

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: COOLDOWN GLOBAL LORS D'UN 429                                */
/*  EN: GLOBAL COOLDOWN ON 429                                       */
/* ═════════════════════════════════════════════════════════════════ */

let globalCooldownUntil = 0; // FR: Timestamp limite du gel / EN: Cooldown expiry timestamp

/**
 * FR: Déclenche un gel global : toutes les requêtes en attente
 *     pauseront jusqu'à expiration, avec un plafond de sécurité.
 * EN: Trigger a global freeze; all queued requests will wait until expiry.
 *
 * @param {number} ms
 */
function setGlobalCooldown(ms) {
  const until = Date.now() + Math.min(ms, MAX_COOLDOWN);
  if (until > globalCooldownUntil) globalCooldownUntil = until;
}

/**
 * FR: Attente silencieuse tant qu'un cooldown global est actif.
 * EN: Silent wait while a global cooldown is active.
 */
async function waitGlobalCooldown() {
  const remaining = globalCooldownUntil - Date.now();
  if (remaining > 0) await sleep(remaining);
}

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: SCHEDULER DE DÉPART VERS NIM (n'attend PAS la réponse)       */
/*  EN: NIM DEPARTURE SCHEDULER (does NOT wait for the response)     */
/* ═════════════════════════════════════════════════════════════════ */

/**
 * FR: Timestamp du prochain départ autorisé. Permet d'espacer les envois
 *     sans attendre que la réponse de la requête précédente soit terminée.
 * EN: Timestamp of next allowed departure. Spaces out sends without waiting
 *     for the previous request's response to finish.
 */
let nextDepartureAt = 0;

/**
 * FR: File logique pour garantir l'ordre des départs sans bloquer les
 *     pipelines de réponse. Chaque maillon ne vit que le temps d'un sleep.
 * EN: Logical queue guaranteeing departure order without blocking response
 *     pipelines. Each link only lives for the duration of a short sleep.
 */
let schedulerTail = Promise.resolve();

/**
 * FR: Réserve un créneau de DÉPART uniquement.
 *     Garantit que deux requêtes ne partent pas vers NIM avec un intervalle
 *     inférieur à INTER_REQUEST_DELAY.
 *     Cette promesse résout immédiatement après le départ autorisé.
 * EN: Reserves a DEPARTURE slot only. Ensures two requests don't leave for
 *     NIM closer than INTER_REQUEST_DELAY apart. Resolves right after the
 *     allowed departure time.
 *
 * @returns {Promise<void>}
 */
function acquireDepartureSlot() {
  let releaseMe;
  const myTicket = new Promise((resolve) => { releaseMe = resolve; });

  const myLink = schedulerTail;

  schedulerTail = myLink.then(async () => {
    // 1. Respecter le cooldown global 429 (si actif)
    await waitGlobalCooldown();

    // 2. Attendre le temps minimum avant prochain départ
    const now = Date.now();
    const waitFor = nextDepartureAt - now;
    if (waitFor > 0) {
      await sleep(waitFor);
    }

    // 3. Réserver le créneau suivant pour la prochaine requête
    nextDepartureAt = Date.now() + INTER_REQUEST_DELAY;

    // 4. Libérer le ticket de notre requête : son handler peut maintenant
    //    appeler doRequest(). La réponse de NIM sera totalement indépendante.
    releaseMe();
  }).catch((err) => {
    // FR: Sécurité : si un bug interne survient dans le scheduler,
    //     on libère quand même le ticket et on réinitialise le timestamp
    //     pour éviter de bloquer pour toujours les futures requêtes.
    // EN: Safety valve: on internal scheduler bug, still release the ticket
    //     and reset the timestamp to avoid wedging future requests forever.
    console.error(`${C.red}[scheduler error]${C.reset} (should never happen):`, err);
    nextDepartureAt = Math.max(nextDepartureAt, Date.now());
    releaseMe();
  });

  return myTicket;
}

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: LOGIQUE DE BACKOFF APRÈS 429                                 */
/*  EN: BACKOFF LOGIC AFTER 429                                      */
/* ═════════════════════════════════════════════════════════════════ */

/**
 * FR: Calcule le délai avant réessai selon trois sources possibles :
 *     1. Header Retry-After, 2. Header X-RateLimit-Reset, 3. Exponentiel maison.
 * EN: Compute retry delay from: 1) Retry-After header, 2) X-RateLimit-Reset, 3) Exponential backoff.
 *
 * @param {number} attempt - Tentative actuelle (1-based).
 * @param {string|undefined} retryAfterHeader
 * @param {string|undefined} rateLimitReset
 * @returns {number} - Délai en millisecondes.
 */
function backoffDelay(attempt, retryAfterHeader, rateLimitReset) {
  const fromHeader = Number(retryAfterHeader);
  if (!isNaN(fromHeader) && fromHeader > 0) return fromHeader * 1000;

  const fromRateLimit = Number(rateLimitReset);
  if (!isNaN(fromRateLimit) && fromRateLimit > 0) return fromRateLimit * 1000;

  const base   = Math.min(RPM_WINDOW * Math.pow(2, attempt - 1), 300_000);
  const jitter = base * 0.1 * (Math.random() * 2 - 1);
  return Math.round(base + jitter);
}

/**
 * FR: Affiche un compte à rebours visuel toutes les 5 secondes pendant le gel.
 * EN: Display a visual countdown every 5 seconds during cooldown.
 *
 * @param {number} ms
 * @param {number} attempt
 */
async function showCountdown(ms, attempt) {
  const secs = Math.round(ms / 1000);
  log.retry(`${C.yellow}429 COOLDOWN${C.reset} (tentative ${attempt}/${MAX_RETRIES}) — gel ${C.cyan}${secs}s${C.reset}  ${ts()}`);
  const step = 5_000;
  for (let remaining = ms; remaining > 0; remaining -= step) {
    const pct = remaining / ms;
    process.stdout.write(`\r  ${C.yellow}[429]${C.reset} ${buildBar(pct, 30)} ${C.cyan}${Math.round(remaining / 1000)}s${C.reset}   `);
    await sleep(Math.min(step, remaining));
  }
  process.stdout.write("\r" + " ".repeat(60) + "\r");
}

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: ENVOI DE LA REQUÊTE VERS L'UPSTREAM NVIDIA NIM               */
/*  EN: FORWARDING REQUEST TO NVIDIA NIM UPSTREAM                    */
/* ═════════════════════════════════════════════════════════════════ */

/**
 * FR: Effectue la requête HTTPS vers l'API NVIDIA.
 *     Retourne soit un stream (succès) soit un buffer d'erreur (≥400).
 * EN: Perform the HTTPS request to NVIDIA API.
 *     Returns either a readable stream (success) or an error buffer (≥400).
 *
 * @param {object} options - Paramètres pour https.request.
 * @param {Buffer} buf - Corps de la requête sérialisé.
 * @returns {Promise<{sc:number, headers:object, errBody:Buffer|null, stream:Stream|null}>}
 */
function doRequest(options, buf) {
  return new Promise((resolve, reject) => {
    const req = https.request({ ...options, agent: httpsAgent }, (upstream) => {
      const sc = upstream.statusCode;

      if (sc >= 400) {
        const chunks = [];
        upstream.on("data", (c) => chunks.push(c));
        upstream.on("end",  () => resolve({ sc, headers: upstream.headers, errBody: Buffer.concat(chunks), stream: null }));
        upstream.on("error", reject);
      } else {
        resolve({ sc, headers: upstream.headers, errBody: null, stream: upstream });
      }
    });

    req.setTimeout(REQUEST_TIMEOUT, () => {
      req.destroy(new Error(`upstream timeout after ${REQUEST_TIMEOUT}ms`));
    });

    req.on("error", reject);
    req.write(buf);
    req.end();
  });
}

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: PATCH DES TOOLS (CONTOURNEMENT BUG 500 NIM)                  */
/*  EN: TOOL PARAMETER STRIPPING (NIM 500 WORKAROUND)                */
/* ═════════════════════════════════════════════════════════════════ */

/**
 * FR: Cache mémoire des outils déjà patchés pour éviter le
 *     recalcul systématique à l'identique.
 * EN: In-memory cache for already-patched tools to avoid recomputation.
 */
const toolCache = new Map();

/**
 * FR: Supprime le schéma JSON des paramètres d'un outil et
 *     migre ces informations dans la description textuelle.
 *     Contourne le crash Python "unhashable type: 'dict'" de NIM.
 * EN: Strip JSON schema from tool parameters and move them into the
 *     text description. Works around NIM's Python "unhashable type: 'dict'" crash.
 *
 * @param {object} tool - Tool au format OpenAI compatible.
 * @returns {object} - Tool édulcoré pour NIM.
 */
function stripToolParameters(tool) {
  if (!tool || tool.type !== "function" || !tool.function) return tool;

  const cacheKey = JSON.stringify(tool);
  if (toolCache.has(cacheKey)) return toolCache.get(cacheKey);

  const fn = tool.function;
  const params = fn.parameters;
  let desc = (fn.description || "").trim();

  /* FR: Réinjection textuelle des paramètres pour que le LLM comprenne
   *     quand même les arguments attendus. */
  if (params?.properties && Object.keys(params.properties).length > 0) {
    const required = new Set(params.required || []);
    const props = Object.entries(params.properties)
      .map(([name, schema]) => {
        const type = schema.type || "any";
        const details = schema.description ? ` — ${schema.description}` : "";
        const reqFlag = required.has(name) ? " (required)" : " (optional)";
        return `  - ${name}: ${type}${reqFlag}${details}`;
      })
      .join("\n");
    desc += `\n\nParameters:\n${props}`;
  }

  const patched = {
    type: "function",
    function: {
      name: fn.name,
      description: desc.slice(0, 4096),
      parameters: { type: "object" },
    },
  };

  toolCache.set(cacheKey, patched);
  return patched;
}

/**
 * FR: Transforme le corps JSON de la requête pour NIM :
 *     - Désactive parallel_tool_calls.
 *     - Strippe les schémas des tools.
 * EN: Mutate the outgoing request body for NIM:
 *     - Disable parallel_tool_calls.
 *     - Strip tool parameter schemas.
 *
 * @param {string} raw - Body JSON brut.
 * @returns {string} - Body JSON modifié ou inchangé si aucun patch.
 */
function patchBody(raw) {
  let body;
  try {
    body = JSON.parse(raw);
  } catch {
    return raw;
  }
  let changed = false;

  if ("parallel_tool_calls" in body) {
    body.parallel_tool_calls = false;
    changed = true;
  }
  if (Array.isArray(body.tools) && body.tools.length > 0) {
    body.tools = body.tools.map(stripToolParameters).filter(Boolean);
    changed = true;
    log.patch(`${body.tools.length} tool(s) stripped  ${ts()}`);
  }
  
  return changed ? JSON.stringify(body) : raw;
}

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: SERVEUR PROXY HTTP LOCAL                                     */
/*  EN: LOCAL HTTP PROXY SERVER                                      */
/* ═════════════════════════════════════════════════════════════════ */

const server = http.createServer((req, res) => {
  const chunks = [];
  req.on("data", (chunk) => chunks.push(chunk));
  req.on("end", async () => {
    let rawBody = Buffer.concat(chunks).toString();

    /* FR: Nettoyage des headers sensibles ou problématiques pour NIM.
     * EN: Remove headers that NIM doesn't like or that may leak info. */
    delete req.headers["accept-encoding"];
	delete req.headers["x-title"];
	delete req.headers["x-billing-invoke-origin"];
	delete req.headers["http-referer"];
	delete req.headers["user-agent"];

    /* FR: TRACE : ce qui arrive au proxy depuis le client. */
    /* EN: TRACE: incoming payload from the client. */
    debugFile("INCOMING_REQUEST", {
      method: req.method,
      url: req.url,
      headers: req.headers,
      body: tryParseJson(rawBody),
    });
	
    const isChat = req.url.includes("/chat/completions");
    const path   = req.url.replace(NVIDIA_BASE, "").split("?")[0];

    /* FR: Application du patch spécifique Kimi K2.6. */
    /* EN: Apply Kimi K2.6-specific patches. */
    if (isChat && rawBody) {
      try {
        rawBody = patchBody(rawBody);
      } catch (e) {
        log.error(`patch: ${e.message}`);
      }
      const ctxTokens = getEffectiveTokens(rawBody);
      const source    = lastKnownTotal > 0 ? "nim" : "est";
      log.req(`${C.white}${req.method} ${path}${C.reset}  ctx=${C.cyan}~${(ctxTokens / 1000).toFixed(1)}k${C.reset}  ${C.dim}(${source})${C.reset}  ${ts()}`);
    } else {
      log.req(`${C.white}${req.method} ${path}${C.reset}  ${ts()}`);
    }

    const buf     = Buffer.from(rawBody);
    const headers = { ...req.headers, host: NVIDIA_HOST, "content-length": String(buf.length) };

    /* FR: Évite le gzip/br que Node devrait décoder manuellement. */
    /* EN: Avoid gzip/br to keep things simple (no manual decompression). */
    delete headers["accept-encoding"];

    const options = {
      hostname: NVIDIA_HOST,
      port: 443,
      path: req.url,
      method: req.method,
      headers,
    };
	
    /* FR: TRACE : ce qui sort vers l'upstream NVIDIA. */
    /* EN: TRACE: outgoing payload to NVIDIA upstream. */
    debugFile("OUTGOING_NIM", {
      target: `https://${NVIDIA_HOST}${req.url}`,
      method: req.method,
      headers: options.headers,
      body: tryParseJson(rawBody),
    });
	
    let estimatedTokens = 0;

    /* ═══════════ BOUCLE DE REPLAY AVEC GESTION 429 ═════════════ */
    for (let attempt = 1; attempt <= MAX_RETRIES; attempt++) {
      await acquireDepartureSlot();

      if (isChat && rawBody) {
        estimatedTokens = getEffectiveTokens(rawBody);
        if (ASSUMED_TPM > 0) await acquireTokens(estimatedTokens);
      }

      let result;
      try {
        result = await doRequest(options, buf);
      } catch (err) {
        if (isChat && estimatedTokens) releaseTokens(estimatedTokens);
        log.error(`upstream: ${err.message}  ${ts()}`);
        if (!res.headersSent) {
          res.writeHead(502);
          res.end(JSON.stringify({ error: { message: err.message } }));
        }
        return;
      }

      const { sc, headers: upHeaders, errBody, stream } = result;
      const scColor = sc < 300 ? C.green : sc < 500 ? C.yellow : C.red;

      /* FR: TRACE : réponse brute de NIM. */
      /* EN: TRACE: raw NIM response. */
      debugFile("NIM_RESPONSE", {
        status: sc,
        headers: upHeaders,
        hasErrorBody: !!errBody,
        errorBody: errBody ? tryParseJson(errBody.toString()) : undefined,
      });
	
      /* ─── CAS 1 : 429 Too Many Requests ─────────────────────── */
      if (sc === 429) {
        if (isChat && estimatedTokens) releaseTokens(estimatedTokens);
		
        if (errBody) {
          debugFile("NIM_ERROR_BODY", errBody);
        }
		
        const retryAfter     = upHeaders["retry-after"];
        const rateLimitReset = upHeaders["x-ratelimit-reset"];
        const nimHeaders     = Object.entries(upHeaders)
          .filter(([k]) => k.startsWith("x-") || k === "retry-after")
          .map(([k, v]) => `${k}=${v}`)
          .join("  ");
        log.retry(`${C.red}429${C.reset}  headers NIM : ${nimHeaders || C.dim + "aucun" + C.reset}`);

        if (attempt >= MAX_RETRIES) {
          log.error(`429 persistant après ${MAX_RETRIES} tentatives — abandon  ${ts()}`);
          if (!res.headersSent) { res.writeHead(429, upHeaders); res.end(errBody); }
          return;
        }

        const delay = backoffDelay(attempt, retryAfter, rateLimitReset);
        setGlobalCooldown(delay);
        await showCountdown(delay, attempt);
        continue;
      }

      /* ─── CAS 2 : Autre erreur HTTP (4xx/5xx) ───────────────── */
      if (sc >= 400) {
        if (isChat && estimatedTokens) releaseTokens(estimatedTokens);
        if (errBody) debugFile("NIM_ERROR_BODY", errBody);
		
        log.nim(`${scColor}${sc}${C.reset}  ${req.method} ${path}  ${ts()}`);
        if (!res.headersSent) { res.writeHead(sc, upHeaders); res.end(errBody); }
        return;
      }

      /* ─── CAS 3 : Succès 2xx ────────────────────────────────── */
      res.writeHead(sc, upHeaders);

      if (!isChat) {
        /* FR: Pas touche, on fait juste transiter le flux. */
        /* EN: Non-chat routes: transparent passthrough. */
        log.nim(`${scColor}${sc}${C.reset}  ${req.method} ${path}  ${ts()}`);
        stream.pipe(res);
        stream.on("error", () => { if (!res.writableEnded) res.end(); });
        return;
      }

      /* ─── Interception SSE pour comptabilisation réelle ─────── */
      let usageLogged = false;
      let sseBuffer   = "";

      stream.on("data", (chunk) => {
        res.write(chunk);
        if (usageLogged) return;

        sseBuffer += chunk.toString();
        const events = sseBuffer.split("\n\n");
        sseBuffer = events.pop() || "";

        for (const event of events) {
          const dataLine = event.split("\n").find((l) => l.startsWith("data:"));
          if (!dataLine) continue;
          const jsonStr = dataLine.slice(5).trim();

          if (jsonStr === "[DONE]") continue;
          try {
            const json  = JSON.parse(jsonStr);
            const usage = json.usage;
            if (!usage) continue;

            const pIn      = usage.prompt_tokens     || 0;
            const pOut     = usage.completion_tokens || 0;
            lastKnownTotal = usage.total_tokens || (pIn + pOut);

            /* FR: Log comptable et correction du budget TPM estimé. */
            /* EN: Accounting log and TPM budget correction. */
            debugFile("NIM_SSE_USAGE", json);
			
            releaseTokens(estimatedTokens);
            tpmUsed += lastKnownTotal;

            usageLogged = true;
            log.nim(
              `${scColor}${sc}${C.reset}  ${req.method} ${path}` +
              `  in=${C.cyan}${pIn}${C.reset}  out=${C.green}${pOut}${C.reset}` +
              `  total=${C.white}${lastKnownTotal}${C.reset}` +
              `  tpm=${C.dim}~${Math.round(tpmUsed / 1000)}k/${Math.round(ASSUMED_TPM / 1000)}k${C.reset}` +
              `  ${ts()}`
            );
          } catch {
            /* FR: Chunk SSE inutile pour les métriques : ignorer. */
            /* EN: SSE chunk not containing usage metrics: ignore. */
          }
        }
      });

      stream.on("end", () => {
        res.end();
        if (!usageLogged) {
          releaseTokens(estimatedTokens);
          log.nim(`${scColor}${sc}${C.reset}  ${req.method} ${path}  ${C.dim}(pas d'usage)${C.reset}  ${ts()}`);
        }
      });

      stream.on("error", (err) => {
        log.error(`stream: ${err.message}  ${ts()}`);
        releaseTokens(estimatedTokens);
        if (!res.writableEnded) res.end();
      });

      return; // FR: Succès, fin du handler. / EN: Success, handler done.
    }
  });
});

/* ═════════════════════════════════════════════════════════════════ */
/*  FR: DÉMARRAGE DU SERVEUR                                         */
/*  EN: SERVER BOOT                                                  */
/* ═════════════════════════════════════════════════════════════════ */

server.listen(PORT, "127.0.0.1", () => {
  console.log();
  console.log(`  ${C.bold}${C.cyan}kimi-proxy${C.reset}  ${C.dim}Kimi K2.6 · NVIDIA NIM (FIXED)${C.reset}`);
  console.log(`  ${C.dim}${"─".repeat(58)}${C.reset}`);
  console.log(`  ${C.green}●${C.reset} Proxy       http://localhost:${PORT}/v1`);
  console.log(`  ${C.blue}●${C.reset} NIM          https://${NVIDIA_HOST}${NVIDIA_BASE}`);
  console.log(`  ${C.magenta}●${C.reset} Pause NIM    ${INTER_REQUEST_DELAY}ms · Keep-Alive TLS (×2)`);
  console.log(`  ${C.cyan}●${C.reset} TPM budget   ~${Math.round(ASSUMED_TPM / 1000)}k tokens/min`);
  console.log(`  ${C.yellow}●${C.reset} Retry 429    cooldown ≤5min · release TPM · ${MAX_RETRIES} max`);
  console.log(`  ${C.dim}${"─".repeat(58)}${C.reset}`);
  console.log(`  ${C.dim}DEBUG=${DEBUG} PORT=${PORT} MAX_RETRIES=${MAX_RETRIES} ASSUMED_TPM=${ASSUMED_TPM} REQUEST_TIMEOUT=${REQUEST_TIMEOUT} MAX_RPM=${MAX_RPM} INTER_REQUEST_DELAY=${INTER_REQUEST_DELAY} ASSUMED_TPM=${ASSUMED_TPM}${C.reset}`);
  console.log(`  ${C.dim}bun run ./kimi-proxy.mjs${C.reset}`);
  console.log();
});

  3. github-actions commented on Jul 23, 2026

    @github-actions
    github-actionsBot
    on Jul 23, 2026 – with GitHub Actions
    Contributor

    To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.

  4. github-actions
    closed this as not plannedon Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions