Skip to content

Repository navigation

Commit bfd28e0

Browse files
ryoppippicodesmith-bot
ryoppippi
and
codesmith-bot
authored
fix(codex): date fallback review model pricing (#1303)
* fix(codex): date fallback review model pricing Codex can emit the internal codex-auto-review label for review runs, but pricing that label as gpt-5.5 for every historical row overstates older usage. Resolve that label while parsing Codex logs, using a small release-date table from models.dev so each event falls back to the newest known Codex/OpenAI model available on the log date without extra I/O in the hot path. Remove the undated pricing alias for codex-auto-review, keep the raw alias handling for gpt-5.3-spark, and document the date-aware fallback behavior in the Codex guide. * fix(codex): clarify review fallback edge cases Address review feedback on the date-aware codex-auto-review fallback. Malformed timestamps now use the same conservative gpt-5 fallback as dates before the known release table, avoiding accidental newest-model pricing for bad data. The Codex guide now states that the parser maps codex-auto-review before pricing, and the fallback table documents its descending date-order invariant. * fix(codex): validate review fallback dates Resolve codex-auto-review model fallback dates from the raw log timestamp used for model selection instead of only the normalized event timestamp. This prevents malformed timestamp strings from being discarded and replaced by file mtime before the fallback decision. Validate YYYY-MM-DD components before comparing release dates, including month ranges, month lengths, and leap years. Invalid or missing dates now conservatively resolve to gpt-5. Add regression coverage for invalid month and non-leap-day Codex review logs. * fix(codex): generate review fallback models Replace the hand-written codex-auto-review release table in the parser with an embedded snapshot generated from the pinned models.dev input. Extend the models.dev snapshot generator to emit Codex auto-review fallback metadata alongside the existing pricing snapshot, and wire the just recipe to refresh both generated files. Keep runtime lookup fast by parsing the tiny embedded snapshot once through LazyLock, while preserving the existing date-aware fallback behavior and malformed-date conservative fallback. * fix(nix): include codex auto-review fallbacks in build source The Nix cleanSource filter only whitelists specific JSON files alongside Cargo sources. The newly added codex-auto-review-fallbacks.json embedded via include_str! by the codex parser was being filtered out, causing the Linux/macOS/Windows native package builds to fail with 'No such file or directory' at compile time. Co-authored-by: Codesmith <[email protected]> * fix(justfile): short-circuit snapshot copy on failure Replace ';' with '&&' between the nix build and two cp commands in the gen-models-dev-pricing recipe so that a failed earlier step doesn't get silently masked by a later successful cp. Co-authored-by: Codesmith <[email protected]> * fix(codex): try alternate timestamp fields when top-level timestamp is malformed The model-date resolution chain in codex_model_timestamp_from_result used raw_or_normalized_codex_timestamp, which returned any non-empty string as-is regardless of whether it parsed as a date. That short-circuited the or_else chain, so a malformed top-level `timestamp` would prevent the parser from looking at `created_at`, `created_at_camel`, nested result-field timestamps, or the file mtime fallback, and forced events through the conservative gpt-5 mapping. Validate the raw string against codex_timestamp_date before short-circuiting; fall through to normalize_codex_timestamp / normalize_value_timestamp and finally None so the rest of the chain can supply a valid date. Mirror the same change for the serde_json::Value path. Also fix the models.dev fallback generator so a gpt-5.x decimal base model is only dropped when a gpt-5.x-codex variant exists on the SAME release date. When the codex variant ships later, keeping the base entry lets events in the gap still resolve to the most recent model actually available then. Co-authored-by: Codesmith <[email protected]> --------- Co-authored-by: Codesmith <[email protected]>
1 parent e455f34 commit bfd28e0

10 files changed

Lines changed: 568 additions & 76 deletions

File tree

‎docs/guide/codex/index.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ These views support `--json`, `--compact`, `--offline`, and `--speed auto|standa
4747

4848
- **Token deltas** – Each `event_msg` with `payload.type === "token_count"` reports cumulative totals. The CLI subtracts the previous totals to recover per-turn token usage (input, cached input, output, reasoning, total).
4949
- **Per-model grouping** – The `turn_context` metadata specifies the active model. We aggregate tokens per day/month and per model. Sessions lacking model metadata (seen in early September 2025 builds) are skipped.
50-
- **Pricing** – Rates come from LiteLLM's pricing dataset via the shared `LiteLLMPricingFetcher`. Model aliases such as `gpt-5.5` are resolved through the pricing cache when available.
50+
- **Pricing** – Rates come from LiteLLM's pricing dataset via the shared `LiteLLMPricingFetcher`. Codex's internal review label is resolved to the newest known model for the log date before pricing is calculated.
5151
- **Speed pricing** – `--speed auto` is the default. It reads `config.toml` from each `CODEX_HOME` root and applies fast pricing when any Codex config has `service_tier = "priority"` or legacy `service_tier = "fast"` configured. Fast mode uses the model-specific LiteLLM multiplier when available and otherwise falls back to 2x pricing. Pass `--speed fast` or `--speed standard` to override config-based detection.
5252
- **Legacy fallback** – Early September 2025 logs that never recorded `turn_context` metadata are still included; the CLI assumes `gpt-5` for pricing so you can review the tokens even though the model tag is missing (the JSON output also marks these rows with `"isFallback": true`).
5353
- **Cost formula** – Non-cached input uses the standard input price; cached input uses the cache-read price (falling back to the input price when missing); and output tokens are billed at the output price. All prices are per million tokens. Reasoning tokens may be shown for reference, but they are part of the output charge and are not billed separately.
@@ -60,7 +60,7 @@ These views support `--json`, `--compact`, `--offline`, and `--speed auto|standa
6060
| `CODEX_HOME` | Override the root directory, or comma-separated directories, containing Codex homes or saved `codex exec --json` JSONL files |
6161
| `LOG_LEVEL` | Adjust log verbosity (0 silent … 5 trace) |
6262

63-
When Codex emits a model alias (for example `gpt-5.5`), the CLI automatically resolves it through the LiteLLM pricing data when possible. No manual override is needed.
63+
When Codex emits a model alias, the CLI automatically resolves it through the LiteLLM pricing data when possible. For `codex-auto-review`, the Codex parser maps the label to the newest known Codex/OpenAI model available on the log date using a pinned models.dev snapshot before pricing uses the resolved model name. No manual override is needed.
6464

6565
## Speed Pricing
6666

‎justfile‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,11 +54,13 @@ update-litellm-pricing:
5454
nix flake update litellm
5555
just check
5656

57-
# Regenerate the committed models.dev pricing snapshot from the pinned input
57+
# Regenerate committed models.dev snapshots from the pinned input
5858
gen-models-dev-pricing:
59-
cp "$(nix build .#models-dev-pricing --no-link --print-out-paths)" rust/crates/ccusage/src/models-dev-pricing.json
59+
snapshots="$(nix build .#models-dev-pricing --no-link --print-out-paths)" && cp "$snapshots/models-dev-pricing.json" rust/crates/ccusage/src/models-dev-pricing.json && cp "$snapshots/codex-auto-review-fallbacks.json" rust/crates/ccusage/src/adapter/codex/codex-auto-review-fallbacks.json
6060
chmod u+w rust/crates/ccusage/src/models-dev-pricing.json
61+
chmod u+w rust/crates/ccusage/src/adapter/codex/codex-auto-review-fallbacks.json
6162
nix fmt rust/crates/ccusage/src/models-dev-pricing.json
63+
nix fmt rust/crates/ccusage/src/adapter/codex/codex-auto-review-fallbacks.json
6264

6365
# Update the pinned models.dev input, regenerate its pricing snapshot, and validate
6466
update-models-dev-pricing:

‎nix/models-dev-gen.ts‎

Lines changed: 82 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,10 @@
77
* Anthropic-relevant models and the few pricing fields ccusage consumes. The
88
* embedded output is a flat map keyed by runtime model id. The output is
99
* committed to the repository and embedded at build time, so every platform
10-
* ships the identical, pinned data without any build-time network access. Run
11-
* via `just gen-models-dev-pricing` (see `nix/models-dev-pricing.nix`).
10+
* ships the identical, pinned data without any build-time network access. The
11+
* same pinned catalog also generates the Codex auto-review fallback metadata
12+
* used by the Rust parser. Run via `just gen-models-dev-pricing` (see
13+
* `nix/models-dev-pricing.nix`).
1214
*/
1315
import { generateCatalog } from './packages/core/src/generate.ts';
1416
import {
@@ -26,13 +28,22 @@ type Cost = {
2628
cache_write?: number | null;
2729
};
2830
type Model = { id?: string; cost?: Cost; limit?: { context?: number | null } };
31+
type ModelMetadata = {
32+
id?: string;
33+
release_date?: string;
34+
};
2935
type Provider = { models?: Record<string, Model> };
3036
type EmbeddedModel = {
3137
cost: Cost;
3238
limit?: { context: number };
3339
};
40+
type CodexAutoReviewFallback = {
41+
releasedOn: string;
42+
model: string;
43+
};
3444

35-
const { providers } = (await generateCatalog('.')) as {
45+
const { models, providers } = (await generateCatalog('.')) as {
46+
models: Record<string, ModelMetadata>;
3647
providers: Record<string, Provider>;
3748
};
3849

@@ -95,3 +106,71 @@ if (outfile == null || outfile.length === 0) {
95106
}
96107

97108
await Bun.write(outfile, `${JSON.stringify(sortObject(out), null, 2)}\n`);
109+
110+
const codexFallbacksOutfile = process.env.CODEX_AUTO_REVIEW_FALLBACKS_OUTFILE;
111+
if (codexFallbacksOutfile != null && codexFallbacksOutfile.length > 0) {
112+
await Bun.write(
113+
codexFallbacksOutfile,
114+
`${JSON.stringify(generateCodexAutoReviewFallbacks(models), null, 2)}\n`,
115+
);
116+
}
117+
118+
function generateCodexAutoReviewFallbacks(
119+
models: Record<string, ModelMetadata>,
120+
): CodexAutoReviewFallback[] {
121+
const entries = Object.entries(models).filter(([modelId, model]) =>
122+
isCodexAutoReviewFallbackCandidate(modelId, model),
123+
);
124+
const codexDecimalVersions = new Set(
125+
entries
126+
.map(([modelId]) => codexDecimalVersion(openAiModelName(modelId)))
127+
.filter((version): version is string => version != null),
128+
);
129+
130+
return entries
131+
.filter(([modelId, model]) => {
132+
const version = baseDecimalVersion(openAiModelName(modelId));
133+
if (version == null || !codexDecimalVersions.has(version)) {
134+
return true;
135+
}
136+
// Drop the base entry only when a `-codex` variant shipped on the same
137+
// date. If the codex variant ships later, keep the base so events in
138+
// the gap still resolve to the most recent model available then.
139+
return !entries.some(
140+
([candidateId, candidateModel]) =>
141+
codexDecimalVersion(openAiModelName(candidateId)) === version &&
142+
candidateModel.release_date === model.release_date,
143+
);
144+
})
145+
.map(([modelId, model]) => ({
146+
releasedOn: model.release_date!,
147+
model: openAiModelName(model.id ?? modelId),
148+
}))
149+
.sort((left, right) => right.releasedOn.localeCompare(left.releasedOn));
150+
}
151+
152+
function isCodexAutoReviewFallbackCandidate(modelId: string, model: ModelMetadata): boolean {
153+
if (model.release_date == null || !/^\d{4}-\d{2}-\d{2}$/.test(model.release_date)) {
154+
return false;
155+
}
156+
const modelName = openAiModelName(modelId);
157+
return (
158+
modelName === 'gpt-5' ||
159+
modelName === 'gpt-5-codex' ||
160+
/^gpt-5\.\d+$/.test(modelName) ||
161+
/^gpt-5\.\d+-codex$/.test(modelName)
162+
);
163+
}
164+
165+
function baseDecimalVersion(modelId: string): string | undefined {
166+
return /^gpt-5\.\d+$/.test(modelId) ? modelId : undefined;
167+
}
168+
169+
function codexDecimalVersion(modelId: string): string | undefined {
170+
const match = /^(gpt-5\.\d+)-codex$/.exec(modelId);
171+
return match?.[1];
172+
}
173+
174+
function openAiModelName(modelId: string): string {
175+
return modelId.startsWith('openai/') ? modelId.slice('openai/'.length) : modelId;
176+
}

‎nix/models-dev-pricing.nix‎

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,13 @@
1-
# Reproducibly regenerate the compacted models.dev pricing snapshot from the
2-
# pinned `models-dev` flake input. The upstream repository ships per-model TOML
3-
# sources rather than a prebuilt catalog, so we run their own `generateCatalog`
4-
# routine (via `nix/models-dev-gen.ts`) with Bun. Its only runtime dependencies
5-
# are `remeda` and `zod` (both zero-dependency packages), which we vendor
6-
# directly from the registry using the integrity hashes pinned in upstream
7-
# `bun.lock`.
1+
# Reproducibly regenerate compacted models.dev snapshots from the pinned
2+
# `models-dev` flake input. The upstream repository ships per-model TOML sources
3+
# rather than a prebuilt catalog, so we run their own `generateCatalog` routine
4+
# (via `nix/models-dev-gen.ts`) with Bun. Its only runtime dependencies are
5+
# `remeda` and `zod` (both zero-dependency packages), which we vendor directly
6+
# from the registry using the integrity hashes pinned in upstream `bun.lock`.
87
#
9-
# The build output is copied into the repository (see `just gen-models-dev-pricing`)
10-
# and embedded at build time, so every platform ships identical pinned data
11-
# without build-time network access.
8+
# The build outputs are copied into the repository (see
9+
# `just gen-models-dev-pricing`) and embedded at build time, so every platform
10+
# ships identical pinned data without build-time network access.
1211
{
1312
pkgs,
1413
modelsDevSrc,
@@ -26,7 +25,7 @@ let
2625
hash = "sha512-lY7CDW43ECgW9u1TcT3IoXHflywfVqDYze4waEz812jR/bZ8FHDsl7pFQoSZTz5N+2NqRXs8GBwnAwo3ZNxqhQ==";
2726
};
2827
in
29-
pkgs.runCommand "models-dev-pricing.json"
28+
pkgs.runCommand "models-dev-snapshots"
3029
{
3130
nativeBuildInputs = [ pkgs.bun ];
3231
}
@@ -52,5 +51,8 @@ pkgs.runCommand "models-dev-pricing.json"
5251
cp ${./models-dev-compact.ts} work/models-dev-compact.ts
5352
5453
cd work
55-
OUTFILE="$out" bun run gen.ts
54+
mkdir -p "$out"
55+
OUTFILE="$out/models-dev-pricing.json" \
56+
CODEX_AUTO_REVIEW_FALLBACKS_OUTFILE="$out/codex-auto-review-fallbacks.json" \
57+
bun run gen.ts
5658
''

‎nix/packages.nix‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,9 @@ in
2727
;
2828
};
2929
ccusageProgram = pkgs.lib.getExe' ccusage "ccusage";
30-
# Regeneration-only output for the committed models.dev pricing snapshot;
31-
# `just gen-models-dev-pricing` builds this and copies it into the source
32-
# tree. It is not part of the ccusage build, which embeds the committed file.
30+
# Regeneration-only output for committed models.dev snapshots;
31+
# `just gen-models-dev-pricing` builds this and copies them into the source
32+
# tree. It is not part of the ccusage build, which embeds the committed files.
3333
models-dev-pricing = pkgs.callPackage ../nix/models-dev-pricing.nix {
3434
modelsDevSrc = inputs.models-dev;
3535
};

‎package.nix‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@ let
1818
|| lib.hasSuffix "/cli-help.json" path
1919
|| lib.hasSuffix "/cli-commands.json" path
2020
|| lib.hasSuffix "/fast-multiplier-overrides.json" path
21-
|| lib.hasSuffix "/models-dev-pricing.json" path;
21+
|| lib.hasSuffix "/models-dev-pricing.json" path
22+
|| lib.hasSuffix "/codex-auto-review-fallbacks.json" path;
2223
};
2324
commonArgs = {
2425
pname = "ccusage";

‎rust/crates/ccusage/src/adapter/codex/codex-auto-review-fallbacks.json‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
[
2+
{
3+
"releasedOn": "2026-04-23",
4+
"model": "gpt-5.5"
5+
},
6+
{
7+
"releasedOn": "2026-03-05",
8+
"model": "gpt-5.4"
9+
},
10+
{
11+
"releasedOn": "2026-02-05",
12+
"model": "gpt-5.3-codex"
13+
},
14+
{
15+
"releasedOn": "2025-12-11",
16+
"model": "gpt-5.2-codex"
17+
},
18+
{
19+
"releasedOn": "2025-11-13",
20+
"model": "gpt-5.1-codex"
21+
},
22+
{
23+
"releasedOn": "2025-09-15",
24+
"model": "gpt-5-codex"
25+
},
26+
{
27+
"releasedOn": "2025-08-07",
28+
"model": "gpt-5"
29+
}
30+
]

0 commit comments

Comments
 (0)