Skip to content

Commit 9515689

Browse files
winding-linesclaude
andcommitted
Link the upstream issues from the correctness section
The correctness section claimed this work "surfaced upstream bugs in PyIceberg, pyarrow and iceberg-rust" without saying which, or where. Each disagreement has now been reduced to a standalone reproducer with no magmalake code in it. Two turned out to be unreported and were filed: - apache/arrow#51097 — the Parquet writer undercounts nulls for a fixed-width leaf under list<struct> when some lists are null or empty, while the BYTE_ARRAY leaf beside it is correct. - apache/iceberg-rust#3118 — an In predicate whose literals straddle a file's bounds is never pruned, because each bound is tested against the whole literal set rather than narrowing it. Three were already open upstream and are linked rather than duplicated: apache/iceberg-python#3690, #3620 and #3833. Two earlier candidates are dropped from the list. pyarrow reporting LZ4_RAW as "LZ4" and deriving no ConvertedType for a non-UTC timestamp are both deliberate API behaviour, not defects. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_015or6oTZmb7EQTXTbALbBXY
1 parent 3970e83 commit 9515689

2 files changed

Lines changed: 105 additions & 16 deletions

File tree

‎src/data/upstream.ts‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
export type UpstreamStatus = "filed" | "confirmed";
2+
3+
export interface UpstreamIssue {
4+
/** The upstream project the issue lives in. */
5+
project: string;
6+
/** Issue reference as it should read, e.g. `apache/arrow#51097`. */
7+
ref: string;
8+
url: string;
9+
/** One line: what is wrong. */
10+
bug: string;
11+
/**
12+
* `filed` — magmalake reported it.
13+
* `confirmed` — it was already open upstream; magmalake reproduced it independently.
14+
*/
15+
status: UpstreamStatus;
16+
}
17+
18+
export const STATUS_LABEL: Record<UpstreamStatus, string> = {
19+
filed: "reported",
20+
confirmed: "reproduced",
21+
};
22+
23+
/**
24+
* Disagreements between magmalake and its oracles that turned out to be the
25+
* oracle's fault. Every one was reduced to a standalone script or unit test
26+
* with no magmalake code in it before it was reported or linked here.
27+
*/
28+
export const upstreamIssues: UpstreamIssue[] = [
29+
{
30+
project: "Apache Arrow",
31+
ref: "apache/arrow#51097",
32+
url: "https://github.com/apache/arrow/issues/51097",
33+
bug: "The Parquet writer undercounts nulls for a fixed-width leaf under list<struct> when some lists are null or empty. The BYTE_ARRAY leaf beside it in the same struct is right, so two leaves disagree about the same level records.",
34+
status: "filed",
35+
},
36+
{
37+
project: "iceberg-rust",
38+
ref: "apache/iceberg-rust#3118",
39+
url: "https://github.com/apache/iceberg-rust/issues/3118",
40+
bug: "An In predicate whose literals straddle a file's bounds is never pruned: the lower and upper bounds are each tested against the whole literal set instead of narrowing it, as Java and PyIceberg do.",
41+
status: "filed",
42+
},
43+
{
44+
project: "PyIceberg",
45+
ref: "apache/iceberg-python#3690",
46+
url: "https://github.com/apache/iceberg-python/pull/3690",
47+
bug: "v3 manifests and manifest lists are read with the v2 projection, so first_row_id, referenced_data_file, content_offset and content_size_in_bytes are dropped on the way in.",
48+
status: "confirmed",
49+
},
50+
{
51+
project: "PyIceberg",
52+
ref: "apache/iceberg-python#3620",
53+
url: "https://github.com/apache/iceberg-python/issues/3620",
54+
bug: "The mirror image on the write path: the manifest writer builds its record schema at DEFAULT_READ_VERSION, so those same v3 fields would serialise as null.",
55+
status: "confirmed",
56+
},
57+
{
58+
project: "PyIceberg",
59+
ref: "apache/iceberg-python#3833",
60+
url: "https://github.com/apache/iceberg-python/issues/3833",
61+
bug: "A null list<struct> or map<K, struct> is rebuilt as an empty one — on the write path too, so the null is gone at rest and no reader can recover it.",
62+
status: "confirmed",
63+
},
64+
];

‎src/pages/index.astro‎

Lines changed: 41 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import TinCard from "../components/TinCard.astro";
66
import PerfTable from "../components/PerfTable.astro";
77
import { tins, tinsByTier, TIER_LABEL } from "../data/tins";
88
import { perfGroups, passes } from "../data/perf";
9+
import { upstreamIssues, STATUS_LABEL } from "../data/upstream";
910
1011
const description =
1112
"An exploration of running data workflows on Mojo: a native Apache Iceberg stack — Parquet, Avro, Thrift, codecs, Roaring bitmaps, object storage and threads — published as tins on mojoshelf and gated against the reference implementation of every format.";
@@ -27,17 +28,6 @@ const notYet = [
2728
"Predicates inside list and map elements",
2829
];
2930
30-
const upstream = [
31-
{
32-
project: "PyIceberg",
33-
bug: "v3 manifest schemas mishandled on both the read and the write path",
34-
},
35-
{
36-
project: "pyarrow",
37-
bug: "statistics under list<struct>, and codec and logical-type reporting",
38-
},
39-
{ project: "iceberg-rust", bug: "metrics pruning" },
40-
];
4131
---
4232

4333
<Base title="Data lake building blocks in Mojo" description={description} bare={false}>
@@ -265,13 +255,21 @@ const upstream = [
265255
<h3 class="upstream__title">Building it found bugs upstream</h3>
266256
<p class="upstream__blurb">
267257
Holding three implementations against each other surfaces disagreements, and not all
268-
of them were magmalake's fault. Each is documented in the repository that found it.
258+
of them were magmalake's fault. Each one below was cut down to a standalone script
259+
or unit test with no magmalake code in it, then either reported upstream or — where
260+
someone had got there first — matched to the report that already existed.
269261
</p>
270262
<ul class="upstream__list">
271263
{
272-
upstream.map((u) => (
264+
upstreamIssues.map((u) => (
273265
<li>
274-
<span class="upstream__project">{u.project}</span>
266+
<a class="upstream__head" href={u.url} rel="noopener">
267+
<span class="upstream__project">{u.project}</span>
268+
<span class="upstream__ref">{u.ref}</span>
269+
<span class:list={["upstream__tag", `upstream__tag--${u.status}`]}>
270+
{STATUS_LABEL[u.status]}
271+
</span>
272+
</a>
275273
<span class="upstream__bug">{u.bug}</span>
276274
</li>
277275
))
@@ -542,13 +540,40 @@ const upstream = [
542540
border-left: 2px solid color-mix(in srgb, var(--ember) 45%, transparent);
543541
padding-left: 0.85rem;
544542
}
543+
.upstream__head {
544+
display: flex;
545+
align-items: baseline;
546+
flex-wrap: wrap;
547+
gap: 0.4rem 0.55rem;
548+
margin-bottom: 0.3rem;
549+
text-decoration: none;
550+
}
551+
.upstream__head:hover .upstream__ref { text-decoration: underline; }
545552
.upstream__project {
546-
display: block;
547553
font-family: var(--font-mono);
548554
font-size: 0.8125rem;
549555
font-weight: 600;
550556
color: var(--ember);
551-
margin-bottom: 0.15rem;
557+
}
558+
.upstream__ref {
559+
font-family: var(--font-mono);
560+
font-size: 0.75rem;
561+
color: var(--text-muted);
562+
}
563+
.upstream__tag {
564+
font-family: var(--font-mono);
565+
font-size: 0.625rem;
566+
letter-spacing: 0.08em;
567+
text-transform: uppercase;
568+
padding: 0.1rem 0.4rem;
569+
border-radius: 999px;
570+
border: 1px solid var(--border-strong);
571+
color: var(--text-faint);
572+
white-space: nowrap;
573+
}
574+
.upstream__tag--filed {
575+
color: var(--ember);
576+
border-color: color-mix(in srgb, var(--ember) 45%, transparent);
552577
}
553578
.upstream__bug {
554579
display: block;

0 commit comments

Comments
 (0)