Skip to content

Introduce duckdb-wasm and apache-arrow#2

Merged
hackerwins merged 1 commit into
mainfrom
duckdb
May 11, 2024
Merged

Introduce duckdb-wasm and apache-arrow#2
hackerwins merged 1 commit into
mainfrom
duckdb

Conversation

@hackerwins

@hackerwins hackerwins commented May 8, 2024

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it?

Experiment with DuckDB Integration and Apache Arrow for Preventing Layered Data Copy

This PR focuses on conducting an experiment to prevent layered data copy by integrating DuckDB with Apache Arrow.

Any background context you want to provide?

Initially, we attempted to upload the CVS files by loading them into memory, leading to inefficiencies in data processing as highlighted below:

+-Application-+    +--Mem--+    +-Store--+
|             |    |       |    |        |
| Spreadsheet |<-->| Grid  |<-->| DuckDB |
|             |    |       |    |        |
+-------------+    +-------+    +--------+

Results:

  • 10,000 Rows: 41.22s
  • slower performance compared to using IDB

For the second phase, we plan to upload the CSV files directly as test files, bypassing the need for loading into memory, as illustrated by:

+-Application-+    +-Arrow-+    +-Store--+
|             |    |       |    |        |
| Spreadsheet |<-->| Grid  |<-->| DuckDB |
|             |    |       |    |        |
+-------------+    +-------+    +--------+

Reference: DuckDB CSV Import Guide

INSERT INTO tbl SELECT * FROM read_csv('input.csv');

Key Findings:

  • CSV import process is highly efficient: less than 1 second
  • Conversion to cell-based tables poses challenges
  • Debugging internal operations of DuckDB is complex

What are the relevant tickets?

Fixes:

Checklist

  • Added relevant tests
  • Maintained system integrity

@hackerwins
hackerwins force-pushed the duckdb branch 2 times, most recently from 98cea9f to 0ca79ec Compare May 11, 2024 09:16
@hackerwins
hackerwins merged commit 3d05317 into main May 11, 2024
@hackerwins
hackerwins deleted the duckdb branch May 11, 2024 09:29
@hackerwins hackerwins mentioned this pull request Mar 9, 2026
5 tasks
hackerwins added a commit that referenced this pull request May 9, 2026
Marks lesson #2 as no longer latent; T12 ghost text was the
trigger that turned the dropped-field bug into a live render
miss, and the fix is one yorkieToPlain unwrap per element
branch. The new lesson cites the audit rule for future agents:
new element fields need both write (addSlide) AND read
(read()) propagation, MemSlidesStore parity tests miss it.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
hackerwins added a commit that referenced this pull request May 9, 2026
Marks every step in the todo file as completed and moves the
todo + lessons pair into docs/tasks/archive/2026/05/ so
docs/tasks/active/ shows only in-flight work. The lessons
file's #2 entry was already updated mid-PR when the
read() placeholderRef stripping turned out to be live, not
latent — that record carries forward into the archive.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
hackerwins added a commit that referenced this pull request May 24, 2026
Fills in the paired lessons file with measured WCAG contrasts, the
factory-default file-discovery surprise, and follow-up signals for
PR #2. The roadmap doc now uses fully qualified package paths to keep
verify:entropy doc-staleness checks green, and the Status table flips
PR #1 to "Ready to merge".

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
hackerwins added a commit that referenced this pull request Jul 5, 2026
Three findings from the PR review:

- knownCaches strongly retained every per-measurer cache map for the
  module lifetime (one leaked pair per disposed editor), defeating the
  WeakMaps. Add disposeMeasureCache(measurer) and call it from
  editor.dispose(); export it from the package index.

- loadingdone is unreliable on WebKit/Safari, so also settle via the
  Promise-based document.fonts.ready when fonts are still loading at mount
  (guarded on dispose). Mid-session Safari picks still rely on loadingdone.

- Remove a stale 'Still open' task-log entry for the already-completed #2.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Claude-Session: https://claude.ai/code/session_015EPywatWADMR2iYrwauVNL
harrykim8672 added a commit to harrykim8672/wafflebase that referenced this pull request Jul 21, 2026
mark-ready gate wafflebase#1 parsed the <!-- harness-verification --> PR comment, which
the author agent could post itself (issues:write). It now reads the authoritative
CI workflow-run conclusion for the head SHA via the Actions API (the author
cannot forge a CI run), matching gate wafflebase#2's trust model. Added actions:read to the
promote job.

Also documented that agent-independent-review must never be sufficient-for-
merge on its own — human CODEOWNER approval stays required (LLM-injection residual
backstop).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant