gh-137855: Improve import time of `sqlite3` #131796

Wulian233 · 2025-03-27T12:17:20Z

Improve import time of sqlite3 by 1.5x faster

Benchmark on Windows10, CPython 3.14.0b1

D:\Python314>hyperfine -i --warmup 8 "./python -c 'from sqlite3 import *'" "./python -c 'from sqlite3_new import *'
Benchmark 1: ./python -c 'from sqlite3 import *'
  Time (mean ± σ):     176.8 µs ± 179.3 µs    [User: 1985.8 µs, System: 2886.4 µs]
  Range (min … max):     0.0 µs … 1018.1 µs    406 runs

Benchmark 2: ./python -c 'from sqlite3_new import *'
  Time (mean ± σ):     245.4 µs ± 200.7 µs    [User: 914.9 µs, System: 2002.8 µs]
  Range (min … max):     0.0 µs … 1787.5 µs    401 runs

Summary
  ./python -c 'from sqlite3 import *' ran
    1.39 ± 1.81 times faster than ./python -c 'from sqlite3_new import *'

I just realized today that I accidentally deleted an unmerged branch last month, which resulted in closing #129118 . My apologies for the oversight. I've now recreated the pull request - the content remains exactly the same as before

Module	sqlite3 self(us)	sqlite3 cumulative(us)	sqlite3_new self(us)	sqlite3_new cumulative(us)
`_sqlite3`	2658	6641	2455	6462
`sqlite3.dbapi2`	6431	31714	5607	28860
`sqlite3` (main)	8592	40306	4213	35121

The time taken by _sqlite3 is not much different.
The time for sqlite3.dbapi2 decreased significantly from 6431 to 5607
The time for the main sqlite3 module dropped noticeably from 8592 to 4213

Issue: Improve import time of various stdlib modules #137855

eendebakpt · 2025-03-27T12:55:02Z

@Wulian233 In your benchmark the sqlite3_new is slower (takes more time to import). Based on the name I would expect it to be the faster one.

Also time is a builtin module. Is that one really a bottleneck?

Lib/sqlite3/dbapi2.py

brianschubert · 2025-03-27T13:07:13Z

I'm also a bit surprised that time would be a bottleneck. When I run ./python -X importtime -c 'import sqlite3' on my system, I get

import time: self [us] | cumulative | imported package
[...]
import time:        80 |         80 |   time
[...]
import time:      2011 |       8096 | sqlite3

Which, if I'm reading it right, says that importing time can only account for only a small fraction sqlite3's import time?

Wulian233 · 2025-03-27T13:13:04Z

@Wulian233 In your benchmark the sqlite3_new is slower (takes more time to import). Based on the name I would expect it to be the faster one.

Also time is a builtin module. Is that one really a bottleneck?

I think I got the name wrong at the time, it was a copy of the previous PR content, and here is the test I just run

D:\Python313>hyperfine -i --warmup 8 "./python -c 'from sqlite3 import *'" "./python -c 'from sqlite3_new import *'"
Benchmark 1: ./python -c 'from sqlite3 import *'
  Time (mean ± σ):     697.1 µs ± 1059.7 µs    [User: 2577.5 µs, System: 3063.5 µs]
  Range (min … max):     0.0 µs … 10960.7 µs    391 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
  Warning: Ignoring non-zero exit code.
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: ./python -c 'from sqlite3_new import *'
  Time (mean ± σ):     635.0 µs ± 755.4 µs    [User: 1821.0 µs, System: 3691.3 µs]
  Range (min … max):     0.0 µs … 3622.5 µs    399 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
  Warning: Ignoring non-zero exit code.

Summary
  ./python -c 'from sqlite3_new import *' ran
    1.10 ± 2.12 times faster than ./python -c 'from sqlite3 import *'

StanFromIreland · 2025-03-27T14:49:37Z

Lib/sqlite3/dbapi2.py

 # 3. This notice may not be removed or altered from any source distribution.

-import datetime
 import time


What's the point of the PR if the import is not removed? Lazy imports are therefore not used, then why should any speedup be expected, a slowdown is more likely?

Already pointed out in #131796 (comment) :)

(But the PR author self-reviewed the review comment by declaring the review comment to be resolved, which is a controversial feature for github to even allow.)

picnixz · 2025-03-29T22:36:51Z

To be precise, it's not importing sqlite3 that is faster, it's importing sqlite3 and loading its content in the global namespace (the benchmarks are done for from sqlite3 import * not for a plain import sqlite3 statement).

Also, the benchmarks are quite unstable. The standard deviation is much higher than the mean itself! In practice, I think we don't gain anything else than stability (the stdev becomes smaller with the new implementation but again this could be noise).

Can you check if import sqlite3 has less noise?

And to be precise, we're not gaining a 2x speed-up. On average, we're only gaining a 1.1x speed-up and the standard deviation of this speed-up is $\pm2$, which is again not really indicative =/

And yes, I'm also surprised that we're gaining "that much" when we just make import time local. Considering it's a built-in module (though not sure if it's present at startup), I don't think we need to do this change (we're maybe gaining a small speed-up but maybe not much; and sqlite3 is already is a heavy module to import). So benchmarks using -X importtime here are much more important IMO compared to interpreter's startup as well.

erlend-aasland · 2025-06-05T14:41:15Z

@Wulian233, can you address @picnixz's last remark?

Wulian233 · 2025-06-06T02:29:53Z

I've reworked another version and it's going to be faster now

I ran the command D:\Python314>python -X importtime -c "import sqlite3" and obtained the following output. Below is a summary table comparing sqlite3 and sqlite3_new import times (GPT helped):

Module	sqlite3 self(us)	sqlite3 cumulative(us)	sqlite3_new self(us)	sqlite3_new cumulative(us)
`_sqlite3`	2658	6641	2455	6462
`sqlite3.dbapi2`	6431	31714	5607	28860
`sqlite3` (main)	8592	40306	4213	35121

_sqlite3 is not much different.
sqlite3.dbapi2 decreased from 6431 -> 5607
sqlite3 main dropped very noticeably from 8592 -> 4213

Raw output:

Details

import time: self [us] | cumulative | imported package import time: 296 | 296 | winreg import time: 248 | 248 | _io import time: 70 | 70 | marshal import time: 312 | 312 | nt import time: 1637 | 2266 | _frozen_importlib_external import time: 1103 | 1103 | time import time: 1089 | 2191 | zipimport import time: 75 | 75 | _codecs import time: 793 | 868 | codecs import time: 1926 | 1926 | encodings.aliases import time: 797 | 797 | encodings._win_cp_codecs import time: 3400 | 6989 | encodings import time: 960 | 960 | encodings.utf_8 import time: 75 | 75 | _codecs_cn import time: 126 | 126 | _multibytecodec import time: 1389 | 1590 | encodings.gbk import time: 86 | 86 | _signal import time: 59 | 59 | _abc import time: 323 | 382 | abc import time: 104 | 104 | _stat import time: 527 | 630 | stat import time: 1745 | 1745 | _collections_abc import time: 126 | 126 | genericpath import time: 229 | 229 | _winapi import time: 3109 | 3463 | ntpath import time: 2543 | 8760 | os import time: 584 | 584 | _sitebuiltins import time: 1004 | 1004 | sitecustomize import time: 563 | 563 | usercustomize import time: 2043 | 12953 | site import time: 1104 | 1104 | linecache import time: 787 | 787 | _datetime import time: 1003 | 1790 | datetime import time: 704 | 704 | itertools import time: 1147 | 1147 | keyword import time: 128 | 128 | _operator import time: 1412 | 1540 | operator import time: 2293 | 2293 | reprlib import time: 164 | 164 | _collections import time: 4104 | 9950 | collections import time: 2917 | 12866 | collections.abc import time: 64 | 64 | _types import time: 1134 | 1197 | types import time: 136 | 136 | _functools import time: 2652 | 3984 | functools import time: 2658 | 6641 | _sqlite3 import time: 74 | 74 | _contextvars import time: 1406 | 1479 | _py_warnings import time: 2508 | 3987 | warnings import time: 6431 | 31714 | sqlite3.dbapi2 import time: 8592 | 40306 | sqlite3

D:\Python314>python -X importtime -c "import sqlite3_new"
import time: self [us] | cumulative | imported package
import time: 142 | 142 | winreg
import time: 284 | 284 | _io
import time: 119 | 119 | marshal
import time: 461 | 461 | nt
import time: 2051 | 2914 | _frozen_importlib_external
import time: 849 | 849 | time
import time: 651 | 1500 | zipimport
import time: 131 | 131 | _codecs
import time: 4036 | 4166 | codecs
import time: 1493 | 1493 | encodings.aliases
import time: 1159 | 1159 | encodings._win_cp_codecs
import time: 6495 | 13311 | encodings
import time: 1028 | 1028 | encodings.utf_8
import time: 72 | 72 | _codecs_cn
import time: 124 | 124 | _multibytecodec
import time: 1305 | 1500 | encodings.gbk
import time: 119 | 119 | _signal
import time: 58 | 58 | _abc
import time: 366 | 424 | abc
import time: 100 | 100 | _stat
import time: 2987 | 3086 | stat
import time: 1215 | 1215 | _collections_abc
import time: 135 | 135 | genericpath
import time: 238 | 238 | _winapi
import time: 673 | 1045 | ntpath
import time: 2186 | 7956 | os
import time: 1146 | 1146 | _sitebuiltins
import time: 1021 | 1021 | sitecustomize
import time: 545 | 545 | usercustomize
import time: 1849 | 12515 | site
import time: 1417 | 1417 | linecache
import time: 163 | 163 | _datetime
import time: 809 | 972 | datetime
import time: 570 | 570 | itertools
import time: 1219 | 1219 | keyword
import time: 154 | 154 | _operator
import time: 1384 | 1538 | operator
import time: 1604 | 1604 | reprlib
import time: 151 | 151 | _collections
import time: 3696 | 8776 | collections
import time: 2846 | 11621 | collections.abc
import time: 68 | 68 | _types
import time: 1052 | 1119 | types
import time: 174 | 174 | _functools
import time: 2716 | 4007 | functools
import time: 2455 | 6462 | _sqlite3
import time: 77 | 77 | _contextvars
import time: 2803 | 2879 | _py_warnings
import time: 1321 | 4200 | warnings
import time: 5607 | 28860 | sqlite3.dbapi2
import time: 1298 | 30158 | sqlite3
import time: 751 | 30908 | sqlite3.dbapi2
import time: 4213 | 35121 | sqlite3_new

hyperfine test:

D:\Python314>hyperfine -i --warmup 8 "./python -c 'from sqlite3 import *'" "./python -c 'from sqlite3_new import *'
Benchmark 1: ./python -c 'from sqlite3 import *'
  Time (mean ± σ):     176.8 µs ± 179.3 µs    [User: 1985.8 µs, System: 2886.4 µs]
  Range (min … max):     0.0 µs … 1018.1 µs    406 runs

Benchmark 2: ./python -c 'from sqlite3_new import *'
  Time (mean ± σ):     245.4 µs ± 200.7 µs    [User: 914.9 µs, System: 2002.8 µs]
  Range (min … max):     0.0 µs … 1787.5 µs    401 runs

Summary
  ./python -c 'from sqlite3 import *' ran
    1.39 ± 1.81 times faster than ./python -c 'from sqlite3_new import *'

eli-schwartz · 2025-06-06T03:30:35Z

Lib/sqlite3/__main__.py

 def main(*args):
+    from argparse import ArgumentParser
+    from textwrap import dedent
+


I do not see a valid reason to do code churn and delay imports for the sake of speeding up import sqlite3.__main__ while assuming that sqlite3.__main__.main() is never called.

I presume the following: execute is used by the REPL in addition to CLI; user code could import to use SQLiteInteractiveConsole. Whether worth it, not my call.

The REPL is of course part of calling main(), which makes my point quite well. :)

User code certainly could import SQLiteInteractiveConsole, as it lies within their technical capability. I think any user code importing an interactive console from a module named __main__ ought to be prepared to use it without worrying about the startup time of importing argparse and textwrap. A module called __main__ feels rather special to me in that sense -- one ought not casually resort to importing it.

I agree with @eli-schwartz. This change looks meaningless.

Lib/sqlite3/__main__.py

serhiy-storchaka · 2025-08-17T06:29:34Z

I agree with @picnixz and @eli-schwartz. The effect of moving import time looks not more than a noise. And it does not make much sense to optimize "import" time of sqlite3.__main__ -- it does not affect the total time.

ok

4b04bdf

Wulian233 requested review from berkerpeksag and erlend-aasland as code owners March 27, 2025 12:17

bedevere-app bot added the awaiting review label Mar 27, 2025

bedevere-app bot mentioned this pull request Mar 27, 2025

Improve import time of various stdlib modules #118761

Closed

Wulian233 mentioned this pull request Mar 27, 2025

gh-118761: Improve import time of sqlite3 #129118

Closed

brianschubert reviewed Mar 27, 2025

View reviewed changes

Lib/sqlite3/dbapi2.py Show resolved Hide resolved

StanFromIreland reviewed Mar 27, 2025

View reviewed changes

ZeroIntensity added the topic-sqlite3 label Apr 21, 2025

erlend-aasland added the pending The issue will be closed if no feedback is provided label Jun 5, 2025

Improve the import time of sqlite3 by 1.5x faster.

d3cf658

Wulian233 added 4 commits June 6, 2025 10:33

Merge branch 'main' into sqlite

4373d47

fix & finish

def1974

Update dbapi2.py

04eedab

fix

ccf3575

Wulian233 mentioned this pull request Jun 6, 2025

gh-134036: Improve error messages for invalid raise statements #134077

Merged

eli-schwartz suggested changes Jun 6, 2025

View reviewed changes

bedevere-app bot added awaiting core review and removed awaiting review labels Jun 6, 2025

ofek mentioned this pull request Aug 16, 2025

Improve import time of various stdlib modules #137855

Open

hugovk changed the title ~~gh-118761: Improve import time of sqlite3~~ gh-137855: Improve import time of sqlite3 Aug 16, 2025

Wulian233 added 2 commits August 17, 2025 08:43

Merge branch 'python:main' into sqlite

4c138d8

Rename pythongh-137855

f434be3

Wulian233 closed this Aug 17, 2025

Wulian233 deleted the sqlite branch August 17, 2025 06:34

Uh oh!

gh-137855: Improve import time of sqlite3 #131796

gh-137855: Improve import time of sqlite3 #131796

Uh oh!

Conversation

Wulian233 commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eendebakpt commented Mar 27, 2025

Uh oh!

Uh oh!

brianschubert commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Wulian233 commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StanFromIreland Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

eli-schwartz Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

picnixz commented Mar 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erlend-aasland commented Jun 5, 2025

Uh oh!

Wulian233 commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eli-schwartz Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

terryjreedy Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

eli-schwartz Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

serhiy-storchaka commented Aug 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

gh-137855: Improve import time of `sqlite3` #131796

gh-137855: Improve import time of `sqlite3` #131796

Wulian233 commented Mar 27, 2025 •

edited

Loading

brianschubert commented Mar 27, 2025 •

edited

Loading

Wulian233 commented Mar 27, 2025 •

edited

Loading

picnixz commented Mar 29, 2025 •

edited

Loading

Wulian233 commented Jun 6, 2025 •

edited

Loading