How to speedup virtual tables? #465
Description
Hi Roger!
I'm not sure if this is a function of SQLite or the C-wrapper but I'm finding the virtual table wrapper very slow.
I'm trying to run it over a table that may have ~700k rows and it's taking a very very long time.
I've boiled down the sample to:
import apsw
import apsw.ext
# 3 lines of code ...
def table_range(start=1, stop=700_000, step=1):
for i in range(start, stop + 1, step):
yield (i,)
# set column names
table_range.columns = ("value",)
# set how to access what table_range returns
table_range.column_access = apsw.ext.VTColumnAccess.By_Index
connection = apsw.Connection(":memory:")
# register it
apsw.ext.make_virtual_module(connection, "range", table_range)
# see it work. we can provide both positional and keyword
# arguments
query = "SELECT * FROM range(90) WHERE step=2"
for row in connection.execute(query):
# don't pay any cost for printing
pass
print("done")
I'm not very proficient at profiling both C-extensions and Python -- cProfile doesn't have anything really to help here.