Fetches all rows by default on query, like libpq.#117
Fetches all rows by default on query, like libpq.#117RangelReale wants to merge 4 commits intolib:masterfrom
Conversation
…ple queries to be active simultaneously * Provides a singlerowmode=true parameter, like libpq (to keep working like today)
conn.go
Outdated
There was a problem hiding this comment.
Why not just [][]driver.Value here?
There was a problem hiding this comment.
Aren't slices costly to append? As I can't know beforehand how many records there will be.
There was a problem hiding this comment.
append behaviour is fairly reasonable, it doesn't just grow the slice by one each time you append. list.List also uses reflection, so there's a performance hit from that.
PASS
BenchmarkSlice1000 10000 125335 ns/op
BenchmarkSlice10000 1000 2363422 ns/op
BenchmarkSlice100000 50 28006284 ns/op
BenchmarkSlice1000000 5 273778224 ns/op
BenchmarkSlice10000000 1 3812814581 ns/op
BenchmarkList1000 5000 468516 ns/op
BenchmarkList10000 500 5936557 ns/op
BenchmarkList100000 20 70525959 ns/op
BenchmarkList1000000 5 737343756 ns/op
BenchmarkList10000000 1 8153616298 ns/op
ok bench 31.456s
There was a problem hiding this comment.
Good to know, I updated my fork.
(conflicts with multiple simultaneous queries) This reverts commit 31326ad. Conflicts: conn.go
Isn't this the wrong way round? Inverting the condition seems to fix the problem with regression tests failing. Also, is reverting 31326ad really necessary? As far as I can tell, you never have two queries actually running on the protocol level, so I don't see how it would cause any problems. |
|
Hmm I made a pull request using my master branch, now GitHub got my latest changes which weren't meant to be here. The revert needed to be done because the scrath buffer was getting overriten when 2 queries were done at the same time, parts of the resultset for one query were appearing on the other one. |
|
Oh, I think I now see how that could happen. I think instead of making everyone pay the performance penalty from reverting that commit, you should only impose it on people using the singlerowmode by copying the data before releasing the connection back into the pool. Also, is there a specific reason you chose not to implement the sql.Rows interface for rowscomplete? |
Oh, looks like you're not even supposed to; database/sql should take care of that. Scratch that part. |
|
@RangelReale any chance you could update this PR? The current code changes the behavior to fetch-all from the current implicit 'single-row mode'. If you also inverted the option so the current behaviour was maintained by default I think you'd address the concerns raised in #81 and stand a better chance of getting this merged. |
pq's implementation of the PostgreSQL query protocol is fairly straight
forward and doesn't support running more than one query at a time.
This means that something like this:
tx, _ := db.Begin()
rows, _ := tx.Query(..)
for rows.Next() {
tx.Query(..) // Sends query while we're reading the first.
}
Will fail, as it sends a query while we're still reading the first
query.
We could use the [extended query protocol], which does allow running
more than one query without waiting for the previous one to complete.
It's an option, I guess, but reading the docs it also seems rather more
complex – I kind of like how straight-forward pq's implementation is
right now and anyone can step through it without much understanding of
PostgreSQL.
The "classic" solution for this is to just immediately read all rows in
memory, which is what many PostgreSQL drivers do. I guess we could add
an option for that (and there's a PR for it at #117). But idk, seems
easy enough to do manually? And we can't do it by default, since people
have written code against pq with the assumption that it won't read all
data in memory: so it will still catch people not aware of this.
So anyway, just change the error message to be clearer. It hasn't come
up that often in the last ~13 years so it's probably "good enough", and
it's the simplest way to improve this.
There was already inCopy to prevent other queries being run while COPY
was in progress, so expand and re-use that. We reset it when we get
ReadyForQuery from the server.
Note we can still do any of the above (extended query protocol, read all
in memory) later: this is not preventing that.
[extended query protocol]: https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY
Fixes #81
Fixes #117
Probably fixes #855
Probably fixes #894
pq's implementation of the PostgreSQL query protocol is fairly straight
forward and doesn't support running more than one query at a time.
This means that something like this:
tx, _ := db.Begin()
rows, _ := tx.Query(..)
for rows.Next() {
tx.Query(..) // Sends query while we're reading the first.
}
Will fail, as it sends a query while we're still reading the first
query.
We could use the [extended query protocol], which does allow running
more than one query without waiting for the previous one to complete.
It's an option, I guess, but reading the docs it also seems rather more
complex – I kind of like how straight-forward pq's implementation is
right now and anyone can step through it without much understanding of
PostgreSQL.
The "classic" solution for this is to just immediately read all rows in
memory, which is what many PostgreSQL drivers do. I guess we could add
an option for that (and there's a PR for it at #117). But idk, seems
easy enough to do manually? And we can't do it by default, since people
have written code against pq with the assumption that it won't read all
data in memory: so it will still catch people not aware of this.
So anyway, just change the error message to be clearer. It hasn't come
up that often in the last ~13 years so it's probably "good enough", and
it's the simplest way to improve this.
There was already inCopy to prevent other queries being run while COPY
was in progress, so expand and re-use that. We reset it when we get
ReadyForQuery from the server. Not sure if atomic.Bool is needed or just
bool will be fine? I couldn't trigger a race in my testing, but err is
protected with syncErr, and it doesn't hurt to be safe.
Note we can still do any of the above (extended query protocol, read all
in memory) later: this is not preventing that.
[extended query protocol]: https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY
Fixes #81
Fixes #117
Probably fixes #855
Probably fixes #894
pq's implementation of the PostgreSQL query protocol is fairly straight
forward and doesn't support running more than one query at a time.
This means that something like this:
tx, _ := db.Begin()
rows, _ := tx.Query(..)
for rows.Next() {
tx.Query(..) // Sends query while we're reading the first.
}
Will fail, as it sends a query while we're still reading the first
query.
We could use the [extended query protocol], which does allow running
more than one query without waiting for the previous one to complete.
It's an option, I guess, but reading the docs it also seems rather more
complex – I kind of like how straight-forward pq's implementation is
right now and anyone can step through it without much understanding of
PostgreSQL.
The "classic" solution for this is to just immediately read all rows in
memory, which is what many PostgreSQL drivers do. I guess we could add
an option for that (and there's a PR for it at #117). But idk, seems
easy enough to do manually? And we can't do it by default, since people
have written code against pq with the assumption that it won't read all
data in memory: so it will still catch people not aware of this.
So anyway, just change the error message to be clearer. It hasn't come
up that often in the last ~13 years so it's probably "good enough", and
it's the simplest way to improve this.
There was already inCopy to prevent other queries being run while COPY
was in progress, so expand and re-use that. We reset it when we get
ReadyForQuery from the server. Not sure if atomic.Bool is needed or just
bool will be fine? I couldn't trigger a race in my testing, but err is
protected with syncErr, and it doesn't hurt to be safe.
Note we can still do any of the above (extended query protocol, read all
in memory) later: this is not preventing that.
[extended query protocol]: https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY
Fixes #81
Fixes #117
Probably fixes #855
Probably fixes #894
pq's implementation of the PostgreSQL query protocol is fairly straight
forward and doesn't support running more than one query at a time.
This means that something like this:
tx, _ := db.Begin()
rows, _ := tx.Query(..)
for rows.Next() {
tx.Query(..) // Sends query while we're reading the first.
}
Will fail, as it sends a query while we're still reading the first
query.
We could use the [extended query protocol], which does allow running
more than one query without waiting for the previous one to complete.
It's an option, I guess, but reading the docs it also seems rather more
complex – I kind of like how straight-forward pq's implementation is
right now and anyone can step through it without much understanding of
PostgreSQL.
The "classic" solution for this is to just immediately read all rows in
memory, which is what many PostgreSQL drivers do. I guess we could add
an option for that (and there's a PR for it at #117). But idk, seems
easy enough to do manually? And we can't do it by default, since people
have written code against pq with the assumption that it won't read all
data in memory: so it will still catch people not aware of this.
So anyway, just change the error message to be clearer. It hasn't come
up that often in the last ~13 years so it's probably "good enough", and
it's the simplest way to improve this.
There was already inCopy to prevent other queries being run while COPY
was in progress, so expand and re-use that. We reset it when we get
ReadyForQuery from the server. Not sure if atomic.Bool is needed or just
bool will be fine? I couldn't trigger a race in my testing, but err is
protected with syncErr, and it doesn't hurt to be safe.
Note we can still do any of the above (extended query protocol, read all
in memory) later: this is not preventing that.
[extended query protocol]: https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY
Fixes #81
Fixes #117
Probably fixes #855
Probably fixes #894
This allows multiple queries to be active simultaneously
Provides a singlerowmode=true parameter, like libpq (to keep working like today)
fixes #81