You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e5ea128
Browse filesBrowse the repository at this point in the historyBrowse files
This fixes yet another bug in the prefilter. Sigh. This only occurs when
doing a stream search. The problem is that the stream handling code
assumes that if no match is found at the end of the buffer, then the
current state of the automaton is correctly updated and the buffer can
be rolled.
With most prefilters that look for a candidate *start* of a match, this
is okay. If a prefilter can't find anything, then there's nothing to
start and the current state remains in the starting state.
But if the prefilter looks for a byte that may not be at the start of
the match---like the rare byte prefilter---then we cannot assume that a
match doesn't begin near the end of the buffer searched. And in this
case, the internal implementation of search doesn't correctly hold up
it's contract because the current state won't be updated. That is, there
is an embedded assumption that if a prefilter fails then there is no
match and thus there is no need to update the current state ID. But of
course, this is just not true in a streaming context.
The right way to fix this is unfortunately to rethink how we've
implemented stream searching and make it aware of these kinds of
prefilters. I think, anyway. The other option would be to fix the lower
level search APIs to always make sure the current state ID is correct.
That would fix everything, but that seems tricky and probably requires
some delicate handling.
So for now, we just disable a prefilter entirely if it's a rare byte
prefilter and we're doing a stream search. We could build a backup
prefilter and still use that, but it feels like a gross hack. At least
now, we preserve correctness.
Kudos to @ogoffart who did the initial investigation here and came up
with a regression test, which is included in this commit. Note though,
that some tests do fail when the buffer's size is set to its minimum. So
there was a regression at some point because we aren't getting the best
test coverage. We should just bite the buffer and make the buffer size
configurable as an internal API so that tests can tweak it and provoke
more edge cases.
Fixes#64
0 commit comments