"sub_it != real_added.end()" guarantee failure (changefeeds) #5561
Description
Crash on RethinkDB 2.2.3:
2016-03-23T12:28:03.253003250 184484.113044s error: Error in src/rdb_protocol/changefeed.cc at line 1124:
2016-03-23T12:28:03.253059027 184484.113099s error: Guarantee failed: [sub_it != real_added.end()]
2016-03-23T12:28:03.253070696 184484.113111s error: Backtrace:
2016-03-23T12:28:03.480228158 184484.340270s error: Wed Mar 23 12:28:03 2016
1 [0xad938a]: backtrace_t::backtrace_t() at ??:?
2 [0xad986a]: format_backtrace[abi:cxx11](bool) at ??:?
3 [0xd65766]: report_fatal_error(char const*, int, char const*, ...) at ??:?
4 [0x8dd2fe]: ql::changefeed::limit_manager_t::commit(rwlock_in_line_t*, boost::variant<ql::changefeed::primary_ref_t, ql::changefeed::sindex_ref_t> const&) at ??:?
5 [0x7f7a39]: /usr/bin/rethinkdb() [0x7f7a39] at 0x7f7a39 ()
6 [0x8d8726]: ql::changefeed::server_t::foreach_limit(boost::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, store_key_t const*, std::function<void (rwlock_in_line_t*, rwlock_in_line_t*, rwlock_in_line_t*, ql::changefeed::limit_manager_t*)>, auto_drainer_t::lock_t const&) at ??:?
7 [0x7f8daa]: rdb_update_single_sindex(store_t*, store_t::sindex_access_t const*, deletion_context_t const*,
rdb_modification_report_t const*, unsigned long*, auto_drainer_t::lock_t, cond_t*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*) at ??:?
8 [0x800388]: callable_action_instance_t<std::_Bind<void (*(store_t*, store_t::sindex_access_t*, deletion_context_t const*, rdb_modification_report_t const*, unsigned long*, auto_drainer_t::lock_t, cond_t*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*))(store_t*, store_t::sindex_access_t const*, deletion_context_t const*, rdb_modification_report_t const*, unsigned long*, auto_drainer_t::lock_t, cond_t*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*)> >::run_action() at ??:?
9 [0x9e6802]: coro_t::run() at ??:?
This was reported by @tatsujin1 . He only ran into this once.
The only other data that we have right now:
"I had a bunch of processes with changefeeds (some of them identical) and only ~5 writes, in total."
@tatsujin1 What types of orderby.limit changefeeds are used in your application? Does any of them use a multi secondary index?