Skip to content

Commit d9b42d5

Browse files
antiguruclaude
andcommitted
adapter: share the subscribe response path for empty subscribes
The empty-subscribe shortcut duplicated the response construction, which had drifted from the regular path. Skip only the sink registration and dataflow shipping, then fall through to the shared response construction with an already-satisfied notify. Retire the subscribe with `Finished` instead of dropping it, as `retire` requires. Treat every `up_to` that is not beyond `as_of` as empty, and explain why the shortcut is required: the sink only finishes once its input frontier reaches `up_to`, which a live input never does at `MAX`. The testdrive regression now checks that no subscription is left running. The FETCH timeout alone passes even when the subscribe never completes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent ea5ba22 commit d9b42d5

3 files changed

Lines changed: 54 additions & 57 deletions

File tree

‎src/adapter/src/coord/sequencer/inner/subscribe.rs‎

Lines changed: 44 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -25,14 +25,16 @@ use mz_sql::plan::{self, QueryWhen, SubscribeFrom};
2525
use mz_sql::session::metadata::SessionMetadata;
2626
use std::collections::BTreeSet;
2727
use std::sync::{Arc, Mutex};
28+
use timely::PartialOrder;
2829
use timely::progress::Antichain;
2930
use tokio::sync::mpsc;
3031
use tokio_stream::wrappers::UnboundedReceiverStream;
3132
use tracing::{Instrument, Span};
3233
use uuid::Uuid;
3334

3435
use crate::active_compute_sink::{
35-
ActiveComputeSink, ActiveSubscribe, ActiveSubscribeOwner, SubscribeBacklogAccounting,
36+
ActiveComputeSink, ActiveComputeSinkRetireReason, ActiveSubscribe, ActiveSubscribeOwner,
37+
SubscribeBacklogAccounting,
3638
};
3739
use crate::command::ExecuteResponse;
3840
use crate::coord::appends::BuiltinTableAppendNotify;
@@ -587,62 +589,50 @@ impl Coordinator {
587589
};
588590
active_subscribe.initialize();
589591

590-
// A subscribe whose `as_of` equals its `up_to` covers the empty range
591-
// `[as_of, up_to)` and is guaranteed to produce no data. The adapter
592-
// reports this with the `EqualSubscribeBounds` notice. Skip installing a
593-
// dataflow for it: the only observable output is the initial progress row
594-
// that `initialize` already emitted (WITH PROGRESS only). Dropping
595-
// `active_subscribe` closes the channel, which completes the subscribe on
596-
// the client side.
592+
// A subscribe whose `up_to` is not beyond its `as_of` covers an empty range and
593+
// produces no data beyond the progress row `initialize` already emitted. It must
594+
// not install a sink: the sink only finishes once its input frontier reaches
595+
// `up_to`, which a live input never does at `MAX`, so `AS OF MAX UP TO MAX` would
596+
// never complete. Retiring `active_subscribe` drops its channel sender, which
597+
// completes the subscribe for the client.
597598
let is_empty = df_desc
598599
.as_of
599600
.as_ref()
600-
.is_some_and(|as_of| as_of == &df_desc.until);
601-
if is_empty {
602-
drop(active_subscribe);
603-
drop(read_holds);
604-
let resp = ExecuteResponse::Subscribing {
605-
rx,
606-
ctx_extra: std::mem::take(ctx_extra),
607-
instance_id: cluster_id,
608-
};
609-
let resp = match plan.copy_to {
610-
None => resp,
611-
Some(format) => ExecuteResponse::CopyTo {
612-
format,
613-
resp: Box::new(resp),
614-
},
615-
};
616-
// No `mz_subscriptions` bookkeeping was deferred, so hand back an
617-
// already-satisfied notify.
618-
return Ok((resp, Box::pin(std::future::ready(()))));
619-
}
620-
621-
// Register bookkeeping for the new SUBSCRIBE and ship its dataflow. The
622-
// `mz_subscriptions` write is deferred to a group commit (see
623-
// `add_active_compute_sink`) rather than committed inline, so it does not block
624-
// the coordinator loop on a timestamp-oracle round trip. We hand the notify back
625-
// so the caller can wait before returning the `SUBSCRIBE` response to the
626-
// subscribing session.
627-
let write_notify =
628-
self.add_active_compute_sink(sink_id, ActiveComputeSink::Subscribe(active_subscribe));
629-
630-
// Ship the dataflow, handling errors gracefully. With the frontend subscribe
631-
// sequencing, a dependency can be dropped between sequencing (on the session
632-
// task) and here. The read holds acquired during sequencing don't prevent that:
633-
// they hold back compaction, not drops.
634-
if let Err(e) = self
635-
.try_ship_dataflow(df_desc, cluster_id, replica_id)
636-
.await
637-
{
638-
// Clean up the active compute sink that was added above, since the dataflow
639-
// was never created. If we don't do this, the sink_id remains in
640-
// `drop_sinks` but no collection exists in the compute controller, causing
641-
// a panic when the connection terminates. This also retracts the deferred
642-
// `mz_subscriptions` write, so `write_notify` can be dropped.
643-
self.remove_active_compute_sink(sink_id).await;
644-
return Err(AdapterError::concurrent_dependency_drop_from_dataflow_creation_error(e));
645-
}
601+
.is_some_and(|as_of| PartialOrder::less_equal(&df_desc.until, as_of));
602+
let write_notify: BuiltinTableAppendNotify = if is_empty {
603+
active_subscribe.retire(ActiveComputeSinkRetireReason::Finished);
604+
// No `mz_subscriptions` bookkeeping was deferred.
605+
Box::pin(std::future::ready(()))
606+
} else {
607+
// Register bookkeeping for the new SUBSCRIBE and ship its dataflow. The
608+
// `mz_subscriptions` write is deferred to a group commit (see
609+
// `add_active_compute_sink`) rather than committed inline, so it does not block
610+
// the coordinator loop on a timestamp-oracle round trip. We hand the notify back
611+
// so the caller can wait before returning the `SUBSCRIBE` response to the
612+
// subscribing session.
613+
let write_notify = self
614+
.add_active_compute_sink(sink_id, ActiveComputeSink::Subscribe(active_subscribe));
615+
616+
// Ship the dataflow, handling errors gracefully. With the frontend subscribe
617+
// sequencing, a dependency can be dropped between sequencing (on the session
618+
// task) and here. The read holds acquired during sequencing don't prevent that:
619+
// they hold back compaction, not drops.
620+
if let Err(e) = self
621+
.try_ship_dataflow(df_desc, cluster_id, replica_id)
622+
.await
623+
{
624+
// Clean up the active compute sink that was added above, since the dataflow
625+
// was never created. If we don't do this, the sink_id remains in
626+
// `drop_sinks` but no collection exists in the compute controller, causing
627+
// a panic when the connection terminates. This also retracts the deferred
628+
// `mz_subscriptions` write, so `write_notify` can be dropped.
629+
self.remove_active_compute_sink(sink_id).await;
630+
return Err(
631+
AdapterError::concurrent_dependency_drop_from_dataflow_creation_error(e),
632+
);
633+
}
634+
write_notify
635+
};
646636

647637
// Explicitly drop read holds, just to make it obvious what's happening.
648638
drop(read_holds);

‎src/compute-types/src/dataflows.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -833,7 +833,7 @@ mod tests {
833833

834834
// `as_of = MAX` with `until = {MAX}` is an empty range, produced by
835835
// `SUBSCRIBE ... AS OF MAX UP TO MAX`. It must not be classified as single
836-
// time, and it must not trip an invariant assertion (see CLU-169).
836+
// time, and it must not trip an invariant assertion (see CPU-169).
837837
assert!(
838838
!desc_with_bounds(
839839
Antichain::from_elem(Timestamp::MAX),

‎test/testdrive/fetch-tail-as-of.td‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,19 +62,23 @@ contains:imestamp
6262

6363
> COMMIT
6464

65-
# Regression test for CLU-169. `AS OF MAX UP TO MAX` describes an empty range
65+
# Regression test for CPU-169. `AS OF MAX UP TO MAX` describes an empty range
6666
# (`as_of` equals the exclusive `up_to`), so the subscribe is guaranteed empty.
6767
# Optimization used to trip the `is_single_time` invariant assertion, which
6868
# expected an empty `until` whenever `as_of` was `MAX`, and surfaced as an
6969
# internal optimizer error. The empty subscribe is now short-circuited without
70-
# installing a dataflow.
70+
# installing a dataflow. Without the short-circuit it never completes, which the
71+
# FETCH timeout hides, so check that no subscription is left running.
7172
> BEGIN
7273

7374
> DECLARE c CURSOR FOR SUBSCRIBE t1 AS OF 18446744073709551615 UP TO 18446744073709551615;
7475

7576
# No rows expected.
7677
> FETCH ALL c WITH (timeout = '5s');
7778

79+
> SELECT count(*) FROM mz_internal.mz_subscriptions
80+
0
81+
7882
> COMMIT
7983

8084
# The short-circuited empty subscribe still honors `WITH (PROGRESS)`: it emits
@@ -87,4 +91,7 @@ contains:imestamp
8791
> FETCH ALL c WITH (timeout = '5s');
8892
<TIMESTAMP> true <null> <null>
8993

94+
> SELECT count(*) FROM mz_internal.mz_subscriptions
95+
0
96+
9097
> COMMIT

0 commit comments

Comments
 (0)