Repository navigation
Grappler MemoryOptimizer TemporaryVariable survives dataset EOF, causing AlreadyExistsError #130303
Description
Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
binary and source
TensorFlow version
tf 2.14.0, 2.18.1, 2.21.0, and nightly 2.23.0-dev20261111
Custom code
Yes
OS platform and distribution
Windows WSL2 docker linux/amd64 with images python:3.11-slim and python:3.11-trixie
Mobile device
No response
Python version
3.11
Bazel version
6.5.0
GCC/compiler version
gcc 14
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
Summary
With legacy TF-Keras, a multi-input AddN in the loss is rewritten by Grappler into TemporaryVariable / AssignAdd / DestroyTemporaryVariable. On an unknown-cardinality dataset, the extra IteratorGetNext call at EOF can fail after the temporary resource was created but before it is destroyed. The next epoch reuses the resource container and raises AlreadyExistsError.
Minimal reproducible repository: https://github.com/FredBill1/tensorflow-temporary-variable-eof-repro
Confirmed scope / investigation
- Reproduced on CPU with TF 2.14.0, TF 2.18.1 + tf-keras 2.18.0, TF 2.21.0 + tf-keras 2.21.0 and TF Nightly 2.23.0-dev20261111 + tf-keras-nightly 2.23.0.dev2026111109 (Python 3.11; no distribution strategy).
- Also reproduced with an unknown-cardinality TFRecordDataset + map/shuffle/prefetch pipeline.
- Disabling
ArithmeticOptimizerdoes not fix an explicitAddN; declaring correct cardinality or replacingAddNwithtf.reduce_sum(tf.stack(inputs))avoids the failure. - Instrumented TF 2.18.1 C++ logs confirmed a successful Create during EOF without the matching Delete; the next epoch's Create used the same
StepContainer,ResourceMgr, and name and returnedALREADY_EXISTS. - The public repro includes optional logging-only C++ instrumentation (TF 2.18.1), if a resource lifecycle trace is useful.
Potentially relevant code: memory_optimizer.cc, variable_ops.cc, eager/context.cc.
Standalone code to reproduce the issue
Minimal reproducible repository: https://github.com/FredBill1/tensorflow-temporary-variable-eof-repro
docker build --platform linux/amd64 -t tf-tempvar:221 .
docker run --rm --cpus=8 tf-tempvar:221 # fails
docker run --rm --cpus=8 tf-tempvar:221 --known-cardinality # succeeds
docker run --rm --cpus=8 tf-tempvar:221 --stack # succeedsRelevant code:
import argparse
import time
import tensorflow as tf
TASKS = 2
BATCH_SIZE = 32
class DirectSumLoss(tf.keras.losses.Loss):
def __init__(self, seed, use_stack):
super().__init__(name="DirectSumLoss")
self.seed = seed
self.use_stack = use_stack
self.focal = tf.keras.losses.BinaryFocalCrossentropy()
def call(self, y_true, y_pred):
first = self.seed.read_value() # Independent of IteratorGetNext.
deps = []
if y_true.shape[0] == BATCH_SIZE:
# >80% of TF's 256 MiB CPU device memory estimate.
with tf.control_dependencies([first]):
deps.append(tf.raw_ops.Empty(shape=[55_000_000], dtype=tf.float32, init=True))
with tf.control_dependencies(deps):
terms = [self.focal(y_true[:, i : i + 1], y_pred[:, i : i + 1]) for i in range(TASKS)]
inputs = [first, *terms] # Three scalar inputs.
if self.use_stack:
return tf.reduce_sum(tf.stack(inputs))
return tf.add_n(inputs)
def rows():
for i in range(128):
yield [i / 128], [(i + j) % 2 for j in range(TASKS)]
time.sleep(0.5) # Let the independent branch execute during EOF probe.
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--stack", action="store_true", help="use Pack+Sum instead of AddN (control)")
parser.add_argument(
"--known-cardinality", action="store_true", help="declare 4 batches to avoid EOF probing (control)"
)
args = parser.parse_args()
# Explicit AddN reproduces even without ArithmeticOptimizer.
tf.config.optimizer.set_experimental_options({"arithmetic_optimization": False})
tf.keras.utils.set_random_seed(42)
print("TensorFlow:", tf.__version__, "Keras:", tf.keras.Model.__module__, flush=True)
train = tf.data.Dataset.from_generator(
rows, output_signature=(tf.TensorSpec([1], tf.float32), tf.TensorSpec([TASKS], tf.float32))
).batch(BATCH_SIZE, drop_remainder=True)
if args.known_cardinality:
train = train.apply(tf.data.experimental.assert_cardinality(4))
options = tf.data.Options()
options.threading.private_threadpool_size = 2
train = train.with_options(options)
validation = tf.data.Dataset.from_tensors((tf.ones([16, 1]), tf.zeros([16, TASKS])))
model = tf.keras.Sequential([tf.keras.Input(shape=(1,)), tf.keras.layers.Dense(TASKS, activation="sigmoid")])
model.zero_seed = model.add_weight(name="zero_seed", shape=(), initializer="zeros", trainable=False)
model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3, clipnorm=1.0),
loss=DirectSumLoss(model.zero_seed, args.stack),
metrics=["binary_accuracy"],
jit_compile=False,
)
print(
"Cardinality:",
int(tf.data.experimental.cardinality(train).numpy()),
"sum:",
"Pack+Sum" if args.stack else "AddN",
flush=True,
)
model.fit(train, validation_data=validation, epochs=3, verbose=2)
if __name__ == "__main__":
main()Relevant log output
Actual: Epoch 1 completes; epoch 2 fails at DirectSumLoss/AddN/tmp_var:
$ docker run --rm --cpus=8 tf-tempvar:221 # FAILS (AddN + unknown cardinality)
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1794579043.792556 1 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1794579043.792839 1 cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
I0000 00:00:1794579044.877451 1 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI AVX_VNNI_INT8 AVX_NE_CONVERT FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1794579047.288676 1 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1794579047.289036 1 cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
TensorFlow: 2.21.0 Keras: tf_keras.src.engine.training
E0000 00:00:1794579047.605945 1 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
Cardinality: -2 sum: AddN
I0000 00:00:1794579047.704598 1 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set.
Epoch 1/3
4/4 - 1s - loss: 0.4239 - binary_accuracy: 0.5000 - val_loss: 1.0595 - val_binary_accuracy: 0.0000e+00 - 1s/epoch - 263ms/step
Epoch 2/3
W0000 00:00:1794579048.766386 52 op_kernel.cc:1858] OP_REQUIRES failed at variable_ops.cc:119 : ALREADY_EXISTS: Resource __per_step_0/DirectSumLoss/AddN/tmp_var/frame:0/iter:0/N10tensorflow19TemporaryVariableOp6TmpVarE
W0000 00:00:1794579048.769876 52 local_rendezvous.cc:412] Local rendezvous is aborting with status: ALREADY_EXISTS: Resource __per_step_0/DirectSumLoss/AddN/tmp_var/frame:0/iter:0/N10tensorflow19TemporaryVariableOp6TmpVarE
[[{{node DirectSumLoss/AddN/tmp_var}}]]
Traceback (most recent call last):
File "/app/repro.py", line 92, in <module>
main()
File "/app/repro.py", line 88, in main
model.fit(train, validation_data=validation, epochs=3, verbose=2)
File "/usr/local/lib/python3.11/site-packages/tf_keras/src/utils/traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.11/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.AlreadyExistsError: Graph execution error:
Detected at node DirectSumLoss/AddN/tmp_var defined at (most recent call last):
<stack traces unavailable>
Resource __per_step_0/DirectSumLoss/AddN/tmp_var/frame:0/iter:0/N10tensorflow19TemporaryVariableOp6TmpVarE
[[{{node DirectSumLoss/AddN/tmp_var}}]] [Op:__inference_train_function_1069]Expected: Dataset exhaustion ends the epoch without leaving a resource that collides with the next invocation.
Activity
Fix tensorflow#130303