Snakemake version
9.13.7
Describe the bug
I am trying to use the continuous input queue as per documentation. However, my update_results() function is rather time-consuming. In that case, snakemake fails to build the DAG and run any rules. I've taken the example from the documentation and added a 20 second delay at the beginning to simulate the behaviour. Without this delay, the example works without problems.
Minimal example
import threading, queue, time
# the finish sentinel
finish_sentinel = object()
# a synchronized queue for the input files
all_results = queue.Queue()
# a thread that fills the queue with input files to be considered
def update_results():
try:
time.sleep(20)
while True:
for i in range(10):
all_results.put(f"test{i}.txt")
time.sleep(1)
all_results.put(finish_sentinel)
all_results.join()
except (KeyboardInterrupt, SystemExit):
return
update_thread = threading.Thread(target=update_results)
update_thread.start()
# target rule which will be continuously updated until the queue is finished
rule all:
input:
from_queue(all_results, finish_sentinel=finish_sentinel)
# job that generates the requested input files
rule generate:
output:
"test{i}.txt"
shell:
"echo {wildcards.i} > {output}"
Snakemake version
9.13.7
Describe the bug
I am trying to use the continuous input queue as per documentation. However, my update_results() function is rather time-consuming. In that case, snakemake fails to build the DAG and run any rules. I've taken the example from the documentation and added a 20 second delay at the beginning to simulate the behaviour. Without this delay, the example works without problems.
Minimal example