This is a feature request for support of caching for hashable data types as implemented in academic serving systems Clipper: A Low-Latency Online Prediction Serving System section 4.2 and also similar frameworks like Triton inference server. This could be very useful for non custom runtimes like HuggingFace runtime.
This is a feature request for support of caching for hashable data types as implemented in academic serving systems Clipper: A Low-Latency Online Prediction Serving System section 4.2 and also similar frameworks like Triton inference server. This could be very useful for non custom runtimes like HuggingFace runtime.