Description

For generating a large number of responses from OpenAI's LLM, we can leverage the Batch API to send asynchronous groups of requests and wait for their completion when rapid response is not required. This approach can also help reduce costs in use cases such as prompt evaluations and applying prompts to each entity in large datasets. Supporting this functionality would be particularly useful for workflows in Airflow, which is commonly used for batch processing of different datasets. It would be great if the OpenAI provider could support the Batch API.

Use case/motivation

To support Batch API, the OpenAI Hook would need to include the following behaviors:

1. Support for OpenAI().files.create, which is already implemented.
2. Support for OpenAI().batches.create to submit a job for OpenAI to make predictions on the file.
3. Support other behaviors that batches have such as retrieve and cancel.

I think that would be nice if we have a new deferrable operator to submit the batch prediction and retrieve the results once finished, that would be helpful.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct