I haven’t found the information but is it possible to use the real time api with the assistant API?
Or many it isn’t usable yet but will be usable in the futur.
7 Likes
Why did they make it a separate API I find it strange as its almost the same concept except real time uses a web socket (which makes it quicker) and has tts capability. Instead of ‘threads’ you have ‘conversations’ which is basically the same thing (a method of saving a chat history on their end) instead of sending it over with every request.
Otherwise the only thing you cant do is ‘make an assistant’ which is really just a gimmick….if you think about it its simply a method of prompt caching the parameters of the call which happens anyway now when making a regular request. I have a feeling they will realise this soon (or maybe they already know) and phase out the assistants API because it doesn’t actually offer any added utility. I think it just an abstraction make it more relatable for non-developers. But please feel free to correct me if I’m missing something.
I think Assistant has a tighter integration with the Vector Store.
File Search - OpenAI API
But I agree if Realtime can incorporate the Assistants filesearch features etc., it will be a complete solution.
2 Likes
yes actually it might be the other way round, the assistants API is set up relatively well with those features, so they could have just made the real time features an option which you can switch on instead of a separate API, so when you initiate a run you can choose to use voice inputs/outputs and a websocket to interact with the model. Even if you are not using tts/stt which is expensive I think text to text communication is preferable with WS and I don’t think it’s more expensive.
Im sure they have a reason that doesnt make sense. But its not hard to recreate a local thread and storage, or you can just make a tool that one of your assistants can use that does the completion.
I think they will prob add to assistants when o1 is out of preview. They’ll probably add o1 models to assistants as well.
If not, they may be steering toward offloading the storage. They did just come out with that agenct swarm thing which doesnt use assistants.
Guess we’ll see soon.
I share your thought and it is a concern for me because I am vested in the Assistants API.
- Assistants API hasn’t seen any enhancements in a while.
- Assistants API hasn’t come out of Beta though it has been there for a very long time (more than a year for sure).
- Realtime API is hot and cool yet, it is not applicable to Assistants API.
- Assistants API is only a layer of abstraction as you mentioned. A different way of looking at things, which one could implement using the other API.
That makes me wonder if Assistants API is going to be deprecated or declared Legacy !!
I had solved the real-time conversation problem myself using various libraries and providers, even before realtime API appeared. It is not as fast as Realtime API, but acceptable and comes at a much more affordable price. Now, I am looking at providing an premium tier service option, using Open AI’s realtime API.
But, I am concerned about spending more effort in Assistants API that might go away. I request Open AI to clarify the future of Assistants API and if it would be integrated with realtime API or similar voice interface API.
If anyone has any insights, please share.
2 Likes
It sounds very inspiring. It would be interesting to hear which services were used to achieve a real-time response without using Open AI Realtime API. I also wanted to inquire whether a similar result was achieved using HTTP or a socket
I had put together Twilio, Google Transcribe, Open AI assistants, Google TTS in a pipeline to provide realtime conversation. I had used web socket for faster audio exchange. Used node.js Streams throughout, so latency is minimized. Implemented barge-in detection as well, using Twilio marks. this was pre-realtime API release from OpenAI. It was good for its time and still good for its price. Its ability to understand the audio is superior compared to OpenAI’s audio-audio model. It can handle a wide variety of accents, both input and output. Now, I have implemented a version with realtime API as well. Both versions are still available. I will decide the course based on how costs and performance evolve.
2 Likes
If you haven’t noticed, as suspected, Assistants API is targeted to sunset in the first half of 2026. Responses API will replace it. Anyone working on Assistants API should make a note and plan their efforts.