Custom AI Provider cannot stream output in LLM Chat component

💬 Queries and Resources
1

Hello Retool community,

I am trying to connect to my own chat completion endpoint, my chat endpoint is custom using fastapi (python), I have tried many ways, but LLM Chat component doesn’t stream my output, instead it displayed everything at the end which is quite a long time.

Anyone knows how can I stream output from custom endpoint?

3

Hey @wonka - we're in the early stages of adding support for HTTP response streaming to the REST API resource. If you're interested, I can raise the corresponding flag for your org. :+1:

1 Like