How to "stream" responses from Databricks

💬 App Building
1

  • Goal: We have a custom chat agent in Databricks that we want to be able to stream the response to retool.

  • Steps: I've tried a HTTP API Resource and a workflow but they both only return the last chunk yielded.

3

Hi @Jordan_Castro_Sauder,

I am not familiar with custom chat agents in Databricks, could you provide me with some more context on this?

My assumption is that Databricks is taking the data you have stored in it, using that as a reference point for an LLM (did you specify a model?) which is connected to a 'chat component' where you are able to ask question via messages about the given data.

I am not sure if Databricks has any documentation on how to use their API to get data from them, my guess is that in Retool a REST API would be the best option to hit a specific URL route/endpoint to request data from them but I need some more context on how they want users to access their data :sweat_smile:

If you are able to get at least the last chunk, you make need to request smaller chunks and store this in some type of intermediary DB or use some type of pagination in the query :thinking:

4

Hi Jack,

Thanks for getting back to me.

I found that in Databricks when I make the request I need to pass stream:true in the JSON body.

I found on another discussion on the retool forum which says that to do streaming I'd have to write a custom component.

For reference: Stream server-sent events as a response from an API (e.g., GPT-3) - #2 by Kenny

Although I may have misunderstood, but it seems as though streaming HTTP requests is currently not supported in Retool resources.

5

UPDATE: Looking back at this, I want to clarify that basic HTTP streaming is supported for various query types. We recently released a flag that enables it for generic REST API requests, which might solve your problem, @Jordan_Castro_Sauder. Let me know if you'd like me to enable it for your org!

This is correct. :+1: Resource queries don't currently support any long-lived connections. In some use cases, you can get around this limitation by implementing a polling query, but the other option is a custom component in this particular case.

1 Like
6

Hi @Darren,

I would be happy to be able to test HTTP Streaming!

1 Like
7

Before I close this out, let me know if you have any additional questions, @Jordan_Castro_Sauder! It does sound like HTTP streaming for REST API resources might solve your problem.