Passing context to AI Agents in Apps (similar to what is done in Cursor)

Hi there Retool Team,

I've been trying to figure out the best way to pass context from an app to the chatAgent component, and currently it seems veeeery limited.

I was THIS close to actually achieve it by creating my own logic with regards to the messages sent to the agent where I could include an extra message with the context. It actually WORKS, except for the fact that the chatAgent component needs to be linked to a Retool Agentsquery.

Here's a video explaining what I managed to do:

In a nutshell below are some comments/feature requests:

  • Enable a setAgentId function for the chatAgent component
  • Enable a setQueryTargetId function for the chatAgent component
  • Enable a new editable property within the chatAgent component settings that dynamically allows to define what is the text pushed for the userMessage. It would allow to add some markup that shows that context has been added.

Looking forward to hearing your thoughts. Tagging in particular @timofey and @kent

Below the js query I used, which has a success event handler to re run the js query unless {{ agentInput.value.action === "invoke"}}

const isPolling = agentInput.value?.action === "getLogs";

if (!isPolling) {
  const chatMessages = agentChat1.messages || [];
  const lastMessage = chatMessages.length > 0 ? chatMessages[chatMessages.length - 1] : null;
  const userMessage = lastMessage?.content?.text || "";
  const selectedData = gmap1.proximityItems;
  const messageHistory = Array.isArray(agentInput.value?.messages) ? agentInput.value.messages : [];

  if (!userMessage?.trim()) {
  return; // Skip processing if the last user message is empty
}

  // Step 1: Build a new array with all valid messages
  const messages = [...messageHistory];

  if (selectedData?.length) {
    messages.push({
      role: "user",
      content: JSON.stringify(selectedData)
    });

    // Prepend context to userMessage
    messages.push({
      role: "user",
      content: `**[Context: Map, selected markers]**\n${userMessage.trim()}`
    });
  } else {
    messages.push({
      role: "user",
      content: userMessage.trim()
    });
  }

  agentInput.setValue({
    messages,
    action: "invoke"
  });

  console.log(messageHistory);
  console.log(userMessage);

  // Step 3: Trigger invokeAgent and set agentInput to getLogs on success
  await invokeAgent.trigger({
    onSuccess: (data) => {
      const agentRunId = data.agentRunId;
      if (!agentRunId) {
        throw new Error("Missing agentRunId in invokeAgent response.");
      }

      // Switch to polling mode: action = getLogs
      agentInput.setValue({
        messages,
        action: "getLogs",
        agentRunId
      });
    }
  });

  // Ensure agentInput.setValue has completed before polling
  await new Promise((res) => setTimeout(res, 100));
}

async function getAgentData() {
  return new Promise((resolve, reject) => {
    invokeAgent.trigger({
      onSuccess: async (data) => {
        if (data.status === "COMPLETED") {
          console.log(data);
          const history = Array.isArray(agentInput.value?.messages)
            ? [...agentInput.value.messages]
            : [];

          history.push({
            role: "assistant",
            content: data.resultText,
          });

          // Set both action and messages
          await agentInput.setValue({
            ...agentInput.value,
            action: "stop",
            messages: history,
          });
        }

        resolve(data); // Return original data
      },
      onFailure: (err) => reject(err),
    });
  });
}

return await getAgentData();
2 Likes

Bumping this as I would love to hear from the Retool team.

Tagging also @KeananKoppenhaver as I'm sure you may have useful insights on this.

Thanks!

Hi @MiguelOrtiz!

Thank you for bumping this and for sharing the video of your set up, that is going to be super helpful for me to distill down the feedback to share with out AI agents team.

I really like the robust use case, and I feel very strongly that users should not need to use that much Javascript to get their agents to understand the context of data from queries or components :sweat_smile:

I can hopefully get to the bottom of why the "query to trigger" is limited to Retool AI queries, as that seems like a bottleneck that we should be able to prioritize and fix. Which can hopefully reduce how much threading of data is needed and streamline the flow of data as well.

I definitely see this as a good place to work out what should be fixed and what ideal best practices look like for having an AI chat component work on page load with data from components/queries, as well as filing off feature requests for more chatAgent methods.

I will work to reproduce this set up and get back to you if I have any further questions :raised_hands:

1 Like

Thanks Jack!

I was at Retool's London event last week, and it made me chuckle that the demo from the Sales Engineer was using the agentChat component and it passed a json array with a message. It did the trick because it was a very small array, but it didn't look "pretty" nor would work out with larger context, but seeing how it was used it seems only natural that this is were the component will go.

I can hopefully get to the bottom of why the "query to trigger" is limited to Retool AI queries

So, what I was able to notice is that:

  • The query is used to define the agent within the chat component property. This is defined on page load, or every time you reset the component. However, it is not clear what's the use of having the agent defined within the chat component, but I guess some sort of match check between the messages passed through (agent start, llm start, llm end, agent end).
  • The other thing to note is that the agent chat component will only take as user messages whatever is inputed directly from the chat, or the sendMessage function, which only accepts a string. It would be nice that it could also accept an array so you can send several messages in one go (one with context, and another with the user message).

As I think through this, I think a logical step would be to incorporate app context into the way the chat component is showing tool use, e.g.

The message array has a runData > trace with each step the agent follows. If you were to allow us to pass user messages with trace (i.e. context, see orange square in screenshot) that follows a very specific format so that it shows in the chat, e.g.

[
  {
    "spanType": "MESSAGE_WITH_CONTEXT",
    "timestamp": 1750701022885,
    "agentId": "e98e2659-7609-4ee5-94ed-e184f3319a1f",
    "userMessage": "Hi there, I would like a summary of all the selected invoices",
  },
  {
    "spanType": "CONTEXT_DETAILS",
    "componentId": "invoiceTable",
    "data": [ { ARRAY } ]
  }
]

it would allow us to pass as many arrays as context. Let's say I want to add data from an invoice table, as well as data from a user. By doing this I'm saving the agent time as it no longer needs to use these tools to search the user, get the invoices, etc.....

Arguably you could use recycle the same UI for user context. Just thinking out loud :slight_smile:

Great to hear you were able to attend the event at the London office!

I am curious just for testings sake, could you share an example what the sales engineer did that did work? I understand that this likely won't scale up to work for larger amounts of contextual data but I would like to play around since I was hitting a bunch of "input must be a string" errors from my 'Agent Input' window on an agent query :sweat_smile:

In terms of potential options, in my thoughts are:

  1. Allow for "Query to Trigger" to accept non-agent queries that can contain context data via double curlies from UI.

My guess is that the agent models need to be formatted in a specific way and we want to protect users from running into errors from models.

But to your point on passing in properly formatted user messages with a trace, that could be a viable solution to fit the current backend code that handles the query builder for sending data to the agent's LLM :brain: :gear:

This also touches on the fact that the Chat Component/Agent Query currently only accepts strings, if we can allow for either synthetically created messages to contain arrays/objects as you suggest, that could also be a good option for adding in more contextual data.

  1. Giving the agent a tool to grab stateful data from frontend/UI components. So that the agent can "scrape" the current Retool app, which would reduce the need to "thread" component data/values :thinking:

  2. We currently have a Feature Request ticket for "Support Variables and In-Memory Context Hub for Agents (Session Data Store)".

Which could be a middle ground between having to thread in each component's state that you want as context, but also wouldn't send all of the data from a large app as that could quickly become a performance bottleneck.

Also just now I saw another feature request for being able to pass in data from components, so waiting to hear back on best practices for doing that. Will let you know as soon as I have updates.

Our overarching goal is to make agents as useful and accurate as possible with minimal code writing or work arounds. But having some set methods could be a good "escape hatch" for when some hacking needs to be done for unique use cases :sweat_smile:

Hey Jack,

could you share an example what the sales engineer did that did work?

Of course, they did something like the below:

  1. We currently have a Feature Request ticket for "Support Variables and In-Memory Context Hub for Agents (Session Data Store)".

I think this is a really good idea. It remains consistent with Retool's variables and local storage (which I heavily use) and would be very nice if this variable is accessible in regular event handlers when selecting the agent chat component. Then the only job for the user to pass context would be to pass it to the variable, and then Retool takes care of the rest (hopefully showing it in the chat component in a similar way as it shows the agents trace steps.

I think the tool for agent to grab state from ui components won't be very helpful. As you say, it may just take a lot of context and not exactly what the user wants....

Our overarching goal is to make agents as useful and accurate as possible with minimal code writing or work arounds.

I really like this, it shows your commitment to remain flexible but accessible. Thanks for sharing.

Hope the above helps!

1 Like

Thank you for sharing that example!

I have created a feature request to allow for agents to be able to subscribe to the state of components and potentially even entire pages in the works so stay tuned :crossed_fingers:

In the meantime, it seems that the best work around will be to:

  1. Grab component data
  2. Stringify it
  3. Pass the stringified payload into an agentChat.sendMessage() call, with some type of explanation/context to help it understand.

While this is definitely not ideal, it will allow for this data to be usable and when given more context to the data should be workable in the short term until we can give the Agents the power to hook in and subscribe to often changing front end state data :sweat_smile:

I will post with any updates I get on this ticket! :saluting_face:

1 Like

The other work around which would be much more static would be to upload data to a DB and then give the agent a tool to query/fetch data from this table.

Thank you Jack,

I do think being able to set up some sort of "Support Variables and In-Memory Context Hub for Agents (Session Data Store)" would be great.

I'm concerned that with agents subscribing to the state of components or event entire pages (!) the context window will be filled rather quickly AND/OR inefficiently. I can think of many instances where I wouldn't want the agent to do this, but I would rather provide users with options of what to pass as context.

There's a lot of trend right now around "context engineering" and I think it will become industry standard. It would be amazing for Retool to enable this in efficient but also granular ways!

Wit regards to creating the data in a DB, this is what @ultraviolet shared here

1 Like

@MiguelOrtiz Yes I fully agree!

Context engineering is the next frontier for having LLM powered tools and it will be a balancing act to get the right data and keep it up to date without filling up the window when working with large apps.

Thank you for linking that other doc, as Retool DB has been coming up as a decent medium for this.

I think it will be very interesting to see how things play out between Vectorization of data and context engineering.

1 Like

Personally, I would have agents subscribe to the page/app only and not to individual components, for 2 reasons. the first being it removes the managing of subscribing (so users don't forget to remove components or add new ones then) and 2nd, because then you could create an embedding of each component (bonus points for going back and 'on component creation' automatically creating an embedding for it). Then when the agent is ran, the user input can be used to do a semantic search on all the components to get only relevant ones and it can also be used to filter the components attribute list (no need to include position or dimensions if the input is only concerned with the text content).... idk I'm a bit late here, sorry if I've misunderstood, but the trick to this is not recalculating the full embedding when an attribute changes and instead finding a way to update the database directly that the vector/embeddings are in (I suppose its the db version of Culling/Clipping in Rendering for you graphics nuts)

just my 2 cents tho =)

2 Likes

@MiguelOrtiz I apologize for the delay here.

To (simply) summarize, would it be fair to say that you want a way to pass context into an agent that isn't so tightly coupled to the message structure that the Agent currently requires?

For example, if we added a property to the Agent Query that was something like "additional context", where you could pass in any value from the app, would that satisfy your need?

2 Likes

Hey @kent,

I appreciate you getting back to me.

Yes, that would definitely be an elegant and short term solution.

As a nice to have on top of that, would be an identical section within the agent chat component, that shows as a caption above the message text box, and which would give the user a confirmation of what has been added as context.

Hello @kent that looks useful for our use case to allow the agent to keep up with changing variables. Would this be included in every request and would they be visible in the messages? Thanks.

1 Like