Understanding How AI Agents Use Tools (with Working Example + Video)

🤝 Community Show & Tell
1

Hey everyone,

When I first started wrapping my head around Retool AI Agents and function calling, I built a prototype to see how tools actually work under the hood. That hands-on exercise turned out to be incredibly useful, especially now that Retool AI Agents are officially launched.

To help others in the same boat, I recorded a short video walkthrough explaining:

  • How tool/function calls are structured and processed by the agent
  • How the LLM (Claude, in this case) chooses to invoke tools
  • How context and state are managed across steps
  • Retool’s current limitations when dealing with multi-step reasoning and tool usage

:link: Watch the video tutorial here:

Key Use Case (Oil & Gas Invoice Coding Assistant)

I built a scenario where the agent helps assign accounting codes to invoices by reasoning through the problem and selectively calling tools. Here’s the JS query powering the logic:

const model = 'claude-3-sonnet-20240229';
const tools = agentTools.value;
const system = "You are a helpful AI assistant to the members of an oil and gas industry company. You follow a ReAct (Reasoning and Acting) framework to help users identify which accounting code to assign a given invoice. You should: Reflect on the user's request and what's been done so far. Choose whether to use several tools or provide a final answer. Clearly communicate your thought process and next steps. Always observe the previous steps taken, before deciding on your next action. Here's how your thinking process should flow: First, carefully read the invoice information provided. Second, decide which parameters you will use for the tools you have been provided with. If a given entity has several words, then you should try partial matching. Third, search for wells, vendors and accounting codes, you should make parallel function calls so that you get more information from each step. Fourth, if the information received from vendors or accounting codes provides you with additional insight, make additional function calls. Finally make a decision and provide an answer.";

// === Step 1: Tool function map (functions take a single value as input)
const toolMap = {
  find_entities_by_well: async (value) => await find_entities_by_well(value),
  find_vendor: async (value) => await find_vendor(value),
  find_accounting_codes: async (value) => await find_accounting_codes(value),
  find_afe: async ({ afe, well_name }) => await find_afe(afe, well_name),
};

// === Step 2: Conversation setup
let conversation = agentConversationHistory.value || [];

if (conversation.length === 0) {
  conversation.push({
    role: 'user',
    content: [
      {
        type: 'text',
        text: `I have invoice data that needs to be matched with entities in our database. Your goal is to recommend an accounting code, based on all of your findings.

Here’s the invoice data:\n\n${analyze_invoice.data.output[0].content[0].text}

Please analyze this data and search for: 
- Any entity names that might match records in our database. Focus on well_name, battery_name, pad_name, and any other entities that might need to be looked up. You must use the provided tools and function calls to perform searches and provide a detailed response. DO NOT PROVIDE A RESPONSE BEFORE MAKING ANY FUNCTION CALLS.
- Matching vendors in our database.
- An accounting code appropriate for the invoice

After finding matches, please return an enriched version of the invoice data with:
1. The original fields
2. Any matched entities with their full details
3. Confidence scores for the matches
4. Any recommendations for further lookups`,
      },
    ],
  });
}

// === Step 3: Loop while Claude is making tool calls
let keepLooping = true;

while (keepLooping) {
    const result = await anthropic_api.trigger({
  additionalScope: {
    body: {
      model: model,
      system: system,
      max_tokens: 4000,
      messages: conversation,
      tools: tools,
    },
  },
});


  const modelMessage = result.content;
  const toolUses = modelMessage.filter((m) => m.type === 'tool_use');
  console.log("Response", modelMessage);

  // Add assistant message to conversation
  for (const block of modelMessage) {
  conversation.push({
    role: 'assistant',
    content: [block],
  });
}
await agentConversationHistory.setValue([...conversation]);

  console.log("Variable", agentConversationHistory.value)

  if (toolUses.length === 0) {
    keepLooping = false;
    break;
  }

  // === Step 4: Handle tool calls
  for (const tool of toolUses) {
    const fn = toolMap[tool.name];
    let inputValue = Object.values(tool.input)[0];
      if (tool.name === "find_afe") {
        inputValue = tool.input; // pass full object { afe, well_name }
      }


    if (!fn) continue;

    try {
      const toolResult = await fn(inputValue);
      console.log ("tool result", toolResult);

      conversation.push({
        role: 'user',
        content: [
          {
            type: 'tool_result',
            tool_use_id: tool.id,
            content: JSON.stringify(toolResult),
          },
        ],
      });

      await agentConversationHistory.setValue([...conversation]);

    } catch (err) {
      conversation.push({
        role: 'user',
        content: [
          {
            type: 'tool_result',
            tool_use_id: tool.id,
            content: `Error during tool execution: ${err.message}`,
          },
        ],
      });

      await agentConversationHistory.setValue([...conversation]);
    }
  }
}

// === Step 5: Return final assistant message
const finalResponse = conversation.filter((m) => m.role === 'assistant').at(-1);
return finalResponse;

This setup helped me understand:

  • How to manage the agent’s internal loop while tools are being used
  • What the agent sees and how it decides to act
  • When it stalls, loops, or generates premature responses
  • The importance of aligning your system prompt with tool logic and expectations

Hope this is helpful!

1 Like