How Retool Triages Support Tickets With AI, Allowing Us To Immediately Identify Critical Issues And Incidents

Isaac-H · February 19, 2024, 4:57pm

In the spirit of the February Community Challenge which is currently under way, I'm going to share some use cases of how Retool Support uses Retool with AI to help make our internal processes more efficient. Starting with our triage workflow for support tickets.

Triaging Support Tickets

What is / How does the use case work?

We use AI to automatically analyse all new incoming support tickets in Intercom to decide their impact, and identify potential incidents and critical issues. We also use AI to decide which product area the ticket should be categorised as. This use case also prepares the ticket for assignment out to one of the members of the support team.

We do all of this via a webhook trigger from Intercom to a Retool workflow. The workflow pulls the Intercom ticket information, and then runs the customer input (the support question) through an AI action to summarise the ticket and assign the impact.
We decide the impact of the issue using a Retool AI Action with Retool Vectors (more detail in the section below), with a reason of why that impact was chosen (a feedback loop allowing us to improve our prompting and Vector guidelines). We also summarise the goal of the customer from the support ticket.

Here is an example of the note which gets left on the Intercom ticket after impact is decided:

After the issue is triaged, for critical issues, we ping our support on-call teammates internally via slack API, otherwise the workflow continues on, and decides which product area the ticket lies under, and automatically assigns the issue to that inbox. This is also done via AI in the same way as we assess ticket impact.

How does the AI portion work? / How do we use AI to decide impact / product areas?

We use AI actions with Retool Vectors. For deciding impact, we break down each of the 4 impact categories (low/medium/high/critical). We create a vector document for each category, with examples for each impact, describing the type of issues which fit into each category.

E.g Here are our guidelines for low impact tickets:

In the AI action block in the workflow, we use a "generate text" action, and supply the following prompt for the input. I have censored out some sensitive prompt info:

Even this basic prompt does a fantastic job of summarising and correctly categorising issues, though there is of course still room for improvement. N.B we format the output in HTML for better formatting when posting to Intercom.

We use a similar setup for deciding which product inbox the issue belongs to, given a list of product areas and a mapping for each team, we assign the ticket to the correct product category so that it can get assigned out (using another assignment workflow) to support members with a preference for certain product areas.

How useful is it? / Why use AI?

AI does a fairly good job of summarising and categorising, which makes it ideal for this kind of automation. This use case allows us to automate our previous triage process, which would otherwise require 3-4 full time staff to handle. It also allows us to immediately identify critical issues (and handle accordingly), which means we can respond to critical customer outages within seconds/minutes vs hours/days.

Summarising issues also makes it easier when the ticket is assigned out, so that Support Engineers can quickly understand the gist of the issue.

Harsha_Wijesooriya · February 20, 2024, 3:18am

Thank you @Isaac-H for posting this. I was thinking something like the following yesterday.

"Creating a Retool-powered community support forum for Retool users. The idea is to fine-tune a virtual user using OpenAI's Assistants API. He will act as a real user and interact with other users just like his human counterpart. He will comment, and reply to comments, and users can chat with him. The finetuning will be an ongoing process with the collection of knowledge daily from the forum (especially from his human counterparts)."

What do you think?

Cheers,
Harsha

Isaac-H · February 20, 2024, 11:36am

I haven't played around with the Assistant API yet, it's not an AI action yet so you wouldn't have access to vectors to supply data. Though, as you said, you could probably fine tune gpt-3.5 and 4.0 (beta) and use a REST API query instead and provide your fine-tuned model. Though IMO this is much more effort than e.g say having a workflow to keep your vectors updated with the daily knowledge increases.

Since I'm not familiar with the Assistant's api, the way i'd probably do this is with workflows, and when a user posts a message on the forum, it sends a webhook request to a Retool workflow to handle the post, store it in a DB to display on the forum, then run it through an AI Action to respond to the user, finally returning a response code which triggers the Retool app to refresh the data sources from the new DB insert.

I'd be curious how you're going to design the app itself to host the forums, as IMO this is harder than the AI part

Harsha_Wijesooriya · February 20, 2024, 4:58pm

Thank you for the reply. I am not saying that we build a complete Forum using Retool. But the forum can be powered by Retool AI. This is what I have in my mind.

Retrieval-Augmented Generation (RAG) can be deployed in community forums to enhance the quality and relevance of automated responses. (OpenAI's Assistants API is more of a lightweight RAG.)

Here's how it can be integrated:

Retrieval Phase

In the retrieval phase, the RAG system would search for and retrieve relevant information from the forum's database or external data sources. This could include previous forum posts, FAQs, or related documentation that matches the context of the user's query

Generation Phase

Using the retrieved information, the RAG system would then generate a response that is coherent and contextually enriched. This response would be tailored to the user's query, leveraging the knowledge extracted during the retrieval phase.

Implementation Steps

Integration with Forum Software: The RAG system would need to be integrated with the forum's software, allowing it to access the database and user queries.
Customization for Forum Content: The retrieval system should be customized to understand the structure and content of the forum to fetch the most relevant information.
User Query Processing: As users post questions or comments, the RAG system processes these inputs and identifies key topics or questions.
Information Retrieval: The system retrieves relevant information from the pre-indexed forum content or external sources.
Response Generation: The AI model synthesizes the retrieved information with the user's query to generate a response.
Posting Responses: The generated responses are posted to the forum, either automatically or after review by moderators.

Benefits for Community Forums with RAG

Enhanced User Support: Users receive accurate and contextually relevant answers to their queries.
Reduced Workload for Moderators: RAG can handle routine questions, allowing human moderators to focus on more complex issues.
Continuous Learning: The RAG system can continuously learn from new content and user interactions, improving over time.

In conclusion, deploying RAG in community forums can significantly enhance the user experience by providing accurate, reliable, and contextually relevant automated responses. However, careful implementation and management are required to ensure the system operates effectively and maintains the trust of the forum's users.