Hi, it would be extremely powerful to be able to dynamically create a document vector database. Currently any time I need to create a new vector database I have to manually create it via the UI. This puts a limit on the ability to create a fully dynamic ai application that can bucketize different content into different vectors.
1 Like
Would be great to have an "out of the box" version of this type of a workflow, maybe something that would allow users to plug in destination (assuming it would probably start with the Retool Vector DB) and perhaps even chunking strategy.
Hello @SplinteredGlassSolutions!
I can definitely pass along suggestions to our AI team. Could you give me some more details on what you mean by 'dynamically' create a document vector db?
How are you imagining your fully dynamic ai app to bucket the content it is given? How would it get access to the content? Would you be uploading documents, giving it URLs to crawl or feeding in text input from a retool app user?
Currently Retool supports Amazon Knowledge Base, as well as Retool Managed Vectors. I would love to know about any shortcomings these options have and how they could be improved or what alternatives be built by the Retool team!
Increasing the viability and usability of our AI product is a major priority for Retool and I am very new to Vectors 
so any and all detailed examples will be very helpful as we would love to get suggestions on what our team can build to help!
Dear Jack,
Thank you for your response and interest in improving the functionality of Retool's AI capabilities. I appreciate the opportunity to provide more context and details regarding our need for dynamically creating document vector databases. As mentioned in my previous message, our organization deals with multiple departments, each with its own set of folders. These uaers frequently upload various types of documents, such as audio files or PDFs, which contain information relevant to their respective departments and sub folders. Our goal is to provide each folder with a personalized AI experience tailored to their specific domain and access privileges. Currently, when a new sub department is introduced to our system, we are required to manually create a new document vector database within the Retool UI. This process involves navigating to the appropriate section, clicking the "Create New Document Vector Database" button, and then finalizing the creation. Once this step is complete, we can then save and associate documents with the newly created vector database using our Retool workflows and applications. However, the manual process of creating these vector databases becomes increasingly cumbersome as the number of subfolders grows. It is not sustainable to rely on this manual approach, especially when new subdepatrtmenta can be added at any given time across different departments. To address this challenge, we propose the ability to create document vector databases programmatically, potentially through an API or an additional action within the existing Retool vector actions. This would allow us to create and associate a new vector database for each subdepartment automatically, without the need for manual intervention.
The rationale behind assigning individual vector databases to each sub department is to ensure data segregation and access control. By leveraging these user-specific vector databases within our Retool workflows and applications, we can tailor the AI experience to each user's access privileges and the information they are authorized to access. To clarify the workflow, when a new sub department is added to our system, our applications would automatically trigger the creation of a dedicated vector database for that user. Subsequently, any documents or content uploaded or associated with that sub department
would be fed into their personalized vector database. This approach would enable us to maintain data separation and privacy while providing a seamless and tailored AI experience for each user within our ecosystem.
Wow, our particular use case deals with sub departments. I could see the need for this being extrapolated to a variety of use cases such as multiple users needing to have their own individual vector database or any other time when there is a dynamic need to segregate the data that is being fed into the various vectors And then used to feed the retool AI actions
I hope this additional context helps clarify our use case and the need for dynamically creating document vector databases. We would greatly appreciate any insights or solutions the Retool team can provide to address this requirement more efficiently and scalably. Please let me know if you require any further information or have additional questions. We are open to exploring alternative approaches or suggestions that align with our goals and the capabilities of the Retool platform. Thank you again for your attention to this matter. We look forward to your response and continued collaboration.
Best regards, Preston Pope Lead Architect Splintered Glass Solutions
1 Like
Thank you so much for the detailed explanation Preston!
The use case makes perfect sense, having a vector DB for each department and their folder upon creation which will then add in sub-folders/files as the department's parent folder grows is a great use case for large and growing orgs to take full advantage of Retool's AI capabilities.
My first question is regarding the folder for a department, is this folder in Retool or are the documents/files stored elsewhere?
My first thought was that when a new sub department is introduced, you could have that trigger a Retool workflow. This workflow could create a new vector DB and could then iterate through the files/folders/resources to feed the data into its corresponding vector DB.
As well as having the creation workflow build out a second workflow that runs when new resource/documents are uploaded to the folder, to continually add in new information to the vector DB. This can also be used for current folders.
Let me know your thoughts on this approach and I will pass along your post to our AI team.
If the department's folder is not in Retool, we currently don't have integrations to automatically trigger specific AI actions when an event occurs in a third party location. However, workflows would likely be the best option for triggering and automating behavior in Retool when specific events occur.
hey Jack. but is there a way to create a new vector db using workflow. i currently have to manually create it
Hello @Ayomide_Bakre,
Unfortunately not yet, currently all vectors must be made manually from the UI at https://jtomasik.retool.com/vectors.
Will be making a feature request as it seems that many users of our AI features want to work with and be able to create new Vector DBs via workflows automation!