Possible to disable URL crawl for vectors?

Feature Requests
,
1

I LOVE the AI actions in Retool. But one problem I'm observing with my use case - when creating a URL vector, the vector is crawling way more data than I need it to from the source URLs. So much so, that I'm noticing a material degrading in the quality and consistency of the AI generation responses that use these vectors. For example, the response is returning perhaps the first answer it finds in some linked URL, but not necessarily the best answer that I know is on the URLs I've directly provided.

I would much rather create a vector with ONLY my specific URLs, not all links found at these URLs. Is there a way to disable this?

I know it's possible to exclude specific URL paths, but I would rather categorically exclude any URLs that are not my direct input.

1 Like
2

Hey @hstan Thanks for posting this!

It seems like you could technically use that link prefix to exclude setting with the same url that you have as your url to crawl, but some kind of checkbox would be even better to exclude any URLs that are not my direct input. In either case, this isn't exposed in the create vector query. As mentioned in office hours, I've shared your feedback and use case with our ai team :slightly_smiling_face:

I'll post back on this thread when I hear from the team internally about adding this