Vector Configuration Web Crawler Options

I have been unable to setup our documentation website to be crawled by Retool's AI Vectors. Our website (docs.redeam.io) does not return an SEO-friendly webpage (a version that the Retool crawler can interpret) unless the User-Agent header provided in the request is a known SEO service (e.g. googlebot), or a particular query parameter is provided in the request to each page (?_escaped_fragment_=).

I would like to have the ability to customize the user-agent header sent by Retool's web crawler, and / or the ability to add custom query parameters to send along with the request to each crawled web page.

Hi @gkahen Thank you for this feedback! I have just shared this internally with our Engineering team. We don't have a timeline on resolution for this yet but I will be in touch when this can be done or when I hear of a workaround how this can be achieved in the meantime :raised_hands:

1 Like

@gkahen as a workaround you can use a third party service (like Playwright, Browserless, Browserbase, etc.) to crawl and then pipe the extracted text into Retool Vectors.