I have been unable to setup our documentation website to be crawled by Retool's AI Vectors. Our website (docs.redeam.io) does not return an SEO-friendly webpage (a version that the Retool crawler can interpret) unless the User-Agent header provided in the request is a known SEO service (e.g. googlebot), or a particular query parameter is provided in the request to each page (?_escaped_fragment_=).
I would like to have the ability to customize the user-agent header sent by Retool's web crawler, and / or the ability to add custom query parameters to send along with the request to each crawled web page.
Hi @gkahen Thank you for this feedback! I have just shared this internally with our Engineering team. We don't have a timeline on resolution for this yet but I will be in touch when this can be done or when I hear of a workaround how this can be achieved in the meantime
@gkahen as a workaround you can use a third party service (like Playwright, Browserless, Browserbase, etc.) to crawl and then pipe the extracted text into Retool Vectors.
I wanted to circle back here to share an update - this request is still in our backlog. Please continue to add your +1s & feedback, if you're looking for this feature
