Vector url crawler

Queries and Resources
1

hi ya'll

i did not find any documentation on vector url crawler:

  • user agent?
  • IPs?
  • crawl speed / throttling?
  • how to adjust crawl "deepness" from 1000 to let's say 5 levels, or 0 to crawl only a list of pages etc.

i'm asking since i don't want to release the crawler on a huge site without knowing the crawler a bit better. also, i want to know for which crawler to punch a hole in our WAF.

thanks in advance

-kimi-

2

Hi :slight_smile:

@victoria Can you point me in the right direction with this?

-Kimi-

3

Hello @Kimi!

Unfortunately Victoria has left the Retool team to adventure around the world.

In the meantime, one of our engineers who specializes in AI is preparing a full formal response with greater detail for Vectors!

Stay tuned :grin:

1 Like