LLM Error - Prompt is too long: >200000 Max

💬 Agents
1
  1. My goal: Trying to build an Chat App that allows internal users to ask questions about internal M&A documents we have in PDF Format (loaded as Retool Vectors). Most of these PDFs are PDFs of Power Points with up to 100 slides. Not sure if Retool Vectors support chunking, if that is potentially causing the token error
  2. Issue: Running into this prompt error in header
  3. Steps I've taken to troubleshoot: at a roadblock
  4. Additional info: (Cloud or Self-hosted, Screenshots); Cloud
2

there is a 100 page/slide limit per file that all major LLMs have. there's also a single file size limit and a batch size limit when uploading (I think Retool has one, i forget cause they use a nifty pricing system when it comes to AIs, but OpenAI and Claude def have this).

I actually discuss a possible solution over here. it involves pulling raw text from a pdf, turning pages/slides into images, running OCR on the images, then chunking/combining the raw text and the OCR results and creating an embedding from there.

a down and dirty solution would be to use Math.Ceiling(num_of_pages % 100) to get the number of files needed for each to be less than 100 pages then use pypdfium2 to create that many new pdf files and iterate over all the pages of the original adding them to these new files you created. you could then create a new vector store and upload these splits individually. the vector store now technically contains the whole pdf.... you'll need to do some finagling with the prompt to get it to return which chunk it used then do some math to figure out which page(s) it used to create the response (if you need to know).... I will note, you may want to consult a lawyer on using this method if you do not own the files you are splitting, as technically you're creating a copy and whoever does own them may have security requirements that don't allow the direct copying of the files

2 Likes