It seems I've found a solution.
I just need to divide my dataset onto separate parts smaller that 1M tokens each and upload them as separate documents to Vector
3 Likes
It seems I've found a solution.
I just need to divide my dataset onto separate parts smaller that 1M tokens each and upload them as separate documents to Vector