How to upload large text dataset to Vector

Hi guys!
I'm excited about your new AI features.

I want to create my own Vector but get limit error trying to upload a relatively large text dataset with 15000 SQL rows.

How to avoid that limits? Or maybe you could give me some advice how to upload such a big dataset.

1 Like

It seems I've found a solution.
I just need to divide my dataset onto separate parts smaller that 1M tokens each and upload them as separate documents to Vector

3 Likes

@Oleksandr_Dovgopol Glad you're excited about our new AI features :smile: and thank you so much for sharing this solution - others may find this useful as well. That does seem like the right approach to stay under Open AI's token per min rate limit :+1:

In parallel, our engineers have been working with OpenAI to increase our limits as well, so this type of workaround may be less necessary in the future as larger inputs are accepted! Our limits were bumped to 10M tokens/min on Friday, for example :rocket:

Great news!
You are amazing guys!

1 Like