How to deal with big datasets (i.e. 100k)?


I started to run into some performance issue while querying data from our JIRA REST API. The API is paginated, and in order to load all required Jira tickets, it takes forever. I know that this is to be expected, but how do others solve this?

The caching option in Retool does not really solve this as the API is paginated. Or maybe I am doing something wrong?

Or, are you using another database in between (Retool and whereever your data comes from) to store the data?

I feel like I am missing something here.. Any help would be appreciated!

Thanks, Marco

Hi Marco!

Do you know around how much data you're currently pulling into Retool? And where do you notice most of the bottleneck? Is your app slowest while the queries pull in data or is it after the data is already in your app?

Also, I'm not sure this will be helpful for you or if you've already seen it, but we do have this doc on performance best practices!

Thanks Victoria. Currently it is 20k items from Jira. The bottleneck are the API calls afaik. They are happening in sequence and sums up to a total of 15-20 minutes.
I have already seen your wiki article, but was not really helpful.
Do you have any other advice?
Thanks, Marco

Hey Marco! Sorry for the delay here. I wanted to sync with a few different teams before getting back to you with a definite answer.

There's likely not too much that can be done here, aside from not loading all the data in at once. If you can store your data in another database that maybe doesn't require so many API calls, maybe that could help. You can potentially even use Retool Workflows to regularly grab data from JIRA and push it to a database (like Retool Database if you don't already have a database spun up). This would all happen "silently" and would not affect app performance :slight_smile:

Let me know if something like this would be helpful at all and I'm sorry I don't have anything better to offer you!

Thanks Victoria! I indeed already found a solution with Retool Workflows. Based on a trigger, I pull data from JIRA API and store it in Retool Database (Postgres). This solved my performance issues. Thanks

1 Like