Pull in all Airtable data

💬 Queries and Resources
1

I am currently using a REST API to pull in data from Airtable, but as you probably know, it will only pull in 100 records. I spent most of last evening trying numerous suggestions for pulling in all the records that I found here in the community and they would either not work, or would not stop running. If anyone knows of a reliable to do this, please let me know.

2

I don't know Airtable, but if you set up the REST API and assuming there is some sort of pagination offset available as a variable in that API, you could create a JS Query with something along the lines of:

// This pseudo code won't work without you making updates...
let allRecords = []; //store all the records here
let offset = null;   //manage offsets here

(async () => {
  while (true) {
    // go get the first set of records
    const data = await airtableFetch.trigger({ additionalScope: offset });

    if (data !== null) {
      // Assuming you got records, append them to allRecords
      // no idea where records will actually be in the data object, but...
      allRecords = allRecords.concat(data.records);

      // Presumably there is some sort of offset returned to use for the next set
      // no idea where it is actually stored, but...
      if (data.offset) {
        offset = data.offset;  //or increment by 100 each loop?
      } else {
        break; // No more records to fetch, exit the loop
      }
    } else {
      break; // An error occurred, exit the loop
    }
  }

  return allRecords;
})();

Did you try something along these lines? If it didn't work, can you share some screenshots and maybe someone can help out.

3

Thanks for your assistance on this. I tried numerous ways to make this work as well as other solutions in the community and nothing works. In fact I think I got myself in a loop and it took me some time to figure out how to get out of it. So, I'm going to see if there are any other programs out there like Retool that won't force me to struggle with this sort of issue. I already had my apps built and this was the last step I needed to accomplish.

4

Hi @tomm, we don't set a limit for how many rows you can get from Airtable. The query size limit on Retool is 100MB, which can be well over a million rows depending on datatypes.

if the following is true:

This would be a limitation on Airtable's end, not on Retool. The workarounds you've found, including @jg80's approach, are using the power of Retool to go around it. But perhaps Airtable offers a different paid plan where such limitation is not present.