In my response, i can only get 1000 records , but i have more which is 13832. Iām not able to view or access the remaining records, and I haven't been able to find a setting or option to display all records.
Is there a limit in Retool on the number of rows that can be displayed in a table component? If so, is there a way to increase this limit for the API call?
It appears that your API is only returning the first 1000 due to paging settings. In your API documentation there should be information about how to handle paging, possibly with a limit parameter during your API call, but more likely you will need to create some looping logic to collect all of your results (if you truly need to return them all at once).
Let us know if you still need assistance with this, @TeeDev! It looks like most of the Coursera Business API endpoints take limit and offset parameters, which allow you to retrieve specific subsets of your data. If you sent a request with limit equal to 1000 and offset to 1500, for example, the API should return the records from 1500 to 2499.
If you are limited to retrieving 1000 records at a time, which seems to be the case, you'll need to make multiple requests in order to get everything. One option is to do this in a loop, like @pyrrho mentioned. The exact implementation can differ depending on the specifics of your API and query, but below is one example what that might look like in a JS script.
let allData = [];
let limit = 1000;
let offset = 0;
let keepFetching = true;
while (keepFetching) {
const response = await queryName.trigger({
additionalScope: { limit , offset }
});
allData = allData.concat(data);
if (data.length < limit) {
keepFetching = false;
} else {
offset += limit;
}
}
return allData;
I did built a following script to pass thought the limit but it seems that with the batch limit that i have put, the code breaks at some point. Any suggestions on this. I am also quite new to the world of retool so bear with me.
async function getAllCourseraContents() {
const allContents = [];
const orgId = "xxxx";
const batchSize = 1000;
let offset = 0;
let hasMoreData = true;
let batchNumber = 0;
let consecutiveErrors = 0;
const maxConsecutiveErrors = 3;
while (hasMoreData) {
batchNumber++;
console.log(`Fetching batch #${batchNumber}, offset: ${offset}`);
try {
const response = await fetch_coursera_contents2.trigger({
additionalScope: {
orgId: orgId,
queryParams: {
contentTypes: 'Course',
limit: batchSize,
offset: offset
}
}
});
consecutiveErrors = 0;
if (!response) {
console.warn("Empty response received");
hasMoreData = false;
continue;
}
if (batchNumber === 1) {
console.log("Response structure:", JSON.stringify(Object.keys(response)));
}
const elements = response.elements || [];
if (Array.isArray(elements)) {
console.log(`Received ${elements.length} elements in batch #${batchNumber}`);
for (let i = 0; i < elements.length; i++) {
allContents.push(elements[i]);
}
if (elements.length < batchSize) {
console.log("Received fewer records than requested batch size. Ending pagination.");
hasMoreData = false;
} else {
offset += batchSize;
}
} else {
console.error("Invalid 'elements' property in response. Expected array, got:", typeof elements);
hasMoreData = false;
}
await new Promise(resolve => setTimeout(resolve, 500));
} catch (error) {
console.error(`Error in batch #${batchNumber}:`, error);
consecutiveErrors++;
if (consecutiveErrors >= maxConsecutiveErrors) {
console.error(`${maxConsecutiveErrors} consecutive errors. Stopping pagination.`);
hasMoreData = false;
} else {
console.log(`Retrying after error (attempt ${consecutiveErrors}/${maxConsecutiveErrors})...`);
await new Promise(resolve => setTimeout(resolve, 2000 * consecutiveErrors));
}
}
if (batchNumber > 1000) {
console.warn("Reached maximum batch limit (1000). Stopping to prevent infinite loop.");
hasMoreData = false;
}
}
console.log(`Completed fetching data. Total records: ${allContents.length}`);
return allContents;
}
return getAllCourseraContents();
fetch_coursera_contents2 query is essentially an API query of endpoint businesses.v1/{{orgId}}/contents?contentTypes=Course&limit=1000&offset=&q=byProgramIds&programIds={{my lists of program Ids}}
Ah got it - I took a closer look at the Coursera documentation and it looks like the API expects the parameter to be called start instead of offset. Give that a shot and let me know if it makes a difference!
