Issue building up list of items by looping through query using last evaluated key

[bpavlov]

• Goal: I have a rest API request (gettasksByType) that can return at max 20 items but if there are more in the database it will provide a last_evaluated_key. I want to use a js query to build up a list of all the tasks by repeatedly querying the API until the last_evaluated_key is empty. My issue is the loop seems to not be able to get the value of a new query trigger as if it's cached the values when it first runs and then sleeps using them.

• Details:

Currently, I have my rest API:

and my js query

but when I run it even though it will wait for the query to finish, it won't update to the new query output when setting the value for the last evaluated key. Instead, it keeps using the old value of when I first ran it:

I have looked and the query is outputting a new last_evluated_key.
lastEvluatedKey is a global variable

I don't know if this is some caching issue? Or am I not correctly waiting for the new value to be set?

• [App json export]
  I put just the query here:
  // Initialize variables
  let allItems = ;
  let loops = 0;
  var newKey = {};

async function resetValues() {
await lastEvaluatedKey.setValue({});
console.log("Reset lastEvaluatedKey:", lastEvaluatedKey.value);
}

async function runRequest() {
await getTasksByType.trigger();
console.log("After runRequest, raw data:", getTasksByType.data);
}

async function setValues() {
newKey = getTasksByType.data.last_evaluated_key;
console.log("New last_evaluated_key:", newKey);
// lastEvaluatedKey.setValue(newKey);
console.log("After setValue, lastEvaluatedKey:", lastEvaluatedKey.value);
allItems.push(getTasksByType.data.items);
console.log("Items added, current count:", allItems.length);
}

async function fetchAll() {
resetValues();
await runRequest();
await setValues();
while (lastEvaluatedKey.value?.completed_at && loops < 4) {
console.log("Loop start, lastEvaluatedKey:", lastEvaluatedKey.value);
await runRequest();
await setValues();
loops += 1;
console.log("Loop end, loops:", loops);
console.log("Loop end, lastEvaluatedKey:", lastEvaluatedKey.value);
}
}

await fetchAll();
await taskItems.setValue(allItems);
console.log("Final allItems length:", allItems.length);
return allItems;

[MikeCB]

Hi @bpavlov,

Welcome to Retool!

I think the issue is your JS functions aren't returning anything, so your await statements aren't acting the way you expect.

I'd suggest something like this:

Adjust your getTasksByType function to accept a parameter called lastEvaluatedKey.

I don't know the API you're using, but assuming you are correctly stringify'ing the contents of an object into the url param called attributes, remove the .value from lastEvaluatedKey.value since you can access params directly by name (so... ..."last_evaluated_key": lastEvaluatedKey})

Adjust you js query to something like

let allData = [] // A place to store your data
let lastKey = null // A place to store the last key
while(true) { 
 let data = await getTasksByType(lastKey) // This actually calls your query and passes in the last key. Null the first time.
 allData.push(data) // Push the data into the allData array
 lastKey = data.lastEvaluatedKey // Update lastKey
 if(!lastKey) break // If there was no lastKey this time, break out of the loop
}
return allData // return all the data

[bpavlov]

@MikeCB

lastEvaluatedKey

Hy, THank you very much for your help. unless I misunderstood you your suggesting I do this:

  let data = await getTasksByType.trigger({
    additionalScope: { lastKey: lastKey }
  });

but according to this post additional scopes cant be used in the URL portion of the rest API query Additional Scope and REST Api Queries?

am I not understanding something?

[MikeCB]

Hi @bpavlov,

I think that might be a bit of a confusing post. depending on the API you're using in Retool the places you can insert values can be limited, but generally anything that needs to accept data is available.

Do you not have access to query parameters below the API endpoint selection?

[bobthebear]

in the code you posted, you have

@bpavlov

// lastEvaluatedKey.setValue(newKey);

you didn't happen to forget to uncomment and await this did you?

i'd almost always agree with @MikeCBs suggestion but if you just want your code to work (for testing or whatnot) you can try this:

async function resetValues() {
console.log("Start resetValues");
const ret = lastEvaluatedKey.setValue({});
console.log("Reset lastEvaluatedKey:", lastEvaluatedKey.value);
console.log("End resetValues");
return ret;
}

async function runRequest() {
console.log("Start runRequest");
const ret = getTasksByType.trigger({
  onSuccess = function(data){
    console.log("SUCCESS getTasksByType:", data);
    await lastEvaluatedKey.setValue(data.last_evaluated_key);
    console.log("End onSuccess");
  }
});
console.log("End runRequest");
return ret;
}

async function setValues() {
console.log("Start setValues");
console.log("Call runRequest:", lastEvaluatedKey.value);
const ret = await runRequest();
console.log("Result runRequest:", ret.data);

console.log("Before allItems.push", allItems.length);
allItems.push(ret.data.items);
console.log("After allItems.push:", allItems);
console.log("End setValues");
}

async function fetchAll() {
console.log("Call resetValues:", allItems);
await resetValues();

for(var loops = 0; lastEvaluatedKey.value?.completed_at && loops < 4; loops++){
console.log("-------------------------------");
console.log("Start Iteration #:", loops);
console.log("-------------------------------");
console.log("Call setValues:");
await setValues();
console.log("End Iteration #:", loops);
}
}

await fetchAll();
await taskItems.setValue(allItems);
console.log("Final allItems length:", allItems.length);
return allItems;

[bpavlov]

Hy I might be getting confused. The last_evaluated_key needs to be passed in the URL. I can't change that. Is it possible to use additional scope in the URL parameter I haven't seen it be done anywhere.

[bpavlov]

Hy. Thank you for your help. Your right I forgot to uncomment it but that's not the issue, unfortunately.
I tried your code after modifying this bit

async function runRequest() {
  console.log("Start runRequest");
  const ret = await getTasksByType.trigger({
    onSuccess: function(data){
      console.log("SUCCESS getTasksByType:", data);
      lastEvaluatedKey.setValue(data.last_evaluated_key);
      console.log("End onSuccess");
    }
  });
  console.log("End runRequest");
  return ret;
}

but this doesn't even return any items as If it doesn't wait at all for the rest API query to complete

[bobthebear]

// return: Promise
function runRequest() {
  console.log("Start runRequest");

  // .trigger() returns promise
  return initAgentAndTestClient.trigger({

    // success callback function
    // param data: return object from query result
    onSuccess: function(data){
      console.log("SUCCESS getTasksByType:", data);
      current_agent.setValue(data.last_evaluated_key);

      console.log("End runRequest");
    },
    // fail callback function
    // param err: error object from query result
    onFailure: function(err){
      console.log("FAIL getTaskByType:", err);

      console.log("End runRequest");
    }
  });
}

// returns the .data as object
return Promise.resolve(runRequest());

tag me if it still doesn't do anything. its kinda bugging me now, i feel like we're missing something small

[Tess]

Hi @bpavlov Checking in here! Were you able to sort this out already?

I'd be happy to take a look in office hours if that's easier