Why do I get Console log for both OnSuccess & OnFailure

App Building
1

The problem:
Console Log shows 'Winner' and 'Loser' for the uspert. But uploads successfully. I will alter Winner/Loser to hold relevant detail (meterID and dates uploaded).

image

The task is follows:
Download data history of an API point and upsert into database, mainly used for new meters so could use insert, but also used for gap filling on faulty meters.

JuggleEnergyAPI is called with emigId (meter ID) , startDate, endDate, and returns (its limited to 5000 rows but I am only pulling 336 otherwise upsert boms)
Then run upsertHistory which is pretty vanilla {{ juggleEnergyAPI.data }} with table and PK set.
Wait till complete, and make next call.

It seems to work with data being stored, but

  • I get two two console Log lines which makes no sense
  • its slow (the upsert seems to happen faster than it loops the API/upsert, is it waiting 15 seconds because its not getting a response from the upsert?
async function runJuggleEnergyAPI() {
  const weeks = 6
  const emigId = meterList.selectedRow.emigId;
  var endDate = moment().startOf('day').format('YYYYMMDD');
  var startDate = moment(endDate).add(-7, 'days').format('YYYYMMDD');

 for (let i = 0; i < weeks; i++) {
  startDate = moment(startDate).add(-7,'days').format('YYYYMMDD')
  endDate = moment(endDate).add(-7,'days').format('YYYYMMDD')
  await new Promise(resolve => setTimeout(resolve, 15000));
   await juggleEnergyAPI.trigger({
    onSuccess:   await upsertHistory.trigger({
        onSuccess: console.log("Winner"),
        onFailure: console.log("Loser")}),
    onFailure: (error) => console.error(error),
    additionalScope: { emigId, startDate, endDate }
  });
 }   
}

return runJuggleEnergyAPI();
2 Likes
2

Hi Footsore,
When you use onSuccces and onFailure, you need to pass them a function. Right now you are calling one of them correctly: onFailure: (error) => console.error(error),, but in the others you are passing them functions that have already been called which is not good. onSuccess: await upsertHistory.trigger({
I think you could also avoid the timeout promise if you rewrote it a little. Also I'm guessing your dates might be a little off. Right now you're not using the initial start and end date values.
You could rewrite this as

async function runJuggleEnergyAPI() {
    const weeks = 6
    const emigId = meterList.selectedRow.emigId;
    let endDate = moment().startOf('day').format('YYYYMMDD');
    let startDate = moment(endDate).add(-7, 'days').format('YYYYMMDD');

    for (let weeksAgo = 0; weeksAgo < weeks; weeksAgo++) {
        startDate = moment(startDate).add(-weeksAgo, 'weeks').format('YYYYMMDD')
        endDate = moment(endDate).add(-weeksAgo, 'weeks').format('YYYYMMDD')
        await juggleEnergyAPI.trigger({
            onSuccess: () => {
                upsertHistory.trigger({
                    onSuccess: () => {
                        console.log("Winner")
                    },
                    onFailure: () => {
                        console.log("Loser")
                    }
                })
            },
            onFailure: (error) => console.error(error),
            additionalScope: { emigId, startDate, endDate }
        });
    }
}

return runJuggleEnergyAPI();

Hope this helps!

2 Likes
3

It starts to, the logic behind the weeksAgo fails because it is taking an incrementing week count from the already altered startDate. Resolved by using initialStartDate/initialEndDate to deduct the count from. I chucked a console log to view the dates as it happens. (I could also switch back to deduct a week each time and increase the initial start date by 7-days - reduces variables use)

But its not waiting for upsertHistory to complete before making the next API call. Which means the returned Success/Fail dates are all for the last period as the APIs are all done long before the upserts roll through.

I tried an await in front of upsertHistory.trigger but it says 'await is not defined'.

It also says 'Functions declared within loops referencing an outer scoped variable may lead to confusing semantics. (upsertHistory, startDate, endDate)' on 'onSuccess: () => {' for ()=>.

Thanks for the help.

async function runJuggleEnergyAPI() {
    const weeks = meterHistoryRequired.value
    const emigId = meterList.selectedRow.emigId;
    let initialEndDate = moment().startOf('day').format('YYYYMMDD');
    let initialStartDate = moment(initialEndDate).add(-7, 'days').format('YYYYMMDD');
  var startDate;
  var endDate;

    for (let weeksAgo = 0; weeksAgo < weeks; weeksAgo++) {
        startDate = moment(initialStartDate).add(-weeksAgo, 'weeks').format('YYYYMMDD')
        endDate = moment(initialEndDate).add(-weeksAgo, 'weeks').format('YYYYMMDD')
      console.log("API Call:" + startDate + " " + endDate)
        await juggleEnergyAPI.trigger({
            onSuccess: () => {
                upsertHistory.trigger({
                    onSuccess: () => {
                        console.log("Success " + startDate + " " + endDate)
                    },
                    onFailure: () => {
                        console.log("Fail " + startDate + " " + endDate)
                    }
                })
            },
            onFailure: (error) => console.error(error),
            additionalScope: { emigId, startDate, endDate }
        });
    }
}

return runJuggleEnergyAPI();

image
image704×1069 117 KB

4

I read something about await only being used in async top level functions.

async function runJuggleEnergyAPI() {
    const weeks = meterHistoryRequired.value
    const emigId = meterList.selectedRow.emigId;
    let initialEndDate = moment().startOf('day').format('YYYYMMDD');
    let initialStartDate = moment(initialEndDate).add(-7, 'days').format('YYYYMMDD');
  var startDate;
  var endDate;

    for (let weeksAgo = 0; weeksAgo < weeks; weeksAgo++) {
        startDate = moment(initialStartDate).add(-weeksAgo, 'weeks').format('YYYYMMDD')
        endDate = moment(initialEndDate).add(-weeksAgo, 'weeks').format('YYYYMMDD')
      console.log("API Call:" + startDate + " " + endDate)
        await juggleEnergyAPI.trigger({
            onSuccess: () => console.log("API Success: " + startDate + " " + EndDate),
            onFailure: (error) => console.error(error),
            additionalScope: { emigId, startDate, endDate }
        });

        await upsertHistory.trigger({
          onSuccess: () => {console.log("Upsert Success " + startDate + " " + endDate)},
          onFailure: () => {console.log("Upsert Fail " + startDate + " " + endDate)}
        })

    }
}

return runJuggleEnergyAPI();

So I have moved the upsertHistory to after the API call, at which point will run regardless of the success/failure of the API call, and then succeed or fail on its own merits.
It now:

  • requests the API call, waits till done
  • requests the upsertHistory, waits till done
  • starts the next loop

Sort of a success but "API: Success" console log from juggleEnergyAP.Itrigger doesn't write a line.

I guess I should be running juggleEnergyAPI, returning (data) and then passing (data) to upsertHistory on success. UpsertHistory can then run on its own timeline. At the minute its a bit clunky with UpsertHistory using {{juggleEnergyAPI.data}} as its source.

image
image719×441 45.9 KB

5

Can we see the code for juggleEnergyAPI? What does that do? Is it trying to get data or post it to somewhere? Did you want to do something with that data? If onsuccess is not firing, than it sounds like that api call isn't actually working.

6

Oh it’s working and brings back data:

const result = data.readings.map(reading => ({
  meter: data.emigId,
  date_time: reading.ts,
  reading: reading.importEnergy.value,
  fakepk: data.emigId + "-" + reading.ts
}));

return result;

The code above calls JuggleEnergyAPI it retrieves data fine, and does a bit of transforming, available through juggleEnergAPI.data. I just don’t get the OnSuccess console log, although it puts the green line to say it ran fine.

Then it calls upsertHistory which is basically {{ juggleEnergyAPI.data }} and upserts it into the database using fakepk as PrimaryKey (upserts cannot handle combined primary keys so I just use meterId and timestamp). All data stored correctly. Happily does 52 iterations of the loop to bring back a years worth of data.

Improvement 1 - I don’t get the OnSuccess; console log from the API call saying it succeeded with dates. Is it because I am returning nothing?

Improvement 2 - I guess juggleEnergyApi-OnSuccess: should really haul back the output from juggleEnergy.data and then pass to upsertHistory to store.
That way it won’t need to wait for each API call and upsertHistory to complete before starting the next loop. Would be quicker.

And OnFailure: quit the whole loop.

(I should build in a check to see if api return is empty and not bother calling upsertHistory if it is, the odd meter has gaps due to meter/comms faults.)

But it’s all running, if a little clunky/dependant.

Dave

7

My feeling is that it's always better to pass data around in this script as opposed to using things like {{ juggleEnergyAPI.data }}. I think the latter can be stale from a previous run and you won't know.

This doesn't look like the code that's calling the api. Can we see that part?

Does this mean you want to make sure that you get all the api requests back before starting the first upsert? Because that is also an option. You could make an array of dates, go over that array and get all the data from the api, and then run all the upserts. But it seems kind of the opposite of what you said in improvement 2.

8

The juggleEnergyAPI query works fine, it’s pulling back data on meters every night and storing data.

IMG_0466
IMG_04661073×1776 184 KB

I’ll just adding additional functionality to pull back history on meters when added.

Large upserts don’t seem to work, so splitting them into week sized chunks. Reading up on the issue it seems 500 rows is optimal. A week is 336 rows and easier to deal with mentally than say 10 days (480 rows). The users can just set the weeks needed.

The API allows 5000 records (104 days).so I could make 4 API calls, store the lot in one array, then split into 500 row upsert batches. But that sounds hard.

But the easier option is an API call for a week, followed by an upsert of the same size.

As you say I could bring back all API calls into an array of arrays, then run each through upsertHistory But I don’t think this brings much benefit over calling the API, then passing off the data to upsertHistory sequentially.
The latter might be marginally quicker as the first upsert is started sooner.

Dave

10

to start, idk if this is the problem, but

  • additional scope doesn't look valid
  • while technically correct, a single statement arrow function doesn't need brackets, we can't guarantee the Retool backend uses the same rule sets. to be safe, might as well add { }
const triggerJuggleEnergyAPI = (startDate, endDate, emigld) => {
    return Promise.all([
       juggleEnergyAPI.trigger({
             additionalScope:{
                emigld: emigld,
                startDate: startDate,
                endDate: endDate
            },
            onSuccess: () => {
                console.log("API Success: " + startDate + " " + EndDate);
                return true;
            },
            onFailure: (error) => {
                console.error(error),
                additionalScope: { emigId, startDate, endDate }
                return false;
            }
       )},
       upsertHistory.trigger({
            onSuccess: () => {console.log("Upsert Success " + startDate + " " + endDate)},
            onFailure: () => {console.log("Upsert Fail " + startDate + " " + endDate)}
       })
    ])
}).then((values) => {
  //values is an array containing the return results from each promise
  console.log(values);
});

Using Await

const triggerJuggleEnergyAPI = async (startDate, endDate, emigld) => {
    const energyapi = await juggleEnergyAPI.trigger({
            additionalScope:{
                emigld: emigld,
                startDate: startDate,
                endDate: endDate
            },
            onSuccess: () => {
                console.log("API Success: " + startDate + " " + EndDate);
                return true;
            },
            onFailure: (error) => {
                console.error(error),                
                return false;
            }
       )},
       upsertHistory.trigger({
            onSuccess: () => {console.log("Upsert Success " + startDate + " " + endDate)},
            onFailure: () => {console.log("Upsert Fail " + startDate + " " + endDate)}
       })
    ])
});

// USAGE
const test_var = await triggerJuggleEnergyAPI(1234567, 1234567, 792069);
// example ret: [true, true, false]

additionalScope isn't valid. additionalScope is an Object of key/value pairs... unless the type of emigld, startData, and endDate are all of type Object. the additionalScope property can only be placed in an object as a parameter to .trigger(). this should work though i hope

11

Not sure this is correct as it moves additionalScope into the onFailure, whereas they are used to send to juggleEnergyAPI. And they seem to work fine not as key value pairs. I'll test and send them as key value pairs if works as its probably neater.

I have worked out the issue with the API onSuccess: it just didn't like startDate & endDate, removing them allowed it to write the text. Altering it to onSuccess: (results) => { console.log("API Success: " + results[0].date_time + " " + results[results.length-1].date_time); return true; }, seemed to make it happy.

I am now returning results from API call to {{dynamic_data}} and sending that as additional scope to upsertHistory (as a Key value pair). Which is probable slicker.

Removing the await on upsertHistory messes up the console logs using startDate & endDate as they get out of sync. But now I can follow them all through I'll most likely comment out some of the logs. The upsert comments are not true reflections as they just repeat the startDate & endDate, but it doesn't return the data, just a confirmation of 'Bulk Upsert by Key'. But testing by tweaking vlaues already stored proves it works.

async function runJuggleEnergyAPI() {
    const weeks = meterHistoryRequired.value
    const emigId = meterList.selectedRow.emigId;
    let initialEndDate = moment().startOf('day').format('YYYYMMDD');
    let initialStartDate = moment(initialEndDate).add(-7, 'days').format('YYYYMMDD');
  var startDate;
  var endDate;

    for (let weeksAgo = 0; weeksAgo < weeks; weeksAgo++) {
        startDate = moment(initialStartDate).add(-weeksAgo, 'weeks').format('YYYYMMDD')
        endDate = moment(initialEndDate).add(-weeksAgo, 'weeks').format('YYYYMMDD')
      console.log("Next API Call:" + startDate + " " + endDate)
        let dynamic_data = await juggleEnergyAPI.trigger({
          onSuccess: (results) => {
              console.log("API Success: " + results[0].date_time + " " + results[results.length-1].date_time);
              return true;
          },
          onFailure: (error) => {
              console.error(error);
              return false;
          },
          additionalScope: { emigId, startDate, endDate }
        });
        await upsertHistory3.trigger({
          onSuccess: () => {console.log("Upsert Success: " + startDate + " " + endDate);},
          onFailure: () => {console.log("Upsert Fail: " + startDate + " " + endDate);},
          additionalScope: {dynamic_data: dynamic_data}
          
        })
      console.log("EndLog:" + dynamic_data[0].date_time + " " + dynamic_data[dynamic_data.length-1].date_time)
//      console.log("lastRead: " + dynamic_data[dynamic_data.length-1].date_time)          
    }
}

return runJuggleEnergyAPI();

But all working, and technique improved by returning the API data to the JS script.

image
image724×329 34.4 KB

Comments welcome

Dave

1 Like
12

If it works, it works! If you wanted it to go faster, you could make these iterations run in parallel. But if it aint broke, don't fix it right?

1 Like
13

oh gees :flushed:, i fell asleep before i finished that post and when i woke up i just figured i forgot to post it... apparently i forgot to finish it though. my bad, I'm not entirely sure what I was thinking. Promise.all() def wouldn't work since the 2nd query uses the results from the 1st, but I'm glad you got your code to work!!

1 Like