Cannot invalidate cache for queries in some apps

[dguzzo]

Video of the steps below, performed in realtime. [21sec duration]

I've been back and forth with Retool support on this issue for a while, in Live Chat (which was removed from our paid Team plan not long ago), email support, and in the forums.

I'm trying again here because it's so obviously not working across a number of our apps. Previous attempts to raise it in the forums would be intermingled with other reports of the similar or even same result (invalidation not working) but with different symptoms / preconditions / etc. [example 1] [example 2]

Not having control of cache invalidation severely limits the type of performance and functionality we'd otherwise expect to be able to control.

I'm able to invalidate the cache in this manner for a similar query to the same endpoint when creating a new barebones app, and for this and previous unrelated issues the troubleshooting line prescribed has sometimes been: "What if you just recreate your entire app?" For various reasons I'd like to avoid doing that (loss of release history, for one), and I'd guess other builders & maintainers of long-established apps would feel similarly. So it seems like app functionality in this regard tends to degrade over time, which is a problem for non-trivial tools.

Steps in video:

1. Run query for the first time via blue button in Code drawer query details: see expected output in console:
   
   

---

2. Run again immediately after: see expected "from cache":
   
   

---

3. "Invalidate cache" from the query's triple-dot options; see the console reflect that. Note that the POST returns 204 and shows the expected request payload sent:
   
   
   
   
   
   
   
   
   
   
   

---

4. Run query again and see unexpectedly that it's still returning "from cache":
   
   

[bobthebear]

I have no idea what you've tried or haven't so hopefully you haven't tried one of these:

• using cache.delete(). docs here

caches.keys().then(function(names) {
    for (let name of names)
        caches.delete(name);
});

caches.open("v1").then((cache) => {
  cache.delete("/images/image.png").then((response) => {
    someUIUpdateFunction();
  });
});

• using .getRegistrations() to manually unregister all service workers, some of which can be responsible for caching resources

if ('serviceWorker' in navigator) {
   navigator.serviceWorker.getRegistrations()
   .then(function(registrations) {
      for(let registration of registrations) {
         registration.unregister();
      }
   });
} 

• location.reload(true). force the page reload to clear the cache and reload it

[dguzzo]

Hi @bobthebear, these approaches seem focused on clearing my browser's local cache(s), whereas Retool's query caching happens on the server. So the response is coming back cached when it shouldn't be.

Regardless, thank you for the suggestions. I tried them — in some cases replacing the inner operation with a console.log as a dry-run, out of precaution — and either they had no effect or would not have had any results to operate on.

[saldmy]

I am encountering the exact same issue myself.

Initially, the cache invalidation mechanism functions adequately across all queries. However, over time, it begins to fail on individual queries, leading to a rapid proliferation of broken queries.

The issue seemed to dissipate temporarily after duplicating the application, but the respite was short-lived. Repeatedly cloning the applications on a weekly basis is not a viable solution either, as they rely on each other's unique identifiers (UIDs), and the client requires stable links to the applications.

This bug poses SIGNIFICANT ramifications for the project, particularly because the client demands a highly responsive application, a requirement challenging to meet without effective database caching in our current setup, thus becoming an obstacle that prevents the program from being released.

I hope that this problem will be solved as soon as possible.

[Tess]

Our team is investigating this issue! So far, it's been intermittently reproducible which can make it hard to pin down where the issue is, but I definitely hear you that it can have serious ramifications We'll reach out with any updates that we get!

[dguzzo]

Thanks for the update, @Tess ! Glad this is getting some attention.

I'd be happy to do a screen-share with Retool support sometime if that'd help troubleshoot it in situ, given the authenticated resource needs in our apps.

[Tess]

Thanks, will follow up if it sounds like that would be helpful in moving this forward!

[Harry_Doan]

Hi @dguzzo

I'm sorry that you're experiencing issue with our cache invalidation mechanism. I'm looking into this issue and I'd like to ask for a few more details to help me debugging this:

• What type of resource are you seeing problem? Is that Rest API in the screenshot?
• Do you use the latest version of the app when seeing this issue?
• Do you use source control/branches when seeing this issue?

[Harry_Doan]

@saldmy Can you answer the questions above as well?

[saldmy]

• What type of resource are you seeing problem? Is that Rest API in the screenshot?
  MySql Resource queries.
• Do you use the latest version of the app when seeing this issue?
  Retool Cloud.
• Do you use source control/branches when seeing this issue?
  No.

I also noticed, that situation stabilized and the proliferation of broken queries seems to have stopped. It seems that most of malfunctioning queries have transformers inside, but I am not sure this factor plays any role.

[Harry_Doan]

Thank you! I've identified that this is an issue when you're not using the latest release of an app (see in the bottom left corner of the app). I'm working on a fix and will let you know when it's released!

[Harry_Doan]

I've merged the change to fix this cache invalidation issue. Expect it to be fixed by the EOW next week!

[dguzzo]

super, thanks for the update, Harry!

[Harry_Doan]

@dguzzo @saldmy I think the fix is released. Can you give it a try and let me know if your problem is fixed?

[dguzzo]

Hi Harry — I only did a quick check but I think this is resolved! I'm cautiously optimistic :]

Would you mind keeping this issue open until I can check closer (likely early next week) and when saldmy has had a chance to chime in?

[dguzzo]

I'm happy to report that all of these work, where previously none would. It was glorious seeing this in the Debug console:

... whereas previously the final line would still retrieve via the cache instead of freshly, as is exhibited now.

I tested the validation in the three ways I know possible: triple-dot query option on the Code drawer (Cmd+Shift+M on Mac), the left sidebar in the query list (also in a triple-dot menu), and finally programmatically via a query's .invalidateCache() method. All worked correctly.

Thank you so much, @Harry_Doan — this was a problem for us for so long, and greatly limited the circumstances in which we could apply caching. Now we're in business!

[Harry_Doan]

Great! Thank you for reporting it and helping us with the debugging! Your video helped me in identifying the issue. It was super valuable!

[dguzzo]

Also thank you @Tess for following through on this one — I know it was not easy to pin down