Workaround for browser javascript integer limitations?

Apologize if it was already asked/solved I did not find my answer after searching for a bit.

My problem is the following: I'm trying to prototype building a small admin app on top of a REST API for my team, everything's great and love the tool, perfect for our need need BUT, I'm encountering an issue:

Most resources in this API are identified by 64 bit integers. And everybody knows a bout javascript big number restrictions. The solution I have in mind, since those numbers are just identifiers and nothing else, no operations would be performed on them, would be to parse them as string.

But I don't know how and where to do that in the context of Retool ?

Hey @Nayef_Derwiche, welcome to the fam :hugs:

That would be a perfect fit for the transformer function in your query. Just map over the results of the query and convert your column to any type that works for you.

Does that help?

Hey! Thanks for the answer,

This does not really help as I am talking about an http query (and not an SQL query) and in the data I have access to in the API Resource, the information is already lost for the id, does not matter if I transform it to a string. Am I missing something ?

I think I would need to access the raw binary data and parse the JSON preserving big int... do you know a way of doing that ?

Got it, so query transformers work on every query.

I'm wondering if directly transforming the reponse in the query transformer via rawData works.

I mean the transformer here, not creating another transformer query.

No that's already what I tried, in rawData the id is already wrong. The rawData object is already parsed from the json as an object, so int fields are parsed as numbers and the data is lost.

Hi @Nayef_Derwiche

Thanks for reaching out about this! This issue comes up from time to time, as Retool uses Javascript and JSON.parse to serialize and deserialize API responses throughout our code base. Our team has investigated whether there is anything we can do on our side to better support big integers, but this isn't something we can easily (and reliably) change. Typically, our recommendation is that the API vendor serializes large integers as strings, so that you're able to work with accurate data coming in to Retool.

Thanks for sharing your use case for an admin app prototype! This does sound like a good use case for Retool. Are you able to share the API you're using?

Hi @Tess !

Thanks for the reply! I have two solutions now:

  • Not using the API resource but use a js query which makes the http request, I parse the raw http text data with a custom parser. But I loose out on build-in nice functionalities of retool I guess... if I go this way what is the best way to use this and take advantage of, for instance, ux components being setup easily on this resource, auto-relaunch when watched input changes etc...
  • A simpler way is to ask the team to change their ids and return big int as str :stuck_out_tongue: , it's an internal API for ML workflows so they usually interact with python and don't have this issue right now but I looked and it's a common issue and it's admitted that using str for big ids is quite common (that's what twitter does for instance.)

Can't share the API unfortunately as it's internal!



1 Like

Hi @Nayef_Derwiche,

Are you asking because JS queries only run manually?

Bradly mentions a workaround here. You can have an automatic query that triggers your js query like this:

  1. Create an automatic query that watches the input. This sample query will run every time textInput2 changes

2. Add an event handler that triggers your js query after the automatic query runs. Since my automatic query is a dummy query, it will always fail. For this reason, I'll add a failure handler that triggers my js query

3. Note that the js query runs 'automatically'

Ah yes sweet, good to know that the "smart refresh" works for all kind of resources, even advanced js queries, as long as you setup your inputs well.

In any way it was easier and simpler to just use str as ids.



Hi there, I wanted to chime back in here to add that we are working on support for BigInt data (as numeric value) with REST API resources :raised_hands: It should be available on Cloud and will be coming to on-prem soon

1 Like