Problem with NULLs being turned into empty strings in Select in Form, (setting default in Select helped)

Summary
Retool turns null values into empty strings even if their type is not string. This causes a type mismatch when writing the data back to the database.

More details
I'm sure I've written about this issue before in connection with columns of type DATE. I can't find that comment right now, but anyway - it has cropped up with type int.

The database (mine at least) allows columns of any type to have value NULL.

This even applies to foreign keys constrained in the schema to only match actual values of the corresponding key in the target table. They can still have default value null.

When I create a table, I always set columns that initially have no need of a particular value, to DEFAULT NULL. Again this can apply even to foreign keys.

Often these columns will appear in table components or forms where I may or may not change them to actual value (not null). When the form content is written back to the database we find that Retool has changed all null values to empty string.

For data of type varchar for example this doesn't cause a problem. But I still find it irritating that I'm getting "" where I wanted null.

In the case of data of type DATE, (as already mentioned somewhere) this causes a type mismatch error and the update/insert fails.

Recently I've noticed the same occurs with data of type int, also causing a failure of the update/insert.

The work-around I've been forced to use previously is - in the query GUI doing the update/insert, to specifically catch empty string and replace it with null.

Update note:
I've just realised I don't really have a valid work-around.
How to check a variable that may have an INT value or NULL (both valid) but could now have value empty-string (invalid)?
End of update note

After doing quite a few of these, and also getting errors and losing time tracking down errors where I've missed to do this, it's getting a but tedious.

Also it seem so cumbersome and unnecessary. Why can't Retool just leave NULLs as NULLs?

I'll continue here so as to not confuse the original post.

Using the ternary operator looking for unwanted empty-string
(new Date())? gives true -OK
(1)? gives true -OK
(0)? gives false -Not ok (0 is valid for an int)
("")? gives false -OK to detect empty-string that should be NULL

So...

{{ (form6[indexToUpdate].data.Price_unit_Id)?
 form6[indexToUpdate].data.Price_unit_Id : null}}

... would work except for cases of an int with real value 0.

a couple tips to help out:

 - if you compare using 'string == number' javascript will convert both sides to a number, then compare
- if you compare using 'string === number' javascript will NOT convert before comparing
 - "123" == 123                                          -> true
 - "123" === 123                                         -> false
 - new Date() == new Date()                              -> false
 - (new Date()).getTime() == (new Date()).getTime()      -> true
 // odd, why is Date different?
 // return (typeof (new Date())
 // this gives us 'object'
 // when javascript compares 2 'object'(s) it doesn't compare the contents of either object
      // instead it compares each objects 'location in memory' a.k.a. 'address'
      // an 'address' is just a hex number (like  0x001)
      // every new object gets its own address"
      // so if we make 1 new object, its address is 0x001
      // now if we make another new object, its address will be 0x005
      // which means:  'new Date() == new Date()' creates 2 new objects, THEN it compares
       // giving you:  '0x001 == 0x005' which is false

var my_date1 = new Date('2025-01-01T10:00:00Z');
var my_date2 =new Date('2025-01-01T10:00:00Z');
var my_date3 = my_date1

console.log(my_date1 == my_date2)  // false
console.log(my_date1 === my_date2) // false
console.log(my_date2 == my_date3)  // fase
console.log(my_date3 == my_date1)  // true

console.log(my_date1.getTime() == my_date2.getTime())  // true
console.log(my_date1.getTime() === my_date2.getTime()) // true
console.log(my_date2.getTime() == my_date3.getTime()) // true
console.log(my_date3.getTime() == my_date1.getTime()) // true

var test_numbers1 = 123
var test_numbers2 = "123"
console.log(test_numbers1 == test_numbers2)                       // true
console.log(test_numbers1 === test_numbers2)                      // false  
console.log(String(test_numbers1) == String(test_numbers2))       // true
console.log(test_numbers1.toString() == test_numbers2.toString()) // true
console.log(Number(test_numbers1) == Number(test_numbers2))       // true

var test_letters1 = "abcd"
var test_letters2 = "abcd"
console.log(test_letters1 == test_letters2 )                        // true
console.log(test_letters1 === test_letters2 )                       // false  
console.log(String(test_letters1 ) == String(test_letters2 ))       // true
console.log(test_letters1 .toString() == test_letters2 .toString()) // true
console.log(Number(test_letters1 ) == Number(test_letters2 ))       // false
console.log(Number(test_letters1 ) === Number(test_letters2 ))      // false
// these next ones can, and have, cause lots of problems when it comes to string comparison implementations
console.log(Number(test_letters1 ))                                 // NaN
console.log(NaN == NaN)                                             // false
// NaN is a special case in Javascript
// NaN is never equal to anything, including itself
// NaN is different from 'null' and 'undefined'
// when '==' is used, 'null' and 'undefined' are the same
    // so if you have 
    // if ('my_expression' == null) {}
    // 'my_expression' can evaluate to 'undefined' or 'null', in either case it will be true
    // but if 'my_expression' evaluates to 'NaN', this if statement will never evaluate to true

var test_mixed1 = "a"
var test_mixed2 = 1
console.log(String(test_mixed1) == String(test_mixed2))  // false
console.log(Number(test_mixed1) == Number(test_mixed2))  // false

TLDR;

  • if you convert values to a string as soon as they're read, then on down the road if you need to do a comparison you'll know at least 1 value will always be a string, which makes things easier since you now only have to worry about type of 1 value instead of 2.

I believe you mentioned that every column could be of any type or null, pointing out problems with mixed type comparisons:

{{  (form6[indexToUpdate].data.Price_unit_Id && ['string', 'number'].includes((typeof form6[indexToUpdate].data.Price_unit_Id).toLowerCase()))? 
    String(form6[indexToUpdate].data.Price_unit_Id) : 
    form6[indexToUpdate].data.Price_unit_Id instanceof Date? 
      String(form6[indexToUpdate].data.Price_uint_Id.getTime()) :
      typeof form6[indexToUpdate].data.Price_unit_Id === 'object'?
        JSON.stringify(form6[indexToUpdate].data.Price_unit_Id) : null  }}

// Date is of type 'Object' so we can't use typeof to know what type of object
// instead we use 'instanceOf', because we want to know if the Object is a direct instance of a Date
// we know comparing .getTime() with another number works, but if you compare .getTime() to a string containing numbers, we get unexpected results
// so instead, cast .getTime() to a String (you could also use .toString() but there are a few edge cases )
// JSON objects can be tricky to compare, you either have to do some sort of depth-first traversal and comparison of all key and values
// OR
// you can cheat, turn the json object into a string and now we can compare the keys and values of 2 json object strings all at once (instead of recursive traversal)

If you know for a fact you only have Numbers, Strings and Dates (never anything else) you could simplify it

{{  
  (form6[indexToUpdate].data.Price_unit_Id && ['String', 'Number'].includes((typeof form6[indexToUpdate].data.Price_unit_Id)))? 
    String(form6[indexToUpdate].data.Price_unit_Id) :   
    String(form6[indexToUpdate].data.Price_uint_Id.getTime())  }}  

// the 1st line tests if the key/values exist and if the value is a Number or String
// 2nd line converts all numbers and strings into a string
// 3rd line uses the rule of elimination to assume if the value isn't a number or string then it has to be a Date, so use getTime() to convert to a number and use String or .toString() to convert to a string.

you could try a more generic approach and handle Date and JSON objects the same:

{{  
  (form6[indexToUpdate].data.Price_unit_Id && ['String', 'Number'].includes((typeof form6[indexToUpdate].data.Price_unit_Id)))? 
    String(form6[indexToUpdate].data.Price_unit_Id) :   
    JSON.stringify(form6[indexToUpdate].data.Price_uint_Id)  }}  

or if you know you're only dealing with Numbers and Strings:

{{ String(form6[indexToUpdate].data.Price_unit_Id) }}

notes:
interestingly you don't necessarily need to do this to whatever value you are comparing form6[indexToUpdate].data.Price_unit_Id to, that's because all of the above options result in a String and since this is forced we know when we compare it to some variable javascript will do what it can to convert the STRING value to a Number OR if we are comparing to an object javascript will do what it can to convert the OBJECT to a String.

if you're wondering, there are certain rules javascript follows when deciding to convert a value from one type to another when using == and just like PEMDAS in math these are in a specific order:

  • If the types are the same, a strict comparison is performed.
  • If one operand is null and the other is undefined, they are considered equal.
  • If one operand is a number and the other is a string, the string is converted to a number.
  • If one operand is a boolean, it is converted to a number (true becomes 1, false becomes 0).
  • If one operand is an object and the other is a primitive, the object is converted to a primitive value.

No, my columns are all off fixed types.

I think you missed my point.

It's in the title. Retool turns my null values into empty strings even if their type is not string.

That is the problem.

my bad, I guess I'm just confused cause wouldn't using the logical or operator work?
{{ form6[indexToUpdate].data.Price_unit_Id || null }}

if Price_unit_Id is 'falsy' (null , undefined , 0 , -0 , '' , NaN or false )

  • then use null
  • otherwise, use the value of 'Price_unit_Id' (String or Number)

btw the code in my first post I feel like would still work and I still think this is a problem with data types between Javascript and the database. on the DB side, NULL is the same as '' or empty string, it's like an alias or something apparently.... if that's the case, that'd mean when retools backend gets the value it's actually getting an empty string and not the NULL that you're expecting, but the code above should fix it

{{ form6[indexToUpdate].data.Price_unit_Id || null }}

Hi Scott,

I appreciate you trying to help, but I think you are making a simple matter more complicated than it needs to be.

Concerning your logical test, it has the same problem as this...

{{ (form6[indexToUpdate].data.Price_unit_Id)?
 form6[indexToUpdate].data.Price_unit_Id : null}}

a) it's just a work-around for an issue and
b) it fails if the integer concerned is 0. 0 will translate to false, but 0 is a valid value for an integer.

More to the point, if the issue was fixed we wouldn't need work-arounds of this kind.

You say...

No, null is not the same as empty-string - that is the basis of the whole issue (see below).

You say...

Wrong again. It's easy to examine the value obtained from a query. Here is an example containing nulls. I just select "State" of the table, find the data array and open a row...

You see? Several nulls all present and correct.

But after using the values in a form, and calling a query GUI to update the database we see the following...

As I have been saying, nulls have been converted to empty-string.

For a database column of type integer, null is a perfectly fine setting, but empty-string is rejected as a type mismatch.

no need to get all snippy

Wrong again. It's easy to examine the value obtained from a query. Here is an example containing nulls. I just select "State" of the table, find the data array and open a row...

how am I suppose to have this information from your state window to know this?

And ya I was somewhat wrong, ok I had to go look at the postgresql docs more closely, because in Oracle I am 100% correct in that NULL is the same as empty string.

I'm trying to work through this with even less information than you have... I can't see the whole flow of code, I can't see logs and I def can't see what you have or haven't already tested.

You say you don't want a workaround, but even in your original post you said you were using a workaround (to find out later it didn't work, but still) so for me to assume you'd be ok with a temp workaround isn't exactly out of this world.

Untill someone from Retool reaches out the best you're going to get out of anybody right now is a workaround. They're swamped... always, so if they see a thread with activity on it that hasn't been closed for a while they might be more likely to pop in and see what's going on. The other option is to read you struggling, shrug my shoulders, then move on to the next post and that doesn't help anybody. so i figured whats the harm in presenting ideas of things to try, you didn't bother showing us much code so i did what i could with what i had and that's what i've been trying to present you with or at least some extra knowledge of to help find another solution. I can't go debug Retools backend and I def can't go implement fixes.

{{ (form6[indexToUpdate].data.Price_unit_Id)?
form6[indexToUpdate].data.Price_unit_Id : null}}

if the code i gave has the same problem as this, then the issue is in here (form6[indexToUpdate].data.Price_unit_Id)? because the versions I gave are un-able to return null, even the long one which is the only one with 'null' in it:

{{  (form6[indexToUpdate].data.Price_unit_Id && ['string', 'number'].includes((typeof form6[indexToUpdate].data.Price_unit_Id).toLowerCase()))? 
    String(form6[indexToUpdate].data.Price_unit_Id) : 
    form6[indexToUpdate].data.Price_unit_Id instanceof Date? 
      String(form6[indexToUpdate].data.Price_uint_Id.getTime()) :
      typeof form6[indexToUpdate].data.Price_unit_Id === 'object'?
        JSON.stringify(form6[indexToUpdate].data.Price_unit_Id) : null  }}

however, this should be un-reachable given your parameters

so instead of using (form6[indexToUpdate].data.Price_unit_Id) to check for existence try (form6?.at(indexToUpdate)?.data?.Price_unit_Id != null) or (![null, NaN, undefined].includes(form6?.at(indexToUpdate)?.data?.Price_unit_Id)

Hi Scott,

I'm sorry if I offended; it wasn't my intention.

I had a different view of the best way to get attention on an issue.

I may well be wrong, but I thought that a long thread with a lot of counter claims and comments would cloud the issue, whereas a simple clear and concise explanation of the issue would make it easier to understand.

I think the best way forward is to accept my description of the problem and hope that Retool can either explain why it has to be that way or that they plan to fix it.

I admit I did start trying with workarounds, and sometimes there is no choice but to do that if you want to keep an App working. But the reason I'm not happy using a workaround in this case is that it means I will have to use it in a large number of places, wherever my starting values could be null. I'll be sure to miss (or have already missed) some, and they will likely cause the error to occur sometime.

Once again it's very simple.
In my environment Retool turns null values in tables and forms into empty strings, which is incorrect for values of type int and Date (and likely for other types I haven't used yet) and this causes a type-mismatch error and failure of the query when writing such data back to the database.

1 Like

Now I have "egg on my face", because I've found the source of the trouble and it's easy to fix.

I did some testing and back-tracked a little....

  • Create a simple table with columns of type Int, Date and String
  • Insert one row in the table with all 3 set to Null.
  • Read the data with Select * from the table
  • Create a table component to display the data and set the column types accordingly (Int, Date and String)
  • Create 3 input components for Number, Date and Text and set them to have default values from selectedRow in the table.

Result...



The null integer is converted to 0
The null Date is converted to empty-string
The null String is converted to empty-string

If these are written back to the database...

  • The Int could be ok, except in cases where it's an int foreign keys - because it would fail the Constraint of matching the target key.
  • The string would be ok,
  • The Date will fail. (but the Date issue has already been reported previously)

So then I realised that in the case of my Integer problem I wasn't using a simple integer input component. I'm using a Select component with integer values.

The values in my Select options are from another table and are all integers.
Then if I don't select anything and leave the component blank - that's when Retool resorts to an empty-string and the write to the database fails.

Edit note: (See also next post)
I thought at first I had made a mistake and forgotten to set the default value for the Select component. But... the Select is in a Form so gets a value from the Form source (currentRow) and shouldn't need the default setting.

But anyway, setting the Select component specifically helped prevent the problem. Like this ....

Screenshot 2025-06-19 135038

(The Select box is in the expanded row of the table - hence use of currentRow)

So now with that set the null value is preserved if I just leave the Select component blank and the write to the database works fine.

So I'm sorry for the distraction and thanks to Scott for causing me to think twice and re-check the issue.

2 Likes

A short follow up to this.

I suddenly had a thought, and went back and checked the components involved. Sure enough, my thought was correct - I hadn't forgotten to set the default value of the Select component.

The reason it wasn't set previously is because all the component in the table's expanded row are inside a Form. So the Form receives the data from the currentRow and serves the data to the components by name (Form data key).

But somehow this wasn't passing the null through correctly in the Select component.

I'm not sure how setting Default value works for a form component which is already tied to a value received by the Form - but it did help get the null handled correctly.

For now I'll leave it at that.

1 Like