Error on connection with Databricks

[deipzza]

Hi hi,

I've been using Databricks as a resource for one of the apps of my company and it has been working well for months but recently every query fails with the same error message:

Error running instance method java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500540) Error caught in BackgroundFetcher. Foreground thread ID: 1. Background thread ID: 53. Error caught: Could not initialize class com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil. at com.databricks.client.hivecommon.dataengine.BackgroundFetcher.run(Unknown Source) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) Caused by: com.databricks.client.support.exceptions.GeneralException: [Databricks][DatabricksJDBCDriver](500540) Error caught in BackgroundFetcher. Foreground thread ID: 1. Background thread ID: 53. Error caught: Could not initialize class com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil. ... 5 more Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil at com.databricks.client.jdbc42.internal.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:228) at com.databricks.client.jdbc42.internal.apache.arrow.memory.ArrowBuf.nioBuffer(ArrowBuf.java:223) at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:87) at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:727) at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:363) at com.databricks.client.spark.arrow.ArrowBuffer.deserializeBatch(Unknown Source) at com.databricks.client.spark.arrow.ArrowBuffer.handleInitializeBuffer(Unknown Source) at com.databricks.client.hivecommon.api.HiveServer2BaseBuffer.initializeBuffer(Unknown Source) at com.databricks.client.hivecommon.api.RowsetBuffer.initializeBuffer(Unknown Source) at com.databricks.client.hivecommon.api.HS2Client.getRowSetInformation(Unknown Source) at com.databricks.client.hivecommon.api.HS2Client.fetchFromServer(Unknown Source) at com.databricks.client.spark.jdbc.DowloadableFetchClient.fetchNRows(Unknown Source) at com.databricks.client.hivecommon.api.HS2Client.fetchRows(Unknown Source) at com.databricks.client.hivecommon.dataengine.BackgroundFetcher.run(Unknown Source) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.RuntimeException: Failed to initialize MemoryUtil. [in thread "pool-2-thread-1"] at com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:136) ... 19 more

Searching a bit with my team we found that it may be related to the version of the JDBC used to connect Databricks with Retool, we don't know if this was upgraded recently.

Here's a screenshot of the error within our app:

Thanks in advance.

[Tess]

Hi @deipzza,

Thanks for reaching out! I am looking into this

Is this for your Cloud hosted Retool Business account? Do you get the same error when running test connection on the resource page?

I believe the version is 2.6.36

[JulianJohansen]

Any update on this? We are too seeing this issue.

[Tess]

Hi @JulianJohansen,

I haven't been able to reproduce the error yet. Are you experiencing this on Cloud or self hosted? What do you see when you run test connection?

Could you share a screenshot of the query & error that you're seeing?

[JulianJohansen]

Hi @Tess ,

It is on cloud. It is essentially any query. A basic select * from a table returns the same error. We can tell on Databricks side that it does post and successfully run the query but evidently not able to return the query result.

Clicking the "Test Connection" on the resource goes fine:

The error we get in retool cloud:

but as mentioned it does post it to our databricks environment and runs the query.

[JulianJohansen]

A colleague pointed out to me that i should share that our databricks query shows: 2.6.32

whereas i see in your post earlier mentioned 2.6.36

[Tess]

Thanks, @JulianJohansen It looks like your screenshot of the query in Retool is a test query, so I just want to confirm, are you getting the same error when querying this resource in an app or Workflow? I think I may have found a fix, but need to double check internally. I will keep you posted with any updates/follow up questions

[Tess]

We should have a fix shipping next week I'll check in when it goes live, but wanted to share the update

[JulianJohansen]

Hi @Tess
For some reason i never got a mail notification of reply to the thread so i just randomly checked today.
Thanks for looking into it.
My picture was just to show that it is ANY query that i run and wanted to share a picture of a very simple SQL query to show that it was not a syntax error.

Happy to hear that a fix is in the making. Is it this week?

[Tess]

Yes we have a fix that is going out this week (tomorrow I believe)

[JulianJohansen]

@Tess Still seems to be same issue ;/

[deipzza]

Hi, sorry I haven't been replying even though it's my post, I have the same issue about the emails, so I thought there wasn't any replies.

As @JulianJohansen I'm still experiencing issues with the queries. @Tess any updates on this? Thank you in advance.

[kbn]

@JulianJohansen @deipzza Hey folks! Thanks for letting us know -- replying here while @Tess is out of office: her estimate was accurate if our normal cadence of releases went as planned. However this week in particular our eng team caught some issues in the 3.88 release that needed to get fully resolved before pushing it out to customers, so the release got delayed.

Right now, if you're on Retool Cloud, it is still on version 3.87. You can check this at the bottom right of the app editor if you click on the ? question mark icon:

Once that goes to 3.88 or above, we expect this error with Databricks connections to be resolved. So currently, it's expected that it's not yet fixed despite our initial timeline estimate. Thank you so much for your patience. The fix should be just around the corner!

Also: thanks for the notes about not getting an email with a reply notification We will monitor whether anything's changed or looks strange in our forum settings related to this. But, one thing you could try is making sure that you're marked as "Watching" this topic via the dropdown at the bottom, which should be the case when you've created or replied to that topic:

[deipzza]

Hey @kbn! My Retool says it's now on version 3.89 and the queries are working fine . @JulianJohansen you can test it too to see if it's all good and mark this as resolved.

Thank you all!

[JulianJohansen]

It works! Success

[Tess]

Thank you both for working with us on troubleshooting! So glad to hear