Invalid SessionHandle when querying via Databricks JDBC connection

aaronmorais · January 15, 2024, 7:51pm

Goal: We're building an admin panel with data from multiple tables in Databricks. Via a Datbricks JDBC connection, we perform one initial query followed by six queries that use the result of the first. Intermittently, we encounter 'Invalid SessionHandle' errors when performing these queries. Is there something that needs to be configured on our end to make this more reliable?
Details:
The error message we see intermittently is as follows:

Error running instance method java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, infoMessages:[*org.apache.hive.service.cli.HiveSQLException:Invalid SessionHandle: SessionHandle [1786667d-500b-4bb1-a15c-38ac72cd194b]:68:67, org.apache.hive.service.cli.session.SessionManager:getSession:SessionManager.java:350, org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager:getSession:SparkSQLSessionManager.scala:204, com.databricks.sql.hive.thriftserver.thrift.ThriftHandlerUtils$:getSessionHandle:ThriftHandlerUtils.scala:60, com.databricks.sql.hive.thriftserver.thrift.SetSessionHandleProtocolVersionThriftHandler:ExecuteStatement:SetSessionHandleProtocolVersionThriftHandler.scala:49, com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63, com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62, com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:super$ExecuteStatement:ThreadLocalContextThriftHandler.scala:73, com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:$anonfun$ExecuteStatement$1:ThreadLocalContextThriftHandler.scala:73, com.databricks.unity.UCSEphemeralState$Handle:runWith:UCSEphemeralState.scala:41, com.databricks.unity.HandleImpl:runWith:UCSHandle.scala:99, com.databricks.unity.HandleImpl:$anonfun$runWithAndClose$1:UCSHandle.scala:104, scala.util.Using$:resource:Using.scala:269, com.databricks.unity.HandleImpl:runWithAndClose:UCSHandle.scala:103, com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:createThriftThreadLocalContext:ThreadLocalContextThriftHandler.scala:47, com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:ExecuteStatement:ThreadLocalContextThriftHandler.scala:73, com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63, com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62, com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:super$ExecuteStatement:ErrorPropagationThriftHandler.scala:73, com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:$anonfun$ExecuteStatement$1:ErrorPropagationThriftHandler.scala:73, com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:runSafely:ErrorPropagationThriftHandler.scala:119, com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:ExecuteStatement:ErrorPropagationThriftHandler.scala:73, org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:431, org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1437, org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1422, org.apache.thrift.ProcessFunction:process:ProcessFunction.java:38, org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39, org.apache.hive.service.cli.thrift.ThriftHttpServlet:handleThriftRequest:ThriftHttpServlet.java:186, org.apache.hive.service.cli.thrift.ThriftHttpServlet:acceptRequest:ThriftHttpServlet.java:168, org.apache.hive.service.cli.thrift.ThriftHttpServlet:doPostInternal:ThriftHttpServlet.java:141, org.apache.hive.service.cli.thrift.ThriftHttpServlet:doPost:ThriftHttpServlet.java:86, javax.servlet.http.HttpServlet:service:HttpServlet.java:523, javax.servlet.http.HttpServlet:service:HttpServlet.java:590, org.eclipse.jetty.servlet.ServletHolder:handle:ServletHolder.java:799, org.eclipse.jetty.servlet.ServletHandler:doHandle:ServletHandler.java:554, org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:143, org.eclipse.jetty.security.SecurityHandler:handle:SecurityHandler.java:600, org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127, org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:235, org.eclipse.jetty.server.session.SessionHandler:doHandle:SessionHandler.java:1624, org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:233, org.eclipse.jetty.server.handler.ContextHandler:doHandle:ContextHandler.java:1440, org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:188, org.eclipse.jetty.servlet.ServletHandler:doScope:ServletHandler.java:505, org.eclipse.jetty.server.session.SessionHandler:doScope:SessionHandler.java:1594, org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:186, org.eclipse.jetty.server.handler.ContextHandler:doScope:ContextHandler.java:1355, org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:141, org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127, org.eclipse.jetty.server.Server:handle:Server.java:516, org.eclipse.jetty.server.HttpChannel:lambda$handle$1:HttpChannel.java:487, org.eclipse.jetty.server.HttpChannel:dispatch:HttpChannel.java:732, org.eclipse.jetty.server.HttpChannel:handle:HttpChannel.java:479, org.eclipse.jetty.server.HttpConnection:onFillable:HttpConnection.java:277, org.eclipse.jetty.io.AbstractConnection$ReadCallback:succeeded:AbstractConnection.java:311, org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105, org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint:onFillable:SslConnection.java:555, org.eclipse.jetty.io.ssl.SslConnection:onFillable:SslConnection.java:410, org.eclipse.jetty.io.ssl.SslConnection$2:succeeded:SslConnection.java:164, org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105, org.eclipse.jetty.io.ChannelEndPoint$1:run:ChannelEndPoint.java:104, org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:runTask:EatWhatYouKill.java:338, org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:doProduce:EatWhatYouKill.java:315, org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:tryProduce:EatWhatYouKill.java:173, org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:run:EatWhatYouKill.java:131, org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread:run:ReservedThreadExecutor.java:409, java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149, java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624, java.lang.Thread:run:Thread.java:750], sqlState:08000, errorCode:0, errorMessage:Invalid SessionHandle: SessionHandle [1786667d-500b-4bb1-a15c-38ac72cd194b]), Query: ***redacted query***. at com.databricks.client.hivecommon.api.HS2Client.buildExceptionFromTStatusSqlState(Unknown Source) at com.databricks.client.hivecommon.api.HS2Client.buildExceptionFromTStatus(Unknown Source) at com.databricks.client.hivecommon.api.HS2Client.checkResponseStatus(Unknown Source) at com.databricks.client.hivecommon.api.HS2Client.executeStatementInternal(Unknown Source) at com.databricks.client.hivecommon.api.HS2Client.executeStatement(Unknown Source) at com.databricks.client.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.executeNonRowCountQueryHelper(Unknown Source) at com.databricks.client.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.executeQuery(Unknown Source) at com.databricks.client.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.<init>(Unknown Source) at com.databricks.client.hivecommon.dataengine.HiveJDBCDataEngine.prepare(Unknown Source) at com.databricks.client.jdbc.common.SPreparedStatement.<init>(Unknown Source) at com.databricks.client.jdbc.jdbc41.S41PreparedStatement.<init>(Unknown Source) at com.databricks.client.jdbc.jdbc42.S42PreparedStatement.<init>(Unknown Source) at com.databricks.client.hivecommon.jdbc42.Hive42PreparedStatement.<init>(Unknown Source) at com.databricks.client.spark.jdbc.SparkJDBCObjectFactory.createPreparedStatement(Unknown Source) at com.databricks.client.jdbc.common.JDBCObjectFactory.newPreparedStatement(Unknown Source) at com.databricks.client.jdbc.common.SConnection$5.create(Unknown Source) at com.databricks.client.jdbc.common.SConnection$5.create(Unknown Source) at com.databricks.client.jdbc.common.SConnection$StatementCreator.create(Unknown Source) at com.databricks.client.jdbc.common.SConnection.prepareStatement(Unknown Source) at com.databricks.client.jdbc.common.SConnection.prepareStatement(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) Caused by: com.databricks.client.support.exceptions.ErrorException: [Databricks][DatabricksJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, infoMessages:[*org.apache.hive.service.cli.HiveSQLException:Invalid SessionHandle: SessionHandle [1786667d-500b-4bb1-a15c-38ac72cd194b]:68:67, org.apache.hive.service.cli.session.SessionManager:getSession:SessionManager.java:350, org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager:getSession:SparkSQLSessionManager.scala:204, com.databricks.sql.hive.thriftserver.thrift.ThriftHandlerUtils$:getSessionHandle:ThriftHandlerUtils.scala:60, com.databricks.sql.hive.thriftserver.thrift.SetSessionHandleProtocolVersionThriftHandler:ExecuteStatement:SetSessionHandleProtocolVersionThriftHandler.scala:49, com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63, com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62, com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:super$ExecuteStatement:ThreadLocalContextThriftHandler.scala:73, com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:$anonfun$ExecuteStatement$1:ThreadLocalContextThriftHandler.scala:73, com.databricks.unity.UCSEphemeralState$Handle:runWith:UCSEphemeralState.scala:41, com.databricks.unity.HandleImpl:runWith:UCSHandle.scala:99, com.databricks.unity.HandleImpl:$anonfun$runWithAndClose$1:UCSHandle.scala:104, scala.util.Using$:resource:Using.scala:269, com.databricks.unity.HandleImpl:runWithAndClose:UCSHandle.scala:103, com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:createThriftThreadLocalContext:ThreadLocalContextThriftHandler.scala:47, com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:ExecuteStatement:ThreadLocalContextThriftHandler.scala:73, com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63, com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62, com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:super$ExecuteStatement:ErrorPropagationThriftHandler.scala:73, com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:$anonfun$ExecuteStatement$1:ErrorPropagationThriftHandler.scala:73, com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:runSafely:ErrorPropagationThriftHandler.scala:119, com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:ExecuteStatement:ErrorPropagationThriftHandler.scala:73, org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:431, org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1437, org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1422, org.apache.thrift.ProcessFunction:process:ProcessFunction.java:38, org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39, org.apache.hive.service.cli.thrift.ThriftHttpServlet:handleThriftRequest:ThriftHttpServlet.java:186, org.apache.hive.service.cli.thrift.ThriftHttpServlet:acceptRequest:ThriftHttpServlet.java:168, org.apache.hive.service.cli.thrift.ThriftHttpServlet:doPostInternal:ThriftHttpServlet.java:141, org.apache.hive.service.cli.thrift.ThriftHttpServlet:doPost:ThriftHttpServlet.java:86, javax.servlet.http.HttpServlet:service:HttpServlet.java:523, javax.servlet.http.HttpServlet:service:HttpServlet.java:590, org.eclipse.jetty.servlet.ServletHolder:handle:ServletHolder.java:799, org.eclipse.jetty.servlet.ServletHandler:doHandle:ServletHandler.java:554, org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:143, org.eclipse.jetty.security.SecurityHandler:handle:SecurityHandler.java:600, org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127, org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:235, org.eclipse.jetty.server.session.SessionHandler:doHandle:SessionHandler.java:1624, org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:233, org.eclipse.jetty.server.handler.ContextHandler:doHandle:ContextHandler.java:1440, org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:188, org.eclipse.jetty.servlet.ServletHandler:doScope:ServletHandler.java:505, org.eclipse.jetty.server.session.SessionHandler:doScope:SessionHandler.java:1594, org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:186, org.eclipse.jetty.server.handler.ContextHandler:doScope:ContextHandler.java:1355, org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:141, org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127, org.eclipse.jetty.server.Server:handle:Server.java:516, org.eclipse.jetty.server.HttpChannel:lambda$handle$1:HttpChannel.java:487, org.eclipse.jetty.server.HttpChannel:dispatch:HttpChannel.java:732, org.eclipse.jetty.server.HttpChannel:handle:HttpChannel.java:479, org.eclipse.jetty.server.HttpConnection:onFillable:HttpConnection.java:277, org.eclipse.jetty.io.AbstractConnection$ReadCallback:succeeded:AbstractConnection.java:311, org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105, org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint:onFillable:SslConnection.java:555, org.eclipse.jetty.io.ssl.SslConnection:onFillable:SslConnection.java:410, org.eclipse.jetty.io.ssl.SslConnection$2:succeeded:SslConnection.java:164, org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105, org.eclipse.jetty.io.ChannelEndPoint$1:run:ChannelEndPoint.java:104, org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:runTask:EatWhatYouKill.java:338, org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:doProduce:EatWhatYouKill.java:315, org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:tryProduce:EatWhatYouKill.java:173, org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:run:EatWhatYouKill.java:131, org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread:run:ReservedThreadExecutor.java:409, java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149, java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624, java.lang.Thread:run:Thread.java:750], sqlState:08000, errorCode:0, errorMessage:Invalid SessionHandle: SessionHandle [1786667d-500b-4bb1-a15c-38ac72cd194b]), Query: ***redacted query***