Goal: Surface database queries that are failing on the user end ie: parameters not filled in by the time a user is triggering the query or database outages and short timeout windows.
Steps: I am on the business plan, cloud hosted so I have access to the audit trail which is helpful but ive yet to identity where in the metadata I can reliably determine which QUERY_RUNS are actually errors because the metadata.query.error value is seemingly always null. RESTful api request have metadata.error and metadata.isError fields so that works great.
Details: I am downloading my audit logs from retool and dumping them into a GCP storage bucket, then using CloudStorage>>>BigLake BigQuery external table>>>Retool BigQuery Resource to query this data back into an app since retool doesn't allow cloud hosted users to access this data through its own platform (very annoying).
Any help on how one might identify these errors would be amazing.
I'm with you that there should be an easier way for Business users to access this.
What I have ended up setting up each query so that on failure they trigger a workflow. This sends all the info available in the query's object.
The workflow saves the data in a database, and sends me an email informing me of user, time, app and query that failed, including the error message. This allows me to intervene fast, and know where to intervene.
I know this is not the ideal as it requires initially to go through each query and add a failure event handler, but once this is done, you just need to make the habit of adding it every time you create a query.
This also provides for more flexibility as you can add actions to your workflow, assign errors to developers and other things.
Yes, this is the general design I used before switching to a Business subscription.
Here's an example:
Create a module with no elements in it, just a onChange event listener that inserts data to your log. You could extend this to run a workflow instead of just submitting it to a database.
In all of your apps, for all of your queries, create an onError event handler that sets a value for variable1, this variable is passed to the module included in your app as in input. Each time this variable is changed this triggers the module to execute the onchange query. This way you don't have to recreate the same query and have centralized place where you can edit the query if you need to change the destination of the log info.
+1 on this from me as well. Like both of you, I have MacGuyver'ed some workarounds. I have also half heartedly skimmed Sentry and DataDog as possible solutions, but would love a more direct, in-tool solution.
