I recently deployed Retool on-prem to our staging AWS VPC to improve our deployment process. The Retool UI appears to be working as expected, however when hitting the Retool API (i.e. when trying to view resources on the UI), we get a 500 internal server error. I checked the API docker logs on the instance and we are seeing the following: Error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
I figured it may have to do with the ENCRYPTION_KEY and tried copying the ENCRYPTION_KEY from our working production Retool on-prem instance and restarting the containers, but still encounter the same 500 error.
I see this has been an issue in the past: Resources deleted on pod restart · Issue #79 · tryretool/retool-onpremise · GitHub
Please let me know what the solution may be. Thanks!
Hi @caroline! Thanks for posting. The reason the resources are not displaying is that we cannot decrypt them probably because the new pod either doesn't have a key, or it's different than before. Has the key variable been changed? And, how are you setting the key variable?
Hi @Kenny, thank you for the reply. The ENCRYPTION_KEY variable did change, but I changed it back to the key that we have on our prod on-prem Retool instance. We also took a snapshot of the prod Retool storage database and used it for the staging on-prem Retool instance, so all of the other data is the same. So the staging on-prem Retool instance is essentially a copy of the working prod Retool instance, just in a different environment and a newer version of Retool.
I am setting the ENCRYPTION_KEY variable in docker.env. When I changed the ENCRYPTION_KEY back to the same one on our production Retool instance, I restarted the docker containers. We still experienced the 500 errors.
Let me know if there is any other information I can provide. Thanks!
Hey @caroline, thanks for sharing that. Could you share with us if this affecting only your Dev or Prod instance (or both)? Are you also able to share a timeline of when you changed the ENCRYPTION_KEY? My understanding is that your initial ENCRYPTION_KEY didn't work so you tested this out with your Prod ENCRYPTION_KEY (which works), am I understanding this correctly?
We are facing the same issue and we tried to use latest helm chart and passed the encryption key but no luck. Any suggestion ?
{"pid":55,"requestId":"ce298861-67fa-4e7a-b1d1-f140f3f537fd","user":{"email":"","sid":"user_0901cf44d2eb45a4b6eef346de8578ff"},"organization":{"id":1,"name":""},"message":{"type":"UNCAUGHT_ERROR","stack":"Error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt\n at Decipher.final (internal/crypto/cipher.js:172:29)\n at decryptCBC (/snapshot/retool_development/backend/transpiled/common/encryption.js)\n at decrypt (/snapshot/retool_development/backend/transpiled/common/encryption.js)\n at Resource.databasePassword (/snapshot/retool_development/backend/transpiled/server/orm/models/resource.js)\n at Resource.get (/snapshot/retool_development/node_modules/sequelize/lib/model.js:3513:41)\n at Resource.get (/snapshot/retool_development/node_modules/sequelize/lib/model.js:3547:33)\n at Resource.toJSONHelper (/snapshot/retool_development/backend/transpiled/server/orm/models/resource.js)\n at Resource.toJSON (/snapshot/retool_development/backend/transpiled/server/orm/models/resource.js)\n at /snapshot/retool_development/backend/transpiled/server/modules/permissions/getPermissions.js\n at tryCatcher (/snapshot/retool_development/node_modules/bluebird/js/release/util.js:16:23)\n at MappingPromiseArray._promiseFulfilled (/snapshot/retool_development/node_modules/bluebird/js/release/map.js:68:38)\n at MappingPromiseArray. (/snapshot/retool_development/node_modules/bluebird/js/release/promise_array.js:115:31)\n at MappingPromiseArray.init (/snapshot/retool_development/node_modules/bluebird/js/release/promise_array.js:79:10)\n at Promise._settlePromise (/snapshot/retool_development/node_modules/bluebird/js/release/promise.js:601:21)\n at Promise._settlePromise0 (/snapshot/retool_development/node_modules/bluebird/js/release/promise.js:649:10)\n at Promise._settlePromises (/snapshot/retool_development/node_modules/bluebird/js/release/promise.js:729:18)\n at _drainQueueStep (/snapshot/retool_development/node_modules/bluebird/js/release/async.js:93:12)\n at _drainQueue (/snapshot/retool_development/node_modules/bluebird/js/release/async.js:86:9)\n at Async._drainQueues (/snapshot/retool_development/node_modules/bluebird/js/release/async.js:102:5)\n at Immediate._onImmediate (/snapshot/retool_development/node_modules/bluebird/js/release/async.js:15:14)\n at processImmediate (internal/timers.js:461:21)"},"level":"error","timestamp":"2022-07-25T06:37:04.601Z"}
Hey @sampathk Looks like you’re currently working with us over chat to find a solution, but for anyone else running into this, we believe you’re currently running into a bug where we define a random string for an encryption key if the variable does not exist or was incorrectly input.
This bug has now been resolved and all you have to do is upgrade your Helm Chart! Hope this helps
