at 17:47 CET time, an alert is received concerning a timeout in the execution of some of our database requests.
at 17h54 CET time, the incident is resolved. The root cause came from the deployment of our software version and the related modification of database schema. During this modification process, a query was stuck during 30 seconds (statement_timeout setting of our databases), and the action was retried 4 times. We were not able to find the reason why the query was stuck and the deployment on our staging environment didn't encounter this issue.
Following these incidents, the following actions are decided:
we will not use anymore the helper function that sends this specific database query. We have other options to achieve the same goal.
Posted Jun 14, 2023 - 15:14 UTC
Resolved
The update of the production database caused some requests to fail from 3:46 PM UTC to 3:54 PM UTC