Leyra - AppGrid /metadata response anomalies – Incident details

On Sept 14, 04:19 CEST, an incident was reported regarding a possible issue with one API endpoint returning a '500 Internal Server Error' on a limited but random set of occassions. This was investigated thoroughly and was reproducible on our testing environment only under very specific circumstances. The root cause of the issue was narrowed down to an error in an industry-standard third party library that AppGrid and its API are dependent on to function. A service pack (2.9.6) containing an update of that library was released on Oct 3, 16:39 CEST that fixed the issue, but had an unforeseen side effect on another, separate part of the system. This issue caused a limited set of calls to the /asset endpoint to return URLs resolving to missing files. A rollback to the 2.9.5 service pack was performed within an hour on Oct 3, 17:42 CEST to mitigate the issue. The root cause of the issue with asset URLs was quickly identified and the 2.9.6 service pack was re-deployed on Oct 4, 12:24 CEST including solutions to both issues. On Oct 4, 22:20 CEST, a few customers reported that calls to /metadata were sometimes returning incorrect values for number fields that were stored as decimals. The investigation started immediately but was made difficult due to the number of factors involved in reproducing the error and the limited number of customers affected. On Oct 6, 10:36 CEST, the 2.9.7 service pack was deployed which fixed the issue. In the light of the above incidents, we have taken several actions in order to prevent this from happening in the future, improving our automated backward compatibility- and load tests. Going forward, we will also be thoroughly investigating alternatives to the the dependency library that caused the issues.

This incident has been resolved.

The fix has now been deployed to production. We will closely monitor the behaviour of the API during the day and will provide a post-mortem analysis shortly.

The root cause of the issue has been identified and a fix is being tested right now. The estimated deployment time is 11:00 CEST.