May 20, 2021: Google Firebase Brought Down Access to the Editor

Summary

Around 09:47 our system detected the Vev editor was not loading. We discovered this issue was related to Google Firebase routing rules not working as expected, this caused the users not to be able to open the editor.

At the time of writing this report, the issue has still not been resolved by Google, but we have deployed a workaround to make the editor run.

Timeline

Reports	Description
`09:47 20.05.2021`	`Issue was detected`
`10:00 20.05.2021`	`Root cause of issue was discoverd`
`10:45 20.05.2021`	`Workarround to the firebase rules was deploy`

Root Cause

The root cause of the incident was a Google Firebase router breaking. This was a widespread issue for all applications using Google Firebase Hosting, utilizing the hosting rewrite rules

Resolution and recovery

First, we tried to replace scripts loaded from the __ folder with full URL to [https://vev-prod.firebaseapp.com/__/](https://vev-prod.firebaseapp.com/__) which was working at the time
- This failed since these scripts disappeared as well
Changed all scripts loaded from __ to be cdnjs script or loaded locally

Corrective and Preventative Measures

Review all scripts not included in the editor bundle if it advantageous to include them

Posted Aug 10, 2021 - 13:07 UTC

Resolved

An issue with Google Firebase has bought down access to the Editors, the platform is currently inaccessible. Stay tuned for updates.

Posted May 20, 2021 - 13:04 UTC