The following is a detailed overview of the incident leading to downtime of the Vev platform on the 18th of May 2021. We sincerely apologize for any inconvenience this caused our users. Since the crash was reported, the Vev team has been working to ensure this does not happen again and to improve our routines when it comes to updating our users of the status of the platform.
From 22:30 on May 17th, 2021 all services connected to the vev.design domain were inaccessible. The DNS registrar (Namecheap) blocked the domain due to malicious content published by Vev users to the free hosting service (eit.vev.design). The incident was first discovered on Monday 17th May 2021 at 22:30, the root cause of the issue was not resolved until 11:38 am on 18th May 2021.
This incident does not in any way compromise security of other Vev accounts or content published through the Vev platform by other clients. The consequence of the incident was 13 hours downtime of the vev.design domain including published sites with content on the Vev CDN such as images, fonts, videos, and scripts that did not load.
| Report | Description |
|---|---|
22:30 17.05.2021 |
First reports of Vev crash came in |
08:28 18.05.2021 |
Root cause was discovered and DNS provider was contacted |
Every 15 min after 8:28 |
Talked with new Namecheap support agent |
11:38 18.08.2021 |
DNS running as normal |
The root cause of the incident was due to users in Vev abusing the free publishing (eit.vev.design) to create phishing sites. These types of sites break with the Acceptable Use Policy (AUP) of our domain registrar (Namecheap), and thus the domain got reported as abuse, and therefore everything connected to the domain was blocked. We have since discovered a set of users in the Vev platform using the platform to create malicious sites.
As soon as we discovered the issue we contacted Namecheap support.
Tried helping clients in urgent need to publish project using the direct google storage URL, but required too much manual work.
At 11.00 we reached Namecheap Legal & Abuse Department and the blocker of the domain was removed.
Preventative measures completed so far prevent this type of incident from happening again, as all published sites breaking with the Acceptable Use Policy (AUP) of our domain registrar (Namecheap) are taken down. Additional measures will be done consecutively to further ensure security routines:
Stricter user control We will introduce a stronger measure to validate our users, to make sure they are not using the platform with bad intentions.
Password login will require verification
All emails will run through an anti-spam check and not be approved
Automatic Safe-browsing checks of the published content We will add an automatic analyzer (Web Risk | Google Cloud ) of all links when publishing projects blocking malicious projects from being published, as well as flagging users, abusing the platform.
Client-hosting of Images and Videos Our Engineering and Product team will look into solutions for our users to host images and videos added to Vev projects. **
In Vev Security and Privacy is something we strive to build into our products by design and by default. So, we rely on international and recognized standards to ensure we get the best of both worlds, from design and ease of use, flexibility and security.
This incident does not in any way compromise security of other Vev accounts or content published through the Vev platform by other clients.