Vev and live content currently down
Incident Report for Vev
Postmortem

May 19, 2021: Platform and Content Downtime

The following is a detailed overview of the incident leading to downtime of the Vev platform on the 18th of May 2021. We sincerely apologize for any inconvenience this caused our users. Since the crash was reported, the Vev team has been working to ensure this does not happen again and to improve our routines when it comes to updating our users of the status of the platform.

Summary

From 22:30 on May 17th, 2021 all services connected to the vev.design domain were inaccessible. The DNS registrar (Namecheap) blocked the domain due to malicious content published by Vev users to the free hosting service (eit.vev.design). The incident was first discovered on Monday 17th May 2021 at 22:30, the root cause of the issue was not resolved until 11:38 am on 18th May 2021.

This incident does not in any way compromise security of other Vev accounts or content published through the Vev platform by other clients. The consequence of the incident was 13 hours downtime of the vev.design domain including published sites with content on the Vev CDN such as images, fonts, videos, and scripts that did not load.

Timeline

Report Description
22:30 17.05.2021 First reports of Vev crash came in
08:28 18.05.2021 Root cause was discovered and DNS provider was contacted
Every 15 min after 8:28 Talked with new Namecheap support agent
11:38 18.08.2021 DNS running as normal

Root Cause

The root cause of the incident was due to users in Vev abusing the free publishing (eit.vev.design) to create phishing sites. These types of sites break with the Acceptable Use Policy (AUP) of our domain registrar (Namecheap), and thus the domain got reported as abuse, and therefore everything connected to the domain was blocked. We have since discovered a set of users in the Vev platform using the platform to create malicious sites.

Resolution and recovery

  • As soon as we discovered the issue we contacted Namecheap support.

    • 1st line support did not have the rights to open the domain because of the abuse reports so had to wait for Namecheap Legal & Abuse Department to review the case.
    • Contacted 1st line support every 15 min after the first contact.
  • Tried helping clients in urgent need to publish project using the direct google storage URL, but required too much manual work.

  • At 11.00 we reached Namecheap Legal & Abuse Department and the blocker of the domain was removed.

Corrective and Preventative Measures

Preventative measures completed so far prevent this type of incident from happening again, as all published sites breaking with the Acceptable Use Policy (AUP) of our domain registrar (Namecheap) are taken down. Additional measures will be done consecutively to further ensure security routines:

  • Suspend all suspicious users from the platform
  • Analyze all content on eit.vev.design and delete malicious sites
  • Transfer vev.design domain to our Cloudflare enterprise account Moving our domain registrar over to Cloudflare will make it easier for us to discover issues, and since we’re enterprise clients of Cloudflare would not block our domain on this short notice.
  • Setup status page for all services in Vev with automatic reporting/subscription features This will enable us to faster discover the root cause of the issue as well as allow users to see what the status of the vev platform is. Live updates will also be shared here in the future .
  • Transfer eit.vev.design to a.vev.site This will be done as a measurement to make sure user content does not affect the authority of the vev.design domain. So if malicious sites were to be published on the free domain, the abuse will not take down our main domain name. **
  • Stricter user control We will introduce a stronger measure to validate our users, to make sure they are not using the platform with bad intentions.

    Password login will require verification

    All emails will run through an anti-spam check and not be approved

  • Automatic Safe-browsing checks of the published content We will add an automatic analyzer (Web Risk | Google Cloud ) of all links when publishing projects blocking malicious projects from being published, as well as flagging users, abusing the platform.

  • Client-hosting of Images and Videos Our Engineering and Product team will look into solutions for our users to host images and videos added to Vev projects. **

Security Status on the Vev platform

In Vev Security and Privacy is something we strive to build into our products by design and by default. So, we rely on international and recognized standards to ensure we get the best of both worlds, from design and ease of use, flexibility and security.

This incident does not in any way compromise security of other Vev accounts or content published through the Vev platform by other clients.

Posted Aug 10, 2021 - 11:57 UTC

Resolved
Platform and all staging and live content is down
Posted May 19, 2021 - 20:30 UTC