Makeswift's Outage on May 19, 2025

Incident Report



On May 19th, the Makeswift API experienced a 57-minute outage. During this time, all services dependent on the API—including the builder and customer websites—were unable to process requests. Sites without caching mechanisms in place were most affected. This was a severe outage, as it directly impacted our customers’ websites.



We sincerely apologize to our customers for this incident. As your website provider, we recognize that Makeswift's reliability is crucial to your business operations. We are implementing additional safeguards to prevent similar disruptions in the future.





Incident Response Timeline



• 10:22 AM: Our API monitoring system detected that app.makeswift.com was unresponsive.
• 10:27 AM: We declared an incident. Our engineering team began investigating.
• 10:50 AM: We had identified the root cause and started developing a fix.
• 11:07 AM: We deployed the fix to our staging environment for testing.
• 11:19 AM: We deployed to production, and the API returned to full operational status.



Service was down for approximately 57 minutes.





What Happened?



The error originated from our presence WebSocket service, where invalid message parsing caused the service to crash in certain edge cases. The situation was compounded by the app continuously attempting to reconnect without proper retry limits, causing API instances to crash in a cascading manner as they came back online.





Short Term Resolution



• We fixed the bug in question by updating the message parsing to safely handle errors.
• We modified our continuous delivery pipeline to allow direct deployment to production when we're confident in a change, bypassing staging. This improvement will allow us to reduce incident response time by 10–15 minutes.





Long Term Mitigation



• We will increase our investment into observability and testing so that issues like these are both shorter and less likely.
• We will implement incremental backoff delays in our presence WebSocket service to prevent cascading failures if similar bugs occur.



Once again, we apologize for the impact this had on your editing capabilities, as well as your live website. If you have any questions or concerns, please reach out to us at support@makeswift.com.