Oomnitza Incident Report - 11/30/2020

Please find the details of Oomnitza's unplanned service interruption on 11/30/2020 below. We strive to provide complete transparency, including the steps that we’re taking to prevent interruptions in the future.

Summary

On Monday, November 30th, Oomnitza experienced a service interruption that began at 6:47pm UTC (10:47am PST) and was resolved at 7:45pm UTC (11:45am PST). Users who encountered this interruption may have experienced problems that ranged from not being able to access the platform to the inability to save data in Oomnitza.

Impact

Processes such as workflows or connectors that began before the interruption were queued and processed after the issue was resolved.  No data was lost.

Cause

We performed a root cause analysis and have determined that the outage was caused by a change in DNS settings on Oomnitza’s back end. This change was intended to only affect a small number of internal processes, essentially, moving certain information behind a VPN. A failure in this process caused a backend Oomnitza service to lose access to the internet, causing the disruption. 

Corrective and Preventative Measures

Oomnitza treats disruption in service with the utmost care and concern, and we understand the pain and frustration it may cause. Thus, we’re taking the following steps to prevent issues like this in the future:

  • Infrastructure improvements that allow changes like this to be made on a smaller scale.
  • Implementation of a new testing environment to check infrastructure changes of this nature before releasing them to production.
  • To limit the scope of any potential disruptions, all changes to production infrastructure will only occur during times of minimal Oomnitza usage. 
  • Infrastructure changes will be further automated to reduce the potential for human error.

For more information, or if you have any questions or concerns, please don’t hesitate to contact support@oomnitza.com

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk