CDP issue
Incident Report for Ortto

This morning at approximately 6:30am Sydney time we noticed that the CDP was performing slowly. Additionally, our system alerted that several of the nodes responsible for the CDP’s operation were getting OOM (out of memory) errors. This was causing People, Audiences, Filters, Reports, Dashboards to load slowly or not at all for a portion of our customers.

On investigation, it was found that this was due to the way our Retention+ audiences were being calculated. To remediate this, we have disabled the operations which caused the out of memory errors.

Two follows ups will be undertaken following this incident which will prevent any issues of this type occurring in the future:

  1. We will be changing the way we load data for large reports to prevent future memory issues.
  2. We will be adding limits to the number of Segment activities per day will be processed for reports.

We expect these changes to be live within 14 days of this notice, however additionally we do not expect to see this addition reoccur during that time due to the other precautions we have taken.

Posted Nov 10, 2022 - 23:40 UTC

We have resolved the issue. We are also working to fix problems like this in the future.
Posted Nov 10, 2022 - 21:10 UTC
We have identified an issue with the CDP being slow to load data or failing completely. There is no risk of data loss, but you may experience issues accessing the "People" and "Audiences" sections of Ortto, and also issues with reports and filters.

We have identified the issue and are working urgently on a fix. We will update this page as soon as the issue is resolved or there is an update.
Posted Nov 10, 2022 - 20:30 UTC
This incident affected: Ortto application and Ortto API.