This morning at approximately 6:30am Sydney time we noticed that the CDP was performing slowly. Additionally, our system alerted that several of the nodes responsible for the CDP’s operation were getting OOM (out of memory) errors. This was causing People, Audiences, Filters, Reports, Dashboards to load slowly or not at all for a portion of our customers.

On investigation, it was found that this was due to the way our Retention+ audiences were being calculated. To remediate this, we have disabled the operations which caused the out of memory errors.

Two follows ups will be undertaken following this incident which will prevent any issues of this type occurring in the future:

1. We will be changing the way we load data for large reports to prevent future memory issues.
2. We will be adding limits to the number of Segment activities per day will be processed for reports.

We expect these changes to be live within 14 days of this notice, however additionally we do not expect to see this addition reoccur during that time due to the other precautions we have taken.