This morning at approximately 6:30am Sydney time we noticed that the CDP was performing slowly. Additionally, our system alerted that several of the nodes responsible for the CDP’s operation were getting OOM (out of memory) errors. This was causing People, Audiences, Filters, Reports, Dashboards to load slowly or not at all for a portion of our customers.
On investigation, it was found that this was due to the way our Retention+ audiences were being calculated. To remediate this, we have disabled the operations which caused the out of memory errors.
Two follows ups will be undertaken following this incident which will prevent any issues of this type occurring in the future:
We expect these changes to be live within 14 days of this notice, however additionally we do not expect to see this addition reoccur during that time due to the other precautions we have taken.