On September 25, 2017 an issue was reported to our Support desk regarding general system slowness and inaccessibility due to degraded performance. The following is the incident report for the etouches issue related to "Registration attendee performance issues for Max Event Totals" that occurred for 2 hours during September 25, 2017. We understand this service issue has impacted our valued customers, and we apologize to everyone who was affected.
From 11:57 AM to 1:50 PM ET, customers experienced a decrease in performance using the etouches platform and occasional outages due to the performance decrease. Since the impact of performance was dependent on server load based on the use of an application feature designed to monitor and enforce configurable event maximum registration limits, the performance impact and outages appeared random in nature to our end users.
All customers in our North America region would have been able to perceive the decrease in system performance and/or inaccessibility. Events that were configured to use the Maximum Event Registration functionality were particularly susceptible to the performance impact and are likely to have experienced a resulting outage.
On September 23rd, a Hot Fix for an unrelated issue was deployed to Production that affected the performance of the Max Event Registrations logic. The fix caused the interrogation of all Registration Attendee information in conjunction with the Max Event Registrations functionality thus causing performance impacts due to server load and increasing database queues.
Once the issue was escalated and identified by the Development team, the code that caused the Registration Attendee information to be interrogated was removed while still maintaining the functionality addressed by the offending Hot Fix as well as the Max Event Registrations feature. Both pieces of functionality were tested and verified before deploying to the Production platform.
In the last two days, we’ve conducted an internal review and analysis of the outage. The following are actions we are taking to address the underlying causes of the issue and to help prevent recurrence and improve response times: