Impact Start Time (UTC) - 05/05/2025 03:23 PM (UTC)
Impact End Time (UTC) - 05/05/2025 04:10 PM (UTC)
Incident Summary
Updated on 05/12/2025 - On 05/05/2025, some CXone Mpower customers experienced latency issues and encountered "503 Service Unavailable" errors while accessing the CXone Mpower Expert application. The impact was caused by a significant increase in requests to an application responsible for processing Application Programming Interface (API) requests for data, which placed extra strain on the system, leading to slow response times and some requests timing out. The impact was resolved when the system processed the backlog of requests, automatically restoring normal operation.
Root Cause
The impact was caused by a significant increase in requests from the web service to an application responsible for processing Application Programming Interface (API) requests for data. This surge strained the system, resulting in slow response times and timeouts due to bottlenecks, and made web resources insufficient to handle the surge efficiently. Additionally, the influx of requests led to a slight slowness in database performance which further exacerbated the system resources.
Corrective Actions
Detection:
Internal teams received reports from customers that they experienced latency issues and encountered "503 Service Unavailable" errors while accessing the CXone Mpower Expert application.
Remediation:
The impact was resolved when the system processed the backlog of requests, automatically restoring normal operation. Completed on 05/05/2025.
Prevention:
Engineers will implement a system enhancement to improve the database queries, allowing more concurrent connections, improving overall database performance, and response times. An update will be provided by End of Day (EOD) MT on 05/19/2025.
Incident Timeline (UTC)
05/05/2025 03:23 PM (UTC) - The first customer case was opened, and Tech Support (TS) engineers began their initial validations and troubleshooting investigation.
05/05/2025 03:37 PM (UTC) - TS engineers notified the Network Operations Center (NOC) engineers about the reported customer impact; a major incident was proposed and confirmed.
05/05/2025 03:59 PM (UTC) - Engineers identified a suspected cause and are working on remediation steps.
05/05/2025 04:10 PM (UTC) - The impact was resolved when engineers observed that the system automatically recovered after it was able to process the backlog of requests. Following successful test validations, the major incident was marked as resolved.