Major Incident# 02793015
Impact Start Time (UTC) 05/07/2026 11:43 AM UTC
Impact End Time (UTC) 05/07/2026 02:07 PM UTC
On 05/07/2026, some NiCE CXone Mpower customers experienced slowness when accessing sites, while others were unable to access the platform entirely with a “504 Gateway Timeout” error within the CXone Mpower Expert knowledge portal. The recurring service degradation incidents were caused by increased traffic volumes combined with performance limitations in certain backend processes, primarily impacting the US regional platform due to higher demand. The impact was resolved after scaling up pod resources and restarting the proxy pods, which restored platform stability.
The recurring service degradation incidents were caused by increased traffic volumes combined with performance limitations in certain backend processes, primarily impacting the US regional platform due to higher demand. Under elevated traffic conditions, including automated crawler activity, some requests followed less optimized processing paths, increasing system load. This was further amplified by legacy or complex page content requiring more intensive processing. While scaling actions helped restore capacity, they also introduced temporary overhead that contributed to intermittent performance degradation. Additionally, although autoscaling functioned as designed, it reached its limits and was insufficient to address constraints related to per-pod Central Processing Unit (CPU) capacity. Overall, evolving traffic patterns exposed underlying performance limitations, highlighting the need for targeted code optimizations and increased per service capacity.
Detection: Although built in alerting mechanisms were in place to detect this type of condition, alerts did not consistently reach the responsible teams as expected. In some cases, alerts were grouped or suppressed, delaying timely visibility of the issue. As a result, internal teams became aware Corrective Actions of the impact primarily through customer reports of slowness when accessing sites within the CXone Mpower Expert knowledge portal.
Remediation: The impact was resolved after scaling up pod resources and restarting the proxy pods, which restored platform stability. Completed on 05/07/2026.
Prevention: The Engineering team enhanced traffic filtering rules at the Web Application Firewall (WAF) layer to identify and block a significant portion of automated bot traffic contributing to elevated system load. These actions reduced unnecessary requests and improved overall platform stability. Completed on 05/07/2026.
The Engineering team implemented interim mitigation measures to maintain system stability and ensure consistent performance while permanent improvements are finalized and deployed. Completed on 05/12/2026.
Baseline system capacity was increased by raising the minimum number of pods and allocating higher CPU resources. This ensures sufficient resources are consistently available, reduces reliance on dynamic scaling, and improves overall system stability during periods of increased demand. Completed on 05/12/2026.
The Engineering team will implement targeted software optimizations, including improvements to a specific endpoint that previously introduced cascading effects during scaling events. These enhancements are designed to reduce resource contention and improve system efficiency under high load conditions. The changes are currently undergoing validation and will be deployed following comprehensive testing. An update will be provided by End of Day (EOD) MT on 05/22/2026.
The Engineering team will enhance alert notification delivery to ensure alarms are reliably triggered and routed to the appropriate response teams as expected, enabling faster detection and more timely corrective action when issues arise. An update will be provided by EOD MT on 05/22/2026.
05/07/2026 11:43 AM (UTC) - The first customer case opened, and Tech Support (TS) engineers began the troubleshooting investigation. 05/07/2026 11:44 AM (UTC) - TS engineers notified the Network Operations Center (NOC) engineers about the reported customer impact; a major incident was proposed and confirmed. 05/07/2026 12:09 PM (UTC) - Engineers identified a suspected cause and increased the resources of the web pods to improve system performance. 05/07/2026 12:18 PM (UTC) - Engineers also scaled up resources for the Application Programming Interface (API) pods to further stabilize performance. 05/07/2026 01:30 PM (UTC) - The platform continued to catch-up and engineers were already seeing improvements in system performance. 05/07/2026 01:52 PM (UTC) - Engineers restarted proxy pods, resulting in continued performance improvements while monitoring system stability. 05/07/2026 02:00 PM (UTC) - Platform performance returned to normal levels, with continued validation and monitoring underway. 05/07/2026 02:07 PM (UTC) - The platform stabilized fully. The impact was resolved following resource scaling, and after successful validation, the major incident was marked as resolved.
Major Incident# 02793390
Impact Start Time (UTC) 05/07/2026 03:03 PM UTC
Impact End Time (UTC) 05/07/2026 05:53 PM UTC
On 05/07/2026, some NiCE CXone Mpower customers experienced slowness when accessing sites, while others were unable to access the platform entirely with a “504 Gateway Timeout” error within the CXone Mpower Expert knowledge portal. The recurring service degradation incidents were caused by increased traffic volumes combined with performance limitations in certain backend processes, primarily impacting the US regional platform due to higher demand. The impact was mitigated after restarting the affected pods, which restored platform stability
The recurring service degradation incidents were caused by increased traffic volumes combined with performance limitations in certain backend processes, primarily impacting the US regional platform due to higher demand. Under elevated traffic conditions, including automated crawler activity, some requests followed less optimized processing paths, increasing system load. This was further amplified by legacy or complex page content requiring more intensive processing. While scaling actions helped restore capacity, they also introduced temporary overhead that contributed to intermittent performance degradation. Additionally, although autoscaling functioned as designed, it reached its limits and was insufficient to address constraints related to per-pod Central Processing Unit (CPU) capacity. Overall, evolving traffic patterns exposed underlying performance limitations, highlighting the need for targeted code optimizations and increased per service capacity.
Detection: Although built in alerting mechanisms were in place to detect this type of condition, alerts did not consistently reach the responsible teams as expected. In some cases, alerts were grouped or suppressed, delaying Corrective Actions timely visibility of the issue. As a result, internal teams became aware of the impact primarily through customer reports of slowness when accessing sites within the CXone Mpower Expert knowledge portal.
Remediation: The impact was mitigated after restarting the affected pods, which restored platform stability. Completed on 05/07/2026.
Prevention: The Engineering team enhanced traffic filtering rules at the Web Application Firewall (WAF) layer to identify and block a significant portion of automated bot traffic contributing to elevated system load. These actions reduced unnecessary requests and improved overall platform stability. Completed on 05/07/2026.
The Engineering team implemented interim mitigation measures to maintain system stability and ensure consistent performance while permanent improvements are finalized and deployed. Completed on 05/12/2026.
Baseline system capacity was increased by raising the minimum number of pods and allocating higher CPU resources. This ensures sufficient resources are consistently available, reduces reliance on dynamic scaling, and improves overall system stability during periods of increased demand. Completed on 05/12/2026.
The Engineering team will implement targeted software optimizations, including improvements to a specific endpoint that previously introduced cascading effects during scaling events. These enhancements are designed to reduce resource contention and improve system efficiency under high load conditions. The changes are currently undergoing validation and will be deployed following comprehensive testing. An update will be provided by End of Day (EOD) MT on 05/22/2026.
The Engineering team will enhance alert notification delivery to ensure alarms are reliably triggered and routed to the appropriate response teams as expected, enabling faster detection and more timely corrective action when issues arise. An update will be provided by EOD MT on 05/22/2026.
05/07/2026 03:03 PM (UTC) - The first customer case was opened, and Tech Support (TS) engineers began their initial validation and troubleshooting investigation.
05/07/2026 05:02 PM (UTC) - TS engineers notified the Network Operations Center (NOC) engineers about the reported customer impact; a major incident was proposed and confirmed.
05/07/2026 05:30 PM (UTC) - Engineers identified a suspected cause and began evaluating the most appropriate remediation approach. 05/07/2026 05:40 PM (UTC) - Engineers began their remediation efforts.
05/07/2026 05:53 PM (UTC) - The impact was resolved after engineers adjusted traffic filtering and handling. Following successful validations, the major incident was marked as resolved.