On Wednesday, March 4, 2026, VIP services experienced a connectivity disruption starting at approximately 4:00 PM ET. The incident, which affected core applications like Beverage, iDIG, VIP Sales, and VIP Delivery, resulted in a service impact for customers. Our teams prioritized immediate mitigation, achieving full restoration of service by 8:20 PM ET

‌

The disruption was caused by a hardware failure: a primary network switch in our Massachusetts data center failed. When the switch attempted to come back online, an immediate surge of traffic overwhelmed our backend server infrastructure, leading to connectivity refusal and service errors.

‌

We restored the primary connection at 5:24 PM ET but some customers still had problems accessing applications until 8:20 PM ET when full connectivity was restored.

‌

Preventative Measures

‌

To prevent a recurrence, we are implementing the following long-term actions:

• Automated Traffic Throttling: We are developing and deploying automated throttling mechanisms to manage the return of traffic to servers after a major outage. This will prevent server overload and ensure a stable, staged service recovery.
• Protocol Standardization: We are establishing a standardized process for the phased re-enabling of user access following any major outage to maintain system stability.

Enhanced Monitoring and Alerting: We are updating our monitoring and alerting systems to ensure key team members receive immediate, critical notifications (including SMS) for hardware and replication issues, ensuring the fastest possible response time.