Incident Overview

Date: June 11th, 2024

Time: 2:30 AM UTC to 5:02 AM UTC

Duration: 2 hours and 32 minutes

Current status: The service has been fully restored https://status.livepeer.studio/

Summary

On June 11th, from 2:30 AM UTC to 5:02 AM UTC, our services experienced a significant disruption due to an outage with our storage provider. This outage impacted our ability to record livestreams and upload Video on Demand (VOD) content. Unfortunately, any livestream recordings or VOD uploads attempted during this period are not available and we apologize for the inconvenience this incident may have caused.

Impact

• Livestreams:

  • Recordings of livestreams during the outage window are lost and unrecoverable.

• VOD Uploads:

  • Any VOD content uploads attempted during the outage were unsuccessful and the data was lost.

Root Cause

The incident was triggered by an unexpected outage with our third-party storage provider, which is responsible for storing all livestream recordings and VOD uploads. The provider's systems experienced a failure that rendered them unable to handle our data storage needs during the specified time.

Resolution

Our team closely monitored the situation and worked with the storage provider to restore service as quickly as possible. Full functionality was restored by 5:02 AM UTC.

Remediation and Future Prevention

To prevent such incidents from impacting our services in the future, we are taking the following actions:

1. Engagement with Storage Provider: We are in ongoing conversations with our storage provider to address the reliability of their systems and to understand the root cause of their outage.
2. Fallback Solutions: We are actively developing a fallback option to ensure that no assets are lost in the event of a similar outage. This includes exploring alternative storage solutions and implementing redundancy measures.