Certain streams are being erroneously rejected

Incident Report for Livepeer Studio

Postmortem

What Happened

On 4/27 from 20:23 UTC to 20:45 UTC all streams that were in progress failed to be transcoded. Customers who had ongoing streams needed to be restarted. This was due to a bug in recently deployed software that caused RTMP streams to be rejected.

What We Are Doing About This

Once discovered, we immediately rolled back the changes that caused this bug. We are adding additional alerting to our staging environment to improve testing so that we’re more confident that our deployments are bug-free. Additionally, we’re making it easier and faster to roll back changes so that if we ever encounter this issue again, we’ll be up and running faster.

Posted May 03, 2021 - 22:11 UTC

Resolved

We have confirmed that no streams are being erroneously rejected.

Posted Apr 27, 2021 - 21:12 UTC

Monitoring

The fix has been rolled out and streams are no longer being erroneously rejected. We will continue to monitor to be sure.

Posted Apr 27, 2021 - 21:07 UTC

Identified

We are rolling out a fix that should stop streams from being erroneously rejected.

Posted Apr 27, 2021 - 21:00 UTC

Update

We have identified the issue.

Posted Apr 27, 2021 - 20:55 UTC

Investigating

We are currently investigating this issue. Only a select few streams are being erroneously rejected.

Posted Apr 27, 2021 - 20:42 UTC

This incident affected: Livepeer Streaming API, Legacy Livepeer Studio API, Media server transcoding via Livepeer (Media server transcoding via Livepeer - US, Media server transcoding via Livepeer - EU), and Livepeer Studio Ingest and Playback (Chicago (MDW) ingest and playback, Frankfurt (FRA) ingest and playback, London (LON) ingest and playback, New York (NYC) ingest and playback, Singapore (SIN) Ingest and playback).