Summary

This is a post-mortem describing the incident being investigated on 5/8/24
https://status.livepeer.studio/incidents/2ljr9lffd5t8

Incident

Description

Users reported experiencing either frequent rebuffering or an inability to view playbacks. Upon investigation, the Livepeer Studio team determined that the issue originated from the Finland region. It was caused by servers exceeding their allocated network bandwidth capacity. By redirecting traffic away from this region, the problem was resolved.

Impact

• Livestreams:

  • N/A

• Viewers:

  • Some viewers were not able to view existing livestreams
  • Some viewers experienced high rebuffed rates while viewing livestreams

• Regions:

  • Finland

Current status

The service has been fully restored

https://status.livepeer.studio/

Timeline

• 2:24 PM EST - An investigation was conducted by the Livepeer Studio team to find the cause of the issue
• 3:46PM EST - The Livepeer studio team was still investigating and isolated the area where the root cause could be coming from
• 4:16PM EST - The Livepeer Studio team identified the underlying cause of the problem with both servers in Finland having more traffic than it can handle. When traffic was then redirect away from those two servers, there was significant improvements with the rebuffering rate
• 4:26 PM EST - After monitoring the fix for the incident, the Livepeer Studio team concluded that the issue was resolved

Prevention

• Enhance alerting for servers with low bandwidth
• Improve systems for monitoring performance of servers in each region