Foxglove - Slow API responses – Incident details

Slow API responses

ResolvedDegraded performance23 hours 50 minutes

Affected

API

Major outage from 8:15 PM to 9:40 PM, Operational from 9:40 PM to 10:35 PM, Degraded performance from 10:35 PM to 7:07 PM, Operational from 7:07 PM to 8:06 PM

  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    We have fully implemented a remediation and see request latency and errors have returned to normal.

  • UpdateUTC

    We are continuing to remediate this issue. At this time we expect most requests to be unaffected however API requests for lake files and some streaming requests may experience issues.

  • UpdateUTC

    We are performing a maintenance operation which may impact service briefly.

  • UpdateUTC

    Timeline updates are currently delayed, we are continuing to work on the root performance issue.

  • UpdateUTC

    We are seeing continued contention on our database servers, and continuing to work on fixes. Most API requests should succeed and respond quickly right now. If you experience a stalled request while visualizing, then retrying may help.

  • IdentifiedUTC

    We are still seeing intermittent issues and continuing to work on a fix for this incident.

  • ResolvedUTC

    This incident has been resolved, we are continuing to monitor performance.

  • IdentifiedUTC

    We're noticing slow API responses and an elevated error rate. We've identified the issue and are working to remediate.