Foxglove - Slow API responses – Incident details

All systems operational

Slow API responses

Resolved
Degraded performance
Started 4 months agoLasted about 24 hours

Affected

API

Major outage from 8:15 PM to 9:40 PM, Operational from 9:40 PM to 10:35 PM, Degraded performance from 10:35 PM to 7:07 PM, Operational from 7:07 PM to 8:06 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We have fully implemented a remediation and see request latency and errors have returned to normal.

  • Update
    Update

    We are continuing to remediate this issue. At this time we expect most requests to be unaffected however API requests for lake files and some streaming requests may experience issues.

  • Update
    Update

    We are performing a maintenance operation which may impact service briefly.

  • Update
    Update

    Timeline updates are currently delayed, we are continuing to work on the root performance issue.

  • Update
    Update

    We are seeing continued contention on our database servers, and continuing to work on fixes. Most API requests should succeed and respond quickly right now. If you experience a stalled request while visualizing, then retrying may help.

  • Identified
    Identified

    We are still seeing intermittent issues and continuing to work on a fix for this incident.

  • Resolved
    Resolved

    This incident has been resolved, we are continuing to monitor performance.

  • Identified
    Identified

    We're noticing slow API responses and an elevated error rate. We've identified the issue and are working to remediate.