Quadric - Failing API calls – Incident details

All systems operational

Failing API calls

Resolved
Major outage
Started about 2 months agoLasted about 4 hours

Affected

Runners

Major outage from 1:15 AM to 4:05 AM, Partial outage from 4:05 AM to 4:25 AM, Operational from 4:25 AM to 5:01 AM

Website

Major outage from 1:15 AM to 3:45 AM, Partial outage from 3:45 AM to 4:05 AM, Operational from 4:05 AM to 5:01 AM

API

Major outage from 1:15 AM to 3:45 AM, Partial outage from 3:45 AM to 4:05 AM, Operational from 4:05 AM to 5:01 AM

Updates
  • Resolved
    Resolved

    Service has been restored.

  • Update
    Update

    Runners are starting to process the backlog of jobs.

  • Update
    Update

    We have restored baseline capacity and are monitoring.

  • Update
    Update

    API calls are restored. We are starting the process of re-allocating runners to execute jobs.

  • Monitoring
    Monitoring

    We are redeploying services.

  • Update
    Update

    The underlying infrastructure has been reconfigured.

  • Update
    Update

    We have corrected the configuration issue and are in the process of applying the changes across our infrastructure.

  • Identified
    Identified

    A recently expired TLS certificate is causing connectivity issues in our underlying infrastructure.

  • Update
    Update

    The high failure rate seems to be because of underlying infrastructure configuration issue. We are investigating.

  • Investigating
    Investigating

    We are observing high rate of failure in API call responses after a recent deployment.