Koyeb - Notice history

All systems operational

Control Panel - Operational

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%

Previous page

Next page

Nov 2023
Dec 2023
Jan 2024

API - Operational

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%

Previous page

Next page

Nov 2023
Dec 2023
Jan 2024

Build / Provisioning - Operational

99.90% uptime
Nov 2023 · 100.0%Dec · 99.71%Jan 2024 · 100.0%

Previous page

Next page

Nov 2023
Dec 2023
Jan 2024

Third Party: Github → API Requests - Operational

Third Party: Github → Webhooks - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → API - Operational

Third Party: Docker → Registry - Operational

Third Party: Quay → Registry - Operational

Notice history

View current status

Jan 2024

Control panel unavailable
ResolvedPartial outage2 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

Delay in provisionning
ResolvedDegraded performance4 hours 12 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

    We had to perform an unplanned maintenance (involving a restart) on some LB components.

    Some services may have experience 502 errors during the maintenance.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    UTC
    Identified

    The root cause has been identified.

    We are continuing to work on a fix for this incident.

  • Investigating
    UTC
    Investigating

    We noticed a delay when provisioning or re deploying services.
    We are currently investigating this incident.

Dec 2023

Some service are no longer reachable
ResolvedPartial outage57 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We believe we found the root cause of the issue, more investigation is required, it seems to be related to one of the load-balancing components misbehaving under a certain load.

  • Investigating
    UTC
    Investigating

    There is currently an elevated rate of 502 errors when requesting some services, we are currently investigating the root cause of the issue.

Provisioning delay
ResolvedPartial outage7 hours 14 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

    The cluster is now back online and the builds should be able to restart.

    We tried to re-route the builds to another region during the outage, but we were sadly not able.

    We are still trying to understand the root cause of the issue.
    And we will try to understand why the re-routing of the build was not possible during the outage.

  • Identified
    UTC
    Identified

    We have identified some provisioning delay for builds and deployments in the Frankfurt region.

    Our engineering team is investigating.

Nov 2023

Services down due to hypervisor failure
ResolvedPartial outage1 hour 9 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    UTC
    Identified

    We are continuing to work on a fix for this incident, the situation is improving.

  • Investigating
    UTC
    Investigating

    Somes services deployed in FRA are unavailable due to a failure of the underlying hypervisors.

    Our engineering team is investigating.

Services down due to multiple hypervisor failure
ResolvedMajor outage1 hour 19 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    UTC
    Identified

    We are continuing to work on a fix for this incident.

    Some services are still experiencing issues and are not re-scheduled yet.

  • Investigating
    UTC
    Investigating

    Multiple services services deployed in Frankfurt are unavailable due to a failure of the underlying hypervisors.

    Hypervisors failures happen and services should be automatically migrated - in this case they didn’t automatically recover.

    We are also seeing increase of 502 error due to some load balancer failure.

    Our engineering team is investigating.

Instabilities on deployment engine
ResolvedDegraded performance5 hours 15 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Investigating
    UTC
    Investigating

    We are observing an increase of failing deployments and some delays in provisioning in Frankfurt. We are currently investigating the issue

Impossible to add new custom domains
ResolvedDegraded performance43 hours 50 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Identified
    UTC
    Identified

    We are continuing to work on a fix for this incident.

  • Investigating
    UTC
    Investigating

    Adding new custom domains is currently unavailable. Existing domains are operational.

    The problem has been identified. However, it is in the hands of a third party service so we cannot commit on a resolution date yet.

New services stuck in Starting status
ResolvedDegraded performance2 hours 31 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    UTC
    Identified

    New services on the platform are not reaching Healthy - they get stuck in Starting.

    The problem has been identified and a fix is currently being deployed.

Previous

Nov 2023 to Jan 2024

Next