Severity Level: High

The incident has been resolved and hosts in our us-east-3 region have been patched. On-demand instances and networking is now working as expected.

Severity Level: [High]  

Our engineers are currently awaiting a fix from our vendor at this time. Once the fix has been provided, they'll begin working to rollout the fix across our fleet in us-east-3.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [High] The proactive reboots completed by our engineers have significantly reduced the likelihood of sporadic disconnects in the near future. We will continue to monitor for further issues in the coming days. In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • https://support.lambdalabs.com/hc/en-us/requests/new To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.

Severity Level: [High]

The proactive reboots completed by our engineers have significantly reduced the uptime of our hosts. This means that the sporadic disconnects are unlikely to surface in the near future. We will continue to monitor for further issues in the coming days.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [High]  

Our engineers completed the proactive reboots at around 3:30 PM EDT. The last couple of hours have been spent working through BGP issues and instance checks. If you're instance still isn't up at this time, please open a support ticket.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [High]

Our engineers are going to begin proactive reboots tomorrow, Wednesday, May 14, starting at 11:00 AM EDT. We're unable to communicate just how long these reboots will take. That being said, we will provide an update here once the reboots have been completed.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [High]  

Our engineers continue to work with the vendor to obtain the fix. Otherwise, we're currently in a holding pattern. More information will provided as it becomes available.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [High]  

Our engineers have placed the most at risk hosts into maintenance. This prevents these hosts from running newly launched instances. Because of their actions, instance launches will be enabled in our Washington, DC region (us-east-3) momentarily.

That being said, the incident is ongoing. We're still working with the vendor to obtain and apply a fix. Until this fix has been applied, unexpected reboots are expected to continue.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [High]  

We continue to encounter the issue previously mentioned across hosts within our Washington, DC region (us-east-3). Once encountered, we must reboot the affected host to restore connectivity.

Our engineers continue to work with the vendor on implementing a fix. Until the fix is implemented, more reboots are expected.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [High]  

Launches in our Washington, DC region (us-east-3) have been disabled. Users will not be able to launch new instances at this time.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [High] UPDATE: We are seeing the same behavior of nodes crashing however it’s now in a new area of our data hall in this region. Users may experience intermittent downtime as we reboot affected nodes. In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.

Severity Level: [High]  

Our engineers are currently awaiting a fix from the vendor at this time. The frequency of communications will be reduced going forward. That being said, updates will still be provided as they're made available.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [Low/Medium/High/Critical]  

Connectivity has been restored to the hosts connected to the pair of switches previously noted. That being said, our engineers are still working on a permanent fix at this time.

This incident is currently in a holding pattern. More information will be provided as it comes available.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [High]  

Our engineers have narrowed the issue down to hosts connected to a pair of switches in the region and they're actively working towards remediation. Instances running on these hosts may still see intermittent network connectivity.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [High]  

Our engineers are currently working diligently to resolve interface errors seen within the Washington, DC region (us-east-3).

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity Level: [High]  

We're still seeing intermittent network connectivity within our Washington, DC region (us-east-3). Our engineers have identified the root cause of the issue and they’re actively working on remediation.

In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

Thank you for your patience and understanding during this time.

Severity: [HIGH] [UPDATE] The rolling reboots have completed. We are now monitoring the situation. Please see below for the other incident details. We’ve identified the cause being related to an existing issue the vendor is aware of however the patch for this won't be available anytime soon. We've found that the nodes that we've power cycled have not crashed again since. So, to prevent longer hard downtime for the nodes that will be affected, we're opting to perform a rolling power cycle of the remaining hosts that will eventually crash. Once the vendor has provided the permanent fix for this, we'll schedule another maintenance window as this will require the hosts to be power cycled again. In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • https://support.lambdalabs.com/hc/en-us/requests/new To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.

Severity Level: [High] [UPDATE]
The rolling reboots have begun however we'll need more time to complete them so we're delaying the end of the power cycle window to 10PM EST. Please see below for the other incident details. We’ve identified the cause being related to an existing issue the vendor is aware of however the patch for this won't be available anytime soon. We've found that the nodes that we've power cycled have not crashed again since. So, to prevent longer hard downtime for the nodes that will be affected, we're opting to perform a rolling power cycle of the remaining hosts that will eventually crash. Once the vendor has provided the permanent fix for this, we'll schedule another maintenance window as this will require the hosts to be power cycled again. In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • https://support.lambdalabs.com/hc/en-us/requests/new To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.

Severity Level: [High] We’ve identified the cause being related to an existing issue the vendor is aware of however the patch for this won't be available anytime soon. We've found that the nodes that we've power cycled have not crashed again since. So, to prevent longer hard downtime for the nodes that will be affected, we're opting to perform a rolling power cycle of the remaining hosts that will eventually crash. The rolling reboots will begin in ~30 minutes at 6PM EST, ending on or before 8PM EST. During this time, users may experience intermittent downtime as nodes reboot, with downtime lasting 15 minutes or less. If your instance has already experienced downtime, you should not experience further downtime. Once the vendor has provided the permanent fix for this, we'll schedule another maintenance window as this will require the hosts to be power cycled again. In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • https://support.lambdalabs.com/hc/en-us/requests/new To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.

Severity Level: [High] We’ve identified the cause being related to an existing issue the vendor is aware of however the patch for this won't be available anytime soon. We've found that the nodes that we've power cycled have not crashed again since. So, to prevent longer hard downtime for the nodes that will be affected, we're opting to perform a rolling power cycle of the remaining hosts that will eventually crash. The rolling reboots will begin in ~30 minutes at 6PM EST, ending on or before 8PM EST. During this time, users may experience intermittent downtime as nodes reboot, with downtime lasting 15 minutes or less. If your instance has already experienced downtime, you should not experience further downtime. Once the vendor has provided the permanent fix for this, we'll schedule another maintenance window as this will require the hosts to be power cycled again. In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.

Severity Level: [High] We’ve identified the cause being related to an existing issue the vendor is aware of however the patch for this won't be available anytime soon. We've found that the nodes that we've power cycled have not crashed again since. So, to prevent longer hard downtime for the nodes that will be affected, we're opting to perform a rolling power cycle of the remaining hosts that will eventually crash. The rolling reboots will begin in ~30 minutes at 4PM EST, ending on or before 6PM EST. During this time, users may experience intermittent downtime as nodes reboot, with downtime lasting 15 minutes or less. If your instance has already experienced downtime, you should not experience further downtime. Once the vendor has provided the permanent fix for this, we'll schedule another maintenance window as this will require the hosts to be power cycled again. In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible.

• https://support.lambdalabs.com/hc/en-us/requests/new

To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.