RDU virtualization infrastructure host issue

Incident Report for Smartt

Resolved

All services in RDU are operating normally.

The storage evacuation completed overnight. We will now proceed to investigate and repair the host. Due to our redundant N+2 infrastructure, bringing this host offline, repairing it, and readding it to the cluster will not adversely affect any client services.

This is our final update for this issue. If you have any questions, please contact Smartt support.
Posted Feb 13, 2025 - 06:00 PST

Update

All services in RDU are operating normally. We continue to monitor the final stage of evacuating storage from the host. We do not expect any issues.

We will provide an update by 0615 PT.
Posted Feb 12, 2025 - 18:01 PST

Update

All services in RDU are operating normally. We continue to monitor the evacuation of storage from the host. We do not expect any issues.

We are in the final stage of the storage evacuation. We do not have a firm estimate for when this will complete. No data is at risk during this stage.

We will provide an update by 1815 PT. After that update, we will change to a 12 hour update cycle.
Posted Feb 12, 2025 - 15:15 PST

Update

All services in RDU are operating normally. We continue to monitor the evacuation of storage from the host. We do not expect any issues.

We will provide an update by 1515 PT.
Posted Feb 12, 2025 - 12:02 PST

Update

All services in RDU are operating normally. We continue to monitor the evacuation of storage from the host. We do not expect any issues.

We will provide an update by 1215 PT.
Posted Feb 12, 2025 - 09:00 PST

Update

All services in RDU are operating normally. We continue to monitor the evacuation of storage from the host. We do not expect any issues.

We will provide an update by 0915 PT.
Posted Feb 12, 2025 - 05:58 PST

Update

All services in RDU are operating normally. We continue to monitor the evacuation of storage from the host. We do not expect any issues.

We will provide an update by 0615 PT.
Posted Feb 12, 2025 - 03:16 PST

Update

All services in RDU are operating normally. We continue to monitor the evacuation of storage from the host. We do not expect any issues.

We will provide an update by 0315 PT.
Posted Feb 12, 2025 - 00:10 PST

Update

All services in RDU are operating normally. We continue to monitor the evacuation of storage from the host. We do not expect any issues.

We will provide an update by 0015 PT.
Posted Feb 11, 2025 - 21:03 PST

Update

All services in RDU are operating normally. We continue to monitor the evacuation of storage from the host. We do not expect any issues.

We will provide an update by 2115 PT.
Posted Feb 11, 2025 - 17:58 PST

Update

All services in RDU are operating normally. We continue to monitor the evacuation of storage from the host. We do not expect any issues.

We will provide an update by 1815 PT.
Posted Feb 11, 2025 - 15:09 PST

Monitoring

All services in RDU are operating normally. We do not expect any issues.

We continue to monitor the evacuation of storage from the host. This will take several hours because the storage cluster is rebalancing load across the cluster members.

We adjusted the storage cluster configuration to decrease the storage evacuation time in the future. Unfortunately, this change will not speed up the current evacuation.

We will provide an update by 1515 PT.
Posted Feb 11, 2025 - 11:51 PST

Update

We evacuated client VPSs off the host.

We are still evacuating VM storage from the host. We expect this evacuation will require several hours.

We will provide an update by 1215 PT.
Posted Feb 11, 2025 - 09:05 PST

Identified

We detected an issue with one of the RDU virtualization hosts that could affect the environment if left unaddressed. At present, client services are operating normally.

We will evacuate the host so we can perform maintenance on it.

VPSs running on the host may experience downtime if we cannot cleanly evacuate them from the host.

We will provide an update by 0915 PT.
Posted Feb 11, 2025 - 08:01 PST
This incident affected: VPS & VPC (RDU Virtual Cluster).