Generator upgrade today went according to plan in most respects in that the new generator and power infrastructure performed as expected and our facility performed nominally. However, we identified an issue where a single rack lost all power as breakers were triggered during handover to generator power. The theory is that some equipment in that specific rack is faulty, causing a short circuit which took power.
For this reason almost all Ryzen and Epyc workloads went offline for some minutes. Almost all services are up again but we are working through some snags in starting certain instances at the moment which we hope to resolve soon.
We apologize for this incident and our electricians are hard at work trying to identify the malfunctioning equipment.
