This is the message that Rusty received from the hosting company. It's a major hardware firmware failure -- and it may not be resolved quickly because they don't have enough hardware to replace the failed components (the blade servers). We may have to accept the fact that when we do get up and running , there may not be a proper backup to restore the site to a recently archived steady state =-- bottom line; a lot of recently posted content may be lost.
_______________________________________________________________________________________________________________________________
"News: Atlanta - Blades in A01-05, B01-05, C01-05, J11-J15 Published: 11/06/2018 We've had a detailed update from our DC about the status of around 120 blade servers that have been affected by a recent firmware update.
At the beginning of June, we began upgrading our servers with the new ILO code to address both TLS vulnerability and the new Java security requirements. During last week we had 1-2 servers that failed their upgrade but completed over 1500 upgrades successfully. On Friday we started upgrading the next batch of systems in A01-05, B01-05, C01-05, J11-J15, there were not incidents detected until Saturday when a total of 174 servers dropped offline over the course of 3 hours, we’ve had a few more drop out on Sunday and Monday as well.
We notified the DC to check through the first systems and they found the server refused to power up, it just sat with a flashing red health error light. They contacted HPE for assistance and carried out their recommend downgrade procedure on a couple of the blades. It made no difference.
HP's tech investigated a few of the blades and discovered that the power management controller had failed to upgrade properly and was preventing the blade from powering up. We have tried a variety of operations with HPE to resurrect these servers, some has come back to life, some are still offline. The DC will continue to work with HPE to find a solution to these issues.
We have swapped hardware where we have available stock but have now depleted our stock of spare compatible systems so we are reliant on HPE resolving the firmware issue. In the meantime we’re shipping a couple of pallets of servers from New York and Denver to Atlanta so we can replace hardware where necessary.
The DC is working round the clock with HPE to find a resolution, if we can’t get a firmware resolution then we’ll replace the hardware as soon as additional kit arrives on site.
Its an unfortunate and messy situation all round and we apologise for the downtime."
