UPDATE: All clusters are up and ready for service.
At this time, all PACE-managed clusters are believed to be working.
You should be able to login to your clusters and submit and run jobs.
Any jobs that were running before the power outage have failed, so please resubmit them.
Please let us know immediately if anything is still broken.
PACE Team
What happened
At around 0810 Thursday morning, Rich lost its N6 feed, half of the feed powering the Rich building and the Rich chiller plant. This also caused multiple failures in the high voltage vault in the Rich back alley, so Rich also lost its other feed, N5. However, the N5 feed was still up in the chiller plant. Though the chillers still had power, as a precaution operators transferred cooling over to the campus loop. Rich office space was without power, but the machine rooms failed over to the generator and UPSes.
PACE systems were powered down gracefully to prevent a hard-shutdown that would make recovery more difficult.
Original Post
This morning (December 19), the Rich datacenter suffered a power loss.
We had to perform an emergency shutdown of all nodes.
As we receive new information we will update this blog and the pace-availability email list.