[ Update 8/14/20 7:00 PM ]
After an additional nearly-48-hour outage in the Rich datacenter due network/InfiniBand issues, we have brought back up PACE resources on the affected systems and released user jobs. We thank you for your patience and understanding during this unprecedented outage, as we understand the significant impact that this outage has continued to have on your research throughout this week. Please note that PACE clusters in the Coda datacenter (Hive and testflight-coda) and CUI clusters in Rich have not been impacted.
While new jobs have not begun over the past two days, already-running jobs have continued. Please check the output of any jobs that are still running. If they are failing or not producing output, please cancel them and resubmit to run again. Some running user jobs were killed in the process of repairing the network, and those should also be resubmitted to the queue.
In addition to previously reported repairs, we removed a problematic spine module from a network switch this morning and further adjusted connections. This module appeared to be causing intermittent failures when under heavy load.
Currently, our network is running at reduced capacity. We have ordered a replacement switch spine module that will be used to replace the removed part. We have conducted extensive stress testing of the network and storage today, which were far beyond tests conducted earlier in the week, that indicate the system is healthy. We will continue to monitor the systems for any further network abnormalities.
Again, thank you for your patience and understanding this week while we addressed one of the most significant outages in the history of PACE.
Please contact us at pace-support@oit.gatech.edu with any questions or if you observe unexpected behavior on the cluster.
[ Update 8/13/20 8:30 PM ]
We continue to work on the network issues impacting the Rich datacenter. We have partitioned the network and adjusted connections in an effort to isolate the problem. As mentioned this morning, we have ordered parts to address potential problematic switches as we continue systematic troubleshooting of them. We continue to run tests on InfiniBand, and we are running an overnight stress test on the network to monitor for reoccurrence of errors. The schedulers remain paused to prevent further jobs being launched on the cluster. We will follow up tomorrow with an update on the Rich cluster network.
Thank you for your continued patience and understanding during this outage.
[ Update 8/13/20 10:10 AM ]
[ Update 8/12/20 6:20 PM ]
[ Update 8/12/20 12:30 AM ]
We continue to work to bring PACE nodes back into production. After turning off all the compute nodes and reseating faulty network connections we identified, we have been slowly bringing nodes back up to avoid overwhelming the network fabric, which has been clean so far. We are carefully testing each group to ensure full functionality, and we continue to identify challenging nodes and repair them where possible. At this time, the schedulers remain paused while we turn on and test nodes. We will provide additional updates as more progress is made.
[ Update 8/11/20 5:15 PM]
We continue to troubleshoot the network issues in the Rich datacenter. Unfortunately, our efforts to avoid disturbing running jobs have complicated the troubleshooting, which has not led to a resolution. At this time, we need to begin systematic rebooting of many nodes, which will kill some running user jobs. We will contact users with current running jobs directly to alert you to the effect on your jobs.
Our troubleshooting today has included reseating multiple spine modules in the main datacenter switch, adjusting uplinks between the two main switches to isolate problems, and rebooting switches and some nodes already.
We will continue to provide updates as more information becomes available. Thank you for your patience during this outage.
[ Update 8/10/20 11:35 PM ]
We have made several changes to create a more stable Infiniband network, including deploying an updated subnet manager, bypassing bad switch links, and repairing GPFS filesystem errors. However, we have not yet been able to uncover all issues the network is facing, so affected schedulers remain paused for now, to ensure that new jobs do not begin when they cannot produce results.
We will provide an update on Tuesday as more information becomes available. We greatly appreciate your patience as we continue to troubleshoot.
[ Update 8/10/20 6:20 PM ]
We are continuing to troubleshoot network issues in Rich. At this time, we are working to deploy an older backup subnet manager, and we will test the network again to determine if communication has been restored after that step.
The schedulers on the affected clusters remain paused, to ensure that new jobs do not begin when they cannot produce results.
We recognize that this outage has a significant impact on your research, and we are working to restore functionality in Rich as soon as possible. We will provide an update when more information becomes available.
[ Update 8/9/20 11:55 PM]
[ Original Post]
At approximately noon today, we began experiencing issues with our primary InfiniBand Subnet Manager in Rich data center. PACE is investigating this issue. We will provide an update when additional information or a resolution is available. At this time, you may experience slowness in accessing storage (home, project, or scratch) or issues with communication within MPI jobs.
In order to minimize impact to jobs, we have paused all schedulers on the affected clusters (accessed via login-s, login-d, login7-d, novazohar, gryphon, and testflight-login headnodes). This will prevent additional jobs from starting, but jobs that are already running will not be stopped, although they may fail to produce results due to the network issues.
This issue does not impact the Coda data center (Hive & testflight-coda clusters) or CUI clusters in the Rich data center.
Please contact us with any questions or concerns at pace-support@oit.gatech.edu.