Summary: The Phoenix scheduler stopped launching new jobs on Friday evening and was restored at approximately 9:30 AM on Saturday.
Details: At some point after 8 PM on Friday evening, the node hosting the Moab workload manager of the Phoenix scheduler lost its network connection, leaving it unable to communicate with the rest of the cluster. The PACE team repaired the connection just before 9:30 AM on Saturday morning, and functionality was restored.
Impact: While jobs could be submitted via “qsub” and checked via “qstat”, no new jobs would launch but would instead remain queued. Moab commands such as “showq” would not have worked. Running jobs were not interrupted.
Thank you for your patience over the weekend. Please contact us at pace-support@oit.gatech.edu with any questions.