PACE A Partnership for an Advanced Computing Environment

Our October 2016 maintenance period is now complete. We’ve compute nodes, login nodes and interactive (post-processing) nodes to the RedHat Enterprise Linux 6.7 previously deployed on the TestFlight cluster.  This included a large number of bugfix and security patches, a major step forward in the Infiniband layer and recompiled versions of various MPI, scientific libraries and applications in /usr/local.  Please do let us know (via email to pace-support@oit.gatech.edu) if you see issues with your jobs.

We have brought compute nodes online released previously submitted jobs. As usual, we have a number of compute nodes that still need to be brought back online, but we are actively working to make them available asap.

DDN/GPFS work
Hardware repairs to the project directory (~/data) system is complete.  Minor repairs to the scratch system will be rescheduled for a future maintenance period.  The issue is minor and should not disrupt performance or availability of the scratch system.  No user actions are expected.

Networking
The OIT Network Engineering team upgraded the software running on many of our switches to match that which is running elsewhere on campus.  This included our firewalls.  No user actions are expected.

Electrical work
These problems were a bit more extensive than originally anticipated.  With some help from the OIT Operations team, we have a alternate solution in place, and will complete this work at a future maintenance period.  No user action is expected.

Bonus objectives
We were able to add capacity to the project directory system, and we now have our first single filesystem that’s greater than a petabyte, coming at about 1.3PB.  Maybe that’ll last us a couple of weeks.  😉  Disks for the scratch system have been installed, we will add them into the scratch filesystem shortly.  This can be done live without impact to running jobs.

Dear PACE users,

It is again time for our quarterly maintenance. Starting at 6:00am Thursday, October 13, all resources managed by PACE will be taken offline. Maintenance is scheduled to continue through Saturday evening. Our next maintenance period is scheduled for January 19-21, 2017.

As previously posted, our major activity this time around will be an upgrade of all compute nodes, head nodes and interactive nodes from RedHat 6.5 to RedHat 6.7. This is simply deploying the operating system and software repository we been testing internally and have made available on the TestFlight cluster for the broader PACE community to perform testing as well. I’ll issue one more plea for additional testing to the user community and ask that you report experiences, either positive or negative, to pace-support@oit.gatech.edu.

DDN/GPFS work
We have some hardware repairs to perform on both DDN devices. Internal redundancy has prevented any availability or integrity problems so we’ll take advantage of the downtime for DDN engineers to make repairs. No user action is expected.

Networking
The OIT Network Engineering team will be upgrading the software running on a number of PACE network devices. These updates are for stability and have been running on similar devices in the campus network for some time. No user action is expected.

Electrical work
OIT Operations will be replacing some faulty rack-level power distribution in a couple of racks. No user action is expected.

Bonus objectives
If sufficient time remains in the maintenance period, we will add capacity to both the GPFS scratch and project systems. We are able to perform these activities during normal operations, so we will prioritize bringing things back into operation as soon as possible.