Our quarterly maintenance is now complete, and the clusters are running previously submitted jobs and awaiting new submissions.

We have successfully completed a number of things:

However, we were unable to complete the upgrade of the TestFlight cluster to RedHat 6.5.  At the moment TestFlight is down, and we will complete the upgrade over the next couple of days.

As always, please contact us (pace-support@oit.gatech.edu) for any problems or concerns you may have. Your feedback is very important for us, especially regarding file transfers in and out of the clusters.  (i.e. between your workstations and the PACE clusters)

PACE Quarterly maintenance has begun

See this space for updates.

PACE Quarterly maintenance notification

It’s time again for our quarterly maintenance.  We will have the clusters down April 15 & 16.

As usual, we’ve instructed the schedulers to avoid running jobs that would cross into a planned maintenance window.  This will prevent running jobs from being killed, but also may mean jobs you submit now may not run until after maintenance completes.  I would suggest checking the wall times for the jobs you will be submitting and, if possible, modify them accordingly so they will complete sometime before the maintenance. Submitting jobs with longer wall times is still OK, but they will be held by the scheduler and released after maintenance completes.

Much of our activities time around are not directly visible, with a couple of notable exceptions.

We will be upgrading the operating system on our TestFlight cluster from RedHat 6.3 to RedHat 6.5.  Please do test your codes on this cluster over the coming weeks and months, as we plan to roll it out (along with any needed fixes) to all other RedHat 6 clusters in July.  This update is expected to bring some performance improvements, as well as some critical security fixes.  Additionally, it adds support for the Intel Ivy Bridge platform, which many of you are ordering.  Any new Ivy Bridge platforms will start with RedHat 6.5.

Other user visible changes include:

• conclude the migration of the Athena cluster to RedHat 6.3.  We’ll plan to take Athena to 6.5 in July.
• conclude the migration of the BioCluster /nv/pb4 filesystem to the DDN/GPFS space.
• migrate mathlocal from /nv/hp24 to /nv/pma1 (Math cluster project space)
• application of recommended and security patches to our Solaris storage systems.  This is a widespread update will affect filesystems that start with /nv.  A rapid reversion process is available should unanticipated events occur.
• firewall upgrades to increase bandwidth between PACE and campus

Not so apparent changes include:

As always, please follow our blog for communications, especially for announcements during our maintenance activities – and let us know of any concerns via pace-support@oit.gatech.edu.

We’ve identified the source of problems which impacted all of the clusters this (4/7) afternoon.  While making preparations to deploy some firewall upgrades for PACE, one of the campus network team members inadvertently applied a misconfiguration to one of our core network links.  This resulted in widespread packet loss across the PACE internal network.

The head nodes seem to have recovered properly, but please let us know if you see continued issues there.  While it is possible that jobs have been lost, we believe that most things will have recovered without loss.

We’ll continue to monitor the situation and address any remaining problems as soon as we are able.

PACE Team