Hey folks,
The fileserver providing access to the filesystems hosted under (/nv)/pc1 has suffered a severe failure, requiring replacement parts before we can bring it online again. We are in contact with the vendor to try and resolve this as quickly as possible.
bnm
We have completed our maintenance day activities, and are now back into regular operation. Please let us know (via email to pace-support@oit.gatech.edu) if you encounter problems.
–Neil Bright
Maintenance day has begun at 6:00am on April 16, 2013.
No users will be allowed to login and no jobs may be submitted until maintenance day is complete.
The next maintenance day (4/16, Tuesday) is just around the corner and we would like to remind you that all systems will be powered off for the entire day. You will not be able to access the headnodes, compute nodes or your data until the maintenance tasks are complete.
None of your jobs will be killed, because the job scheduler knows about the planned downtime, and does not start any jobs that would be still running by then. You might like to check the walltimes for the jobs you will be submitting and modify them accordingly so they will complete sometime before the maintenance day, if possible. Submitting jobs with longer walltimes is still OK, but they will be held by the scheduler and released right after the maintenance day.
We have many tasks to complete, and here’s a summary:
1) Job Resource Manager/Scheduler maintenance
Contrary to the initial plan, we decided NOT to upgrade the resource manager (torque) and job scheduler (moab) software yet. We have been testing the new versions of these software (with your help) and, unfortunately, identified significant bugs/problems along the way. Despite being old, the current versions are known to be robust, so we will maintain the status quo until we resolve all of the problems with the vendor.
2) Interactive login prevention mechanism
Ideally, compute nodes should not allow for interactive logins, unless the user has active jobs on the node. We noticed that some users can directly ssh to compute nodes and start jobs, however. This may lead to resource conflicts and unfair use of the cluster. We identified the problem and will apply the fix on this maintenance day.
3) continued RHEL-6 migration
We are planning to convert all of the remaining Joe nodes to RHEL6 in this cycle. We will also convert the 25% of the remaining RHEL5 FoRCE nodes. We are holding off the migration for Aryabhata and Atlas clusters per request of those communities.
4) Hardware installation and configuration
We noticed that some of the nodes in the Granulous, Optimus and FoRCE are still running diskless, although they have local disks. Some nodes also not using the optimal choice for their /tmp. We will fix these problems.
We received (and tested) a replacement for the fileserver for the Apurimac project storage (pb3), since we have been experiencing problems there. We will install the new system and swap the disks. This is just a mechanical process and your data will is safe. As an extra precaution, we have been taking incremental backups (in addition to the regular backups) of this storage since it first started showing the signs of failure.
5) Software/Configurations
We will also patch/update/add software, including:
6) Electrical Work
We will also perform some electrical work to better facilitate the recent and future additions to the clusters. We will replace some problematic PDUs and redistribute the power among racks.
7) New storage from Data Direct Networks (DDN)
Last, but not least! In concert with a new participant, we have procured a new high performance storage system from DDN. In order to make use of this multi-gigabyte/sec performing monster, we are installing the GPFS filesystem. This is a commercial filesystem which PACE is funding. We will continue to operate the Panasas in parallel with DDN, and both storage systems can be used at the same time from any compute node. We are planning a new storage offering that allows users to purchase additional capacity on this system, so stay tuned.
As always, please contact us form pace-support@oit.gatech.edu for any questions/concerns you may have.
Thank you!
PACE Team
What could you do with over 25,000 computer cores? Join faculty and students at the April 30 High Performance Computing Town Hall to find out. The event will be held in the MaRC auditorium and is sponsored by PACE, Georgia Tech’s Advanced Computing Environment program.
When: April 30, 3-5pm
Where: MaRC Auditorium (Map to location)
PACE provides researchers with a robust computing platform that enables faculty and students to carry out research initiatives without the burden of maintaining infrastructure, software, and dedicated technicians. The program’s services are managed by OIT’s Academic & Research Technologies department and include physical hosting, system management infrastructure, high-speed scratch storage, home directory space, commodity networking, and common HPC software such as RedHat Enterprise Linux, VASP, LAMMPS, BLAST, Matlab, Mathematica, and Ansys Fluent. Various compilers, math libraries and other middleware is available for those who author their own codes. All of these resources are designed and offered with the specific intention of combining intellect with efficiency, in order to advance the research presence here at Tech to the peak of its abilities.
There are many ways to participate with PACE. With a common infrastructure, we support clusters dedicated to individual PIs or research groups, clusters that are shared amongst participants and our FoRCE Research Computing Environment (aka “The FoRCE”). The FoRCE is available to all campus users via a merit-based proposal mechanism.
The April 30 HPC Town Hall is open to members of the Tech research community and will feature presentations on the successes and challenges that PACE is currently experiencing, followed by a panel discussion and Q&A.
For more information on the PACE program, visit the official website at www.pace.gatech.edu, and also the program’s blog at blog.pace.gatech.edu.
Powered by WordPress