PACE A Partnership for an Advanced Computing Environment

August 29, 2017

Please Use iw-dm-4 for File Transfers

Filed under: Uncategorized — Semir Sarajlic @ 7:25 pm

We experienced a slowdown yesterday on all headnodes caused by an unusually large amount of user file operations from headnodes. All headnodes are virtual machines and connect through a network file system gateway to the GPFS filesystem. This gateway became overwhelmed by the user file operations, and subsequently slowed down all headnode file operations.

If you have heavy file operations (i.e. winSCP, FileZilla, SCP), please perform these by logging directly into iw-dm-4.pace.gatech.edu instead of a headnode. Additionally, other file operations such as tarring/zipping are best performed on compute nodes by submitting interactive or batch jobs, as well as iw-dm-4.

We’re actively looking into alternatives to virtual machine headnodes, and will provide more detailed updates as we approach our upcoming scheduled maintenance in November (via blog.pace.gatech.edu). If you have any questions, please email us at pace-support@oit.gatech.edu.

August 14, 2017

Resolved: Apparent CSH and TCSH Problems

Filed under: Uncategorized — Semir Sarajlic @ 3:41 pm

We’ve addressed some of the problems with the TrueNAS storage and CSH/TCSH should now be working again. As it turns out, this problem wasn’t actually related to the maintenance last week, and we will continue to work with the vendor regarding the cause.

Apparent CSH and TCSH Problems

Filed under: Uncategorized — Semir Sarajlic @ 3:13 pm

Currently, we’ve observed a correlation between hanging processes on all PACE systems and csh/tcsh, and are continuing to investigate. For the time being, if you have commands related to csh or tcsh, we ask that you please refrain from running them for the time being. It appears to be related to the TrueNAS storage system. We’re currently working with OIT and iXsystems to resolve the issue.

August 12, 2017

[RESOLVED] Post-maintenance Storage Issues

Filed under: Uncategorized — Semir Sarajlic @ 4:27 am

Update: GPFS storage is stabilized again. There remains several steps we need to take to complete this work, which can be completed without a downtime. We may need to take some nodes temporarily offline, which will be done in coordination with you without impacting the running jobs.
—–

PACE systems started experiencing wide-spread problems with the GPFS storage shortly after releasing jobs after the maintenance tasks are complete. At fist glance,  they seem to be related to the Infiniband network.

We would like to advise against submitting new jobs until these issues are fully resolved. We will continue to work on a resolution and keep you updated on the progress.

Thank you for your patience.

August 9, 2017

PACE systems experiencing problems

Filed under: Uncategorized — Semir Sarajlic @ 2:37 pm
Update: The immediate issue is now resolved. We continue to work with the vendor to find more about what caused the problem. If you need assistance with failed jobs or unavailable headnodes, please contact pace-support@oit.gatech.edu.
We are experiencing a storage issue that’s impacting majority of the home directories and some services, with potential impact on running jobs. We are currently in process of assessing the overall impact and investigating the root cause. We’ll update our blog (https://blog.pace.gatech.edu/) as we know more about the failures.
We are sorry of the inconvenience and thank you for your patience.
Update: Impacted filesystems:
ap1
ap2
ap3
ap4
ap5
ap6
hase1
hb51
hbiobot1
hcee2
hcfm1
hchpro1
hcoc1
hface1
hggate1
hhygene1
hjabberwocky1
hmart1
hmedprint1
hmeg1
hmicro1
hneutrons1
hnjord1
hp1
hp10
hp11
hp12
hp13
hp14
hp15
hp16
hp17
hp18
hp19
hp2
hp20
hp21
hp22
hp23
hp24
hp25
hp26
hp27
hp28
hp29
hp4
hp5
hp6
hp7
hp8
hp9
hpampa1
hsemap1
hska1
hsonar1
hthreshold1
html1

August 3, 2017

PACE quarterly maintenance – (Aug 10-12, 2017)

Filed under: Uncategorized — Semir Sarajlic @ 8:44 pm

Dear PACE users,

PACE clusters and systems will be taken offline at 6am on Thursday, Aug 10 through the the end of Saturday (Aug 12). Jobs with long walltimes will be held by the scheduler to prevent them from getting killed when we power off the nodes. These jobs will be released as soon as the maintenance activities are complete.

Planned improvements are mostly transparent to users, requiring no user action before or after the maintenance.

Storage (no user action needed)

We are working with the vendor to reconfigure our multiple GPFS filesystems for performance fine-tuning and better support for large scale deployments. Some of these configurations will be applied on the maintenance day because they require downtime.

Network (no user action needed)

PACE infiniband (IB) network requires some configuration changes to handle the rapid growth in the number of PACE nodes. We have identified several configuration parameters that will potentially reduce the occurrence of nodes losing their connectivity to GPFS (which relies on the IB network), causing intermittent job crashes.

Schedulers (postponed, no user action needed)

We have communicated our plans to upgrade scheduler in several occasions in the past, but skipped this task in the past maintenance days due to the bugs that we had uncovered. Despite the promising progress on the resolution of these bugs by the vendor, they are not fully resolved and tested yet. For this reason, we decided to once again postpone upgrading plans and keep the current versions until we have a bug-free and well-tested version.

Scheduler-based monitoring and analysis (no user action needed)

PACE research scientists started a collaboration with Texas Advanced Computing Center (TACC) to develop a new tool to analyze scheduler logs to gain insights about usage trends. For more details, please check https://doi.org/10.1145/3093338.3093351

This tool heavily relies on a widely used utility named ‘PBSTools’, which is developed by Ohio Supercomputer Center. Our current installation of PBSTools is old, buggy and very slow. We will upgrade this tool and its database on the maintenance day to ensure that no job info will be lost during transition.

Power Work  (no user action needed)

PACE and the OIT Operations team will perform some work in the datacenter to improve electrical capacity and balance. This includes moving and/or replacing some of the in-rack Power Distribution Units (PDUs).

Software Repository (no user action needed)

Our NIST SP800-171 conforming secure cluster has a specific need for a dedicated copy of the PACE software repository. We have previously completed an initial replication of the files and will simply point the nodes to the replica during the maintenance window.

 

 

 

 

Powered by WordPress