We experienced a slowdown yesterday on all headnodes caused by an unusually large amount of user file operations from headnodes. All headnodes are virtual machines and connect through a network file system gateway to the GPFS filesystem. This gateway became overwhelmed by the user file operations, and subsequently slowed down all headnode file operations.
If you have heavy file operations (i.e. winSCP, FileZilla, SCP), please perform these by logging directly into iw-dm-4.pace.gatech.edu instead of a headnode. Additionally, other file operations such as tarring/zipping are best performed on compute nodes by submitting interactive or batch jobs, as well as iw-dm-4.
We’re actively looking into alternatives to virtual machine headnodes, and will provide more detailed updates as we approach our upcoming scheduled maintenance in November (via blog.pace.gatech.edu). If you have any questions, please email us at pace-support@oit.gatech.edu.