[Update 2/20/20 4:40 PM]
Use of the scratch filesystem is restored. It appears that the automated migration task did run but could not keep up with the rate of scratch usage. We will monitor scratch for recurrence of this issue.
Please check any running jobs for errors and resubmit if necessary.
[Original message 2/20/20 4:30 PM]
Shortly before 4 PM, we noticed that PACE’s mounted GPFS scratch filesystem (~/scratch) is experiencing an issue that is preventing users from writing to their scratch directories. Any running jobs that write to scratch may experience failures due to write errors.
The scratch filesystem writes first to SSDs, and an automated task migrates data to another location when those drives near capacity. This task did not run as expected, prompting users to receive errors that scratch was full. We have manually begun the migration and will update the blog post when scratch is again available.
We apologize for this disruption. Please contact us at pace-support@oit.gatech.edu with any concerns.