About 8:30pm this evening, one of the PACE systems that provides services to the GPFS files to headnodes and other PACE internal systems failed. When this happens, users may see the message “stale file handle” or you may notice there are no files under the /gpfs directory. This is a temporary condition that should be fixed shortly.
Please note: All files that were already written and all files accessed or written by any compute node are unaffected. However, if you were in the process of editing a file on a headnode, only your most recent changes may be unavoidably lost. In addition, any process you may have had running on a headnode system using these files may have been killed due to this failure.
To prevent this from recurring, PACE had ordered and very recently received a new computer to replace the system that failed this evening. Our staff will undertake the testing and replacement as soon as possible and we will post an announcement here once the new system is in service.
We apologize for this inconvenience and thank those users who let us know quickly.