[Update – 11/03/2020 – 11:01am]
As of late last night, the slowness experienced on Phoenix storage was resolved. Thank you for your patience and understanding while we worked to address this issue.
What is happening and what we have done: In response to reports from users about slowness in accessing files on Phoenix’s Lustre storage, PACE team was able to replicate this issue during our investigation, and through our troubleshooting that included Lustre Metadata Service (MDS) reboots, we were able to resolve the slowness. The Phoenix Lustre storage is stable at this time, and there was no loss of user data during this incident.
What we will continue to do: PACE will continue to monitor the Phoenix storage out of abundance of caution, and we will update as needed.
Again, this issue did not impact any of the other resources in Coda and Rich Datacenter.
Thank you for your attention to this message, and we apologize for this inconvenience.
[Original Post – 11/02/2020 – 1:03pm]
Dear PACE Users,
PACE is aware of the slowness experienced on Phoenix’s storage. At this time, PACE is able to replicate the issue, and we are investigating the root cause of the storage issue.
What is happening and what we have done: We’ve received couple reports from users about slowness in accessing files from ‘data’ and ‘scratch’ directories on Phoenix’s Lustre storage. Some users are experiencing slowness in accessing their files, and running commands such as ‘ls’ or opening a file with ‘vim’ may be very slow. During our investigation, PACE team is able to replicate this issue, and we are investigating the root cause of the slowness with storage.
What we will continue to do: This is an active situation, and we will follow up with updates as they become available.
This issue does not impact any of the other resources in Coda and Rich Datacenter.
Thank you for your attention to this message, and we apologize for this inconvenience.
The PACE Team