0830
Panasas engineers on their way to be on-site. Expecting tracking information for Force-10 switch replacement.
We re-configured the networking for the Panasas shelves bypassing part of the Force-10 switch. The Panasas realm appears stable but unavailable via the Infiniband. We will continue to work with Panasas on this resolution.
1430
The replacement Force-10 switch has not been received. Panasas and PenguinComputing (now both on site) are investigating. Alternate networking topologies are being explored with the campus backbone team and level 3 Panasas engineers. The PACE staff continues to work as quickly as possible in an effort to restore service for this critical resource. We are investigating some ways we can restore service access to the files but at a temporarily reduced capacity until a final solution may be obtained. We are also investigating how we may bring access to your files to the head nodes so you may be able to access them or copy them as you need though, perhaps, at a reduced speed than normal.
These possibilities are being examined by PACE, OIT network, Panasas and Penguin Computing personnel. We will continue to update you here with information as it becomes available.
1850
The replacement Force-10 switch has not been received and will not be on-site until tomorrow morning. We have a FedEx tracking number and will monitor it’s progress. Some NFS-access to the Panasas has been restored by creating an alternate network topology. While not ideal, this will allow us to bring a few parts of the clusters back into operation. However, at this time, we are still unable to access the Panasas from most of the systems and head nodes. We continue to work as quickly as possible to restore service with this critical resource. We can confirm at this time, no files have been lost or corrupted through all of the actions to date. All that is left is to restore access to them.
Panasas engineers remain on-site and and engaged and will continue working on the panfs access problem.