GT Home : : Campus Maps : : GT Directory

Author Archive

[Resolved] Phoenix scheduler timeout

Posted by on Friday, 20 May, 2022

Summary: A timeout on the Phoenix scheduler prevented new jobs from beginning earlier today.

Details: A setting caused a timeout issue in the communication between the Torque and Moab portions of the Phoenix scheduler this morning, beginning at 10:20 AM. The PACE team restored communication between the services before 12:20 PM today.

Impact: During the intervening period, no new jobs could start. Running jobs were not interrupted, and submitting new jobs to queue remained functional. Commands such as “qsub” and “qstat” continued to work.

Thank you for your patience this morning. Please contact us at pace-support@oit.gatech.edu with any questions.

PACE Firebird Login Node Outages

Posted by on Wednesday, 27 April, 2022

[Update 4/27/22 5:45 PM]

The remaining headnode has been repaired, and service is restored. Thank you for your patience.

[Original Post 4/27/22 5:20 PM]

Summary: A storage server issue made headnodes for two projects on Firebird inaccessible. One has been recovered, while repairs are in progress on the second one.

Details: The storage server housing two Firebird projects had an NFS issue earlier today. The login nodes were impacted. The PACE team has repaired one project’s login node and is currently repairing the second that has a more complex issue.

Impact: Researchers on impacted projects are/were not able to log into Firebird today. Running jobs were not impacted, as only the login node is/was affected.

We apologize for the disruption. Please email us at pace-support@oit.gatech.edu with any questions.

Campus network disaster recovery testing June 10-13

Posted by on Wednesday, 27 April, 2022

Summary: Campus network disaster recovery testing will disable Phoenix, Hive, PACE-ICE, and COC-ICE from 5:00 PM on Friday, June 10, through 12:00 noon on Monday, June 13.  

Details: In accordance with USG security requirements, OIT will be conducting disaster recovery testing on the Georgia Tech campus network during the weekend of June 11, which will close access to most of PACE’s clusters as well as some other campus resources.  PACE’s Phoenix, Hive, PACE-ICE, and COC-ICE clusters will be impacted. Firebird and Buzzard will remain in production.  

Impact: PACE will set a reservation to prevent any jobs from running during the downtime. You will not be able to log in, access your data, nor run jobs during the outage.  

Longer jobs will be held until the testing is complete if their walltime request will not lead the job to conclude before the outage, just as they are during quarterly maintenance periods. Researchers who run long jobs should note the duration between PACE’s May maintenance period (May 11-13) and the testing period, beginning June 10. In particular, Hive researchers who submit 30-day jobs to the hive-nvme, hive-sas, or hive-nvme-sas queues should note that any 30-day job submitted after April 12 will not begin until at least June 13. Researchers are encouraged to submit jobs with reduced walltimes whenever feasible to make use of the cluster between maintenance and disaster recovery testing.  

Thank you for your understanding and cooperation during this campus network testing. Please contact us at pace-support@oit.gatech.edu with any questions or concerns. 

Phoenix scheduler outage

Posted by on Friday, 22 April, 2022

Summary: The Phoenix scheduler became nonresponsive yesterday evening and was restored at approximately 11:30 PM last night.

Details: Yesterday evening, the Torque resource manager on the Phoenix scheduler became overloaded, likely shortly after 7:30 PM. The PACE team restarted the scheduler and restored its function just before 11:30 PM last night.

Impact: Commands such as “qsub” and “qstat” would not have worked, so new jobs could not be submitted. Running jobs were not interrupted.

Thank you for your patience last night. Please contact us at pace-support@oit.gatech.edu with any questions.

Hive project & scratch storage cable replacement

Posted by on Friday, 22 April, 2022

Summary: Hive project & scratch storage cable replacement and potential for an outage

Details: A cable connecting the Hive GPFS device, hosting project (data) and scratch storage, to one of its controllers needs to be replaced, beginning around 11:30 AM Tuesday (April 26).

Impact: Since there is a redundant controller, no impact is expected. However, a similar previous replacement caused storage to become unavailable, so this is a possibility. If the redundant controller fails, your job may fail or run without making progress. If you have such a job, please cancel it and resubmit it once storage availability is restored. PACE will monitor Hive GPFS storage throughout this procedure. In the event of a loss of availability occurs, we will update you.

If you have any questions or concerns, please direct them to pace-support@oit.gatech.edu.

Phoenix scheduler outage

Posted by on Monday, 18 April, 2022

Summary: The Phoenix scheduler stopped launching new jobs on Friday evening and was restored at approximately 9:30 AM on Saturday.

Details: At some point after 8 PM on Friday evening, the node hosting the Moab workload manager of the Phoenix scheduler lost its network connection, leaving it unable to communicate with the rest of the cluster. The PACE team repaired the connection just before 9:30 AM on Saturday morning, and functionality was restored.

Impact: While jobs could be submitted via “qsub” and checked via “qstat”, no new jobs would launch but would instead remain queued. Moab commands such as “showq” would not have worked. Running jobs were not interrupted.

Thank you for your patience over the weekend. Please contact us at pace-support@oit.gatech.edu with any questions.

Phoenix scheduler outage

Posted by on Thursday, 14 April, 2022

Summary: The Phoenix scheduler became nonresponsive overnight and was restored at approximately 9:00 AM today.

Details: Last night, the Phoenix scheduler became nonresponsive, likely shortly after midnight. The PACE team restarted the scheduler and restored its function just before 9:00 this morning.

Impact: Commands such as “qsub” and “qstat” would not have worked, so new jobs could not be submitted early this morning. Running jobs were not interrupted.

Thank you for your patience early this morning. Please contact us at pace-support@oit.gatech.edu with any questions.

Hive project & scratch storage cable replacement

Posted by on Friday, 8 April, 2022

Summary: Hive project & scratch storage cable replacement and potential for an outage

Details: A cable connecting the Hive GPFS device, hosting project (data) and scratch storage, to one of its controllers needs to be replaced, beginning around 10:00 AM Tuesday (April 12).

Impact: Since there is a redundant controller, no impact is expected. However, a similar previous replacement caused storage to become unavailable, so this is a possibility. If the redundant controller fails, your job may fail or run without making progress. If you have such a job, please cancel it and resubmit it once storage availability is restored. PACE will monitor Hive GPFS storage throughout this procedure. In the event of a loss of availability occurs, we will update you.

If you have any questions or concerns, please direct them to pace-support@oit.gatech.edu.

[Resolved] Phoenix Charge Account Authorization

Posted by on Friday, 1 April, 2022

[Update 4/4/22 12:25 PM]

Summary: [Resolved] Free tier charge account balances did not reset on April 1. A manual reset was performed on April 4.

Details: The deleted Perl library that prevented job submissions last Thursday night and Friday morning also caused an error in the monthly reset of free tier charge account balances at midnight on Friday, April 1. Other accounts that reset on a monthly basis were not impacted. PACE manually reset all free tier account balances just before noon today.

Impact: Job submissions to free tier accounts over the last three days would have succeeded only if sufficient leftover balance from March remained. At this time, all free tier accounts have been reset to their full monthly allocation, and jobs run prior to the reset will not count towards April utilization. All faculty and their teams now have access to their full April free tier allocation. Researchers can run the “pace-quota” command to view their available charge accounts and balances.

We apologize for any disruption this may have caused. Please contact us at pace-support@oit.gatech.edu with any questions.

[Original Message 4/1/22 8:49 AM]

Summary: [Resolved] Phoenix users attempting to submit jobs have received an error message that they are not authorized for their charge account.

Details: Beginning yesterday evening, Phoenix users attempting to submit jobs have at times received an error message indicating that they are not authorized for charge accounts to which they should have access. PACE deployed a temporary repair at 6:45 PM yesterday. The issue recurred at midnight, and the temporary repair was again made at 8:15 AM today. We have now identified the root cause as a deleted Perl library on the scheduler and deployed a permanent fix.

Impact: At this time, researchers are again able to submit jobs. Please resubmit any rejected jobs with the usual charge account. Researchers can run the “pace-quota” command to view their available charge accounts and balances. No running jobs were impacted.

We apologize for this disruption. Please contact us at pace-support@oit.gatech.edu with any questions.

 

 

PACE Data Retention Updates

Posted by on Thursday, 31 March, 2022

PACE is nearing the conclusion of our user records and data audit, and we would like to share with you several updates to how PACE handles data stored on our storage systems and the role of schools in managing that data.

PACE takes responsibility for the storage of all data from our old systems in the Rich datacenter that has not been assigned to a current member of the faculty. All of this data has been inaccessible to all users since December 1, 2020. Any school that wishes to do so may request data of a former faculty member and thereby assume responsibility for the data and for the cost of continuing to store it on PACE (unless it is relocated), in accordance with the Data Retention Guidelines provided by the GT administration and shared with you at this time. If the school does not make such a request, PACE will cover the cost of storing this data until July 2024, then delete anything that has not been requested.

All data left on PACE by faculty who departed the Institute after July 1, 2021, will follow the Data Retention Guidelines. Under these guidelines, the faculty member’s former school will be responsible for the data and the cost of storing it on PACE, relocating it to another storage system, or determining it can be deleted while complying with any relevant regulations and contracts. The 1 TB provided to each school at no charge on Phoenix may be used to store these files. Schools also have the option of purchasing PACE Archive storage, which is designed for long-term retention of data that does not need to be used regularly on PACE compute clusters.

If you have any questions about PACE storage, please contact PACE at pace-support@oit.gatech.edu, and we’ll be happy to discuss it with you.