PACE A Partnership for an Advanced Computing Environment

[Update – March 25, 2021 – 3:07pm] 

This is a follow up to yesterday’s message about the campus network connectivity issues that impacted PACE.  By 3:24pm yesterday, OIT’s network team was able to rapidly resolve the connectivity issues, which was quickly updated on the status page link that was provided earlier.   Analysis of the incident, which was made available to us at a later point, revealed that the issue was identified to be a network spanned into the Coda data center from the Rich building that experienced a spanning tree issue (a network loop).   This caused a cascade of issues  with core network equipment due to specific failure scenario that caused widespread connectivity issues across the campus.  OIT’s network team resolved the issue with the affected network that resolved the other connectivity issues affecting the campus.   OIT’s network team will conduct further investigation regarding this to prevent future occurrence.

Since yesterday at about 3:30pm, all PACE users should have been able to access PACE managed resources without issues.  There was no impact to running jobs unless they required external resources (outside of PACE).   If you have any questions or concerns, please direct them to pace-support@oit.gatech.edu

[Original Message – March 24, 2021 2:48pm]

Dear PACE Users,

At around 2:30pm, OIT’s network team reported connectivity issues. This may impact users’ ability to connect to PACE managed resources at Coda, such as Phoenix, Hive, Firebird, PACE-ICE, CoC-ICE and Testflight-Coda. Currently, the source of the problem is being investigated, but at this time, there is no impact to running jobs unless they require external resources (i.e., from the Web). We will provide further information as it’s available.

Please refer to the OIT’s status page for the developments on this issue: https://status.gatech.edu/pages/incident/5be9af0e5638b904c2030699/605b8495e2838505358d3af3

If you have any questions or concerns, please don’t hesitate to contact us at pace-support@oit.gatech.edu

We apologize for this inconvenience.

Best,

The PACE Team

[Update 3/31/21 10:45 AM]

As previously announced, we have applied quotas to Phoenix project storage today based on each faculty PI’s choice. You can run the pace-quota command to check your research group’s utilization and quota at any time. Project storage quotas on Phoenix are set for a faculty member’s research group, not for individual users. Each user also has 10 GB of home storage and 15 TB of short-term scratch storage (not backed up).

You can learn more about Phoenix storage at http://docs.pace.gatech.edu/phoenix_cluster/storage_phnx/. April will be the first month where storage quotas incur charges for PIs choosing quotas beyond the 1 TB funded by the Institute.

If your research group exceeds your quota, you will not be able to write to your project storage, and jobs running in your project storage may fail. We are in the process of directly contacting all users in storage projects over quota today.

Please contact PACE Support with any questions about Phoenix project storage quotas. Faculty may also choose to contact their PACE Research Scientist liaison.

[Update 3/24/21 12:30 PM]

We’d like to remind you that storage quotas on Phoenix project storage will be set one week from today, on March 31.

You can run the pace-quota command to check your research group’s utilization at any time. Your PI/faculty sponsor is choosing the quota that will be set, based on your group’s storage needs.  You can learn more about Phoenix storage at http://docs.pace.gatech.edu/phoenix_cluster/storage_phnx/.

After quotas are set on March 31, we will notify all users, and you will be able to see your quota via pace-quota.

Users and faculty should contact their PACE Research Scientist liaison or PACE Support with any questions about Phoenix project storage quotas.

[Original Post]

As part of completing the migration to Phoenix, we will set quotas on Phoenix project storage on March 31, ending the period of unlimited project storage. Project storage quotas on Phoenix are set for a faculty member’s research group, not for individual users. Each user also has 10 GB of home storage and 15 TB of short-term scratch storage (not backed up).

All users can run the “pace-quota” command on Phoenix to see their research group’s storage usage. Quotas will generally show as unlimited (zero) until March 31.

Please contact us at pace-support@oit.gatech.edu with any questions about Phoenix project storage.