PACE A Partnership for an Advanced Computing Environment

April 29, 2021

PACE Advisory Committee Assembled

Filed under: Uncategorized — Semir Sarajlic @ 10:58 am

Dear PACE research community, 

We are pleased to announce that the faculty-led PACE advisory committee was formed and assembled on March 30, 2021. The PACE Advisory Committee is a joint effort between the EVPR and OIT to ensure that shared research computing services are both meeting faculty needs and resourced in a sustainable way.   The committee consists of a representative group of PACE and faculty members, encompassing a wide range of experience and expertise on advanced computational and data capabilities provided by OIT’s research cyberinfrastructure.  An important goal of the committee is to provide essential feedback which will help continuously improve this critical service. The committee will meet regularly and: 

  1. Function as communication channel between the broader research computing community and PACE.
  2. Serve as a sounding board for major changes to the PACE infrastructure
  3. Maintain an Institute-level view of the shared resource
  4. Help craft strategies that balance the value and benefits provided by the resources with a sustainable cost structure in the face of ever-increasing demand.

PACE Advisory Committee Members: 

  • Srinivas AluruIDEaSDirector (ex-officio) 
  • Omar Asensio, Public Policy 
  • Dhruv Batra, Interactive Computing/ML@GT 
  • Mehmet Belgin, PACE 
  • Annalisa Bracco, Earth and Atmospheric Sciences 
  • Neil Bright, PACE 
  • Laura Cadonati, Physics 
  • Umit  Catalyurek, Computational Science and Engineering 
  • Sudheer ChavaScheller College of Business 
  • Yongtao Hu, Civil and Environmental Engineering 
  • Lew Lefton, EVPR/Math (ex-officio) 
  • Steven Liang, Mechanical Engineering/GTMI 
  • AJ Medford, Chemical and Biomolecular Engineering 
  • Joe Oefelein, Aerospace Engineering 
  • Annalise Paaby, Biological Sciences 
  • Tony Pan, IDEaS 
  • David Sherrill, Chemistry and Biochemistry 
  • Huan Tran, Materials Science and Engineering  

If you have any questions or comments, please direct them to the PACE Team <> and/or to Dr. Lew Lefton <>.  

All the best, 

The PACE Team 

April 28, 2021

Parallel Computing with MATLAB and Scaling to HPC on PACE clusters at Georgia Tech

Filed under: Uncategorized — Michael Weiner @ 4:08 pm

MathWorks and PACE are partnering to offer a two-part parallel computing virtual workshop, taught by a MathWorks engineer, to PACE users and other members of the Georgia Tech community.

During this self-paced, hands-on workshop, you will be introduced to parallel and GPU computing in MATLAB for speeding up your application and offloading computations.  By working through common scenarios and workflows, you will gain an understanding of the parallel constructs in MATLAB, their capabilities, and some of the issues that may arise when using them. You will also learn how to take advantage of PACE resources, which are available to all researchers at Georgia Tech (including a free tier available at no cost), to scale your MATLAB computations.

Register by noon on May 14 at

·      Speeding up programs with parallel computing
·      Working with large data sets
·      GPU computing
·      Scaling to PACE clusters (Phoenix, Hive, ICE, or Firebird)

This virtual workshop will be held in two parts:
Part I, Tuesday, May 18, 1-4 PM, will focus on speeding up MATLAB with Parallel Computing Toolbox.
Part II, Tuesday, May 25, 1-4 PM, will focus on running MATLAB parallel code on PACE clusters.

Who should attend?
PhD students, post docs and faculty at Georgia Tech that want to (Part I) use parallel and GPU computing in MATLAB, and (Part II) scale their computations to take advantage of PACE resources.

·      Basic working knowledge of MATLAB
·      Access to the Georgia Tech VPN. You do NOT need to be a PACE user, and all participants will receive access to PACE-ICE for hands-on activities.

Please contact PACE at with any questions.

April 23, 2021

PACE Update: Compute and Storage Billing

Filed under: Uncategorized — Semir Sarajlic @ 7:35 pm

Dear PACE research community,

During our extended grace period, nearly 1M user jobs from nearly 160 PI groups have completed,  consuming nearly 40M CPU hours on the Phoenix cluster. The average wait time in queue per job was less than 0.5 hours, confirming the effectiveness of the measures to ensure fair use of the Phoenix cluster to maintain an exceptional level of quality of service.

With the billing for both storage and compute usage in effect as of April 1st, we are following up to provide an update on a few important points.

Compute billing started April 1: 

Throughout March, we’ve sent communications to all PIs  in accordance with the PACE’s new cost model, including  the amount of compute credits  based on the refreshed compute equipment as part of the migration to Coda data center and/or  recently purchased equipment from FY20 Phase 1/2/3 purchase(s).

PACE has identified and fixed some discrepancies since our initially communicated information as part of our compute audit, which included resources purchased but not provisioned on time. We apologize for this oversight and encourage users to run pace-quota command to verify the updated list of charge accounts. We’ll follow up with the impacted PIs/users in a separate communication.

Please note that most school-owned accounts, as well as those jointly purchased by multiple faculty members, will show a zero balance, but you can still run jobs with them. We are working to make the balances in those accounts visible to you.

As of April 1, all the jobs that run on Phoenix and/or Firebird clusters will be debited/charged to the provided charge account (e.g., GT-gburdell3, GT-gburdell3-CODA20), and a statement will be sent to PIs at the start of May.

This does NOT necessarily mean that you must immediately begin providing funding to use Phoenix. All faculty and their research groups have access to our free tier. Additionally, if you had access to paid resources in Rich, they have been refreshed with an equivalent prepaid account intended to last for 5 years. 

Project storage billing started on April 1: 

As announced, quotas to Phoenix project storage were applied on March 31 based on PI choices as part of our storage audit.   Users may run pace-quota command to check their research group’s utilization and quota at any time.  For further information about the Phoenix storage, please see our documentation.  April is the first month where storage quotas incur charges for PIs who have chosen quotas above the 1 TB funded by the Institute.

Showback statements sent to PIs: 

Throughout March, we sent out “showback” statements for the prior months’ usage on the Phoenix cluster, which covered the usage for Oct 2020 through Feb 2021.   We are in a process of sending the March 2021 showback statements that will also include a storage report.  Overall, these statements provided PIs with an opportunity to review their group’s usage and follow up with PACE as needed.  Explanations for each of the metrics can be found in our documentation.

No charges were incurred for usage during the grace period, so the showback statements are solely for your information and to guide your usage plans going forward. 

User account audit completed: 

Users of ECE and Prometheus resources migrated in Nov 2020 did not have all their charge accounts provisioned  during their groups’ migration.  Since then, we have provided access to these additional accounts for the impacted users.  We apologize for any inconvenience this may have caused.   Also, as part of our preparation to start billing for computation, on Feb 8, the PACE team sent out a notification to PIs to conduct a review of the job submission accounts and corresponding user lists.  We appreciate PIs input throughout this process, and if any changes have occurred in your group since then, or if you would like to add a new user(s) to your account(s), please don’t hesitate to send a request to  Users may run the pace-whoami command to see a list of charge accounts they may use.

Additionally, we have created a blog page for the frequently asked questions we have received from our community after the end of the extended grace period on March 31, which we would like to share with you at this time.

If you have any questions, concerns or comments about the Phoenix cluster or the new cost model, please direct them to

Thank you,

The PACE Team

FAQ after the end of the grace period on the Phoenix cluster

Filed under: Uncategorized — Semir Sarajlic @ 6:25 pm

The following are frequently asked questions we have received from our user community after the end of the extended grace period on March 31 in accordance to the new cost model., which we are sharing with the community:

Q: Where can I find an updated NSF style facilities and equipment document? 

A:  Please see our page at  

Q: I had a cluster I bought back in 2013, can I access this cluster? 

A: No.  We have decommissioned all clusters from Rich datacenter as part of the Rich to Coda datacenter migration plan.   As part of our earlier communication to PIs, if a PI owned a cluster in Rich datacenter, they received a detailed summary of their charge account(s) for the Phoenix cluster that included the amount of compute credits allocated to their account based on the compute equipment that was refreshed.  To see your list of available charge account(s) and their credit balance, please run pace-quota on the Phoenix cluster. 

Q: I do not have funds to pay for the usage of the Phoenix cluster at this time, can I get  access to Phoenix at no cost? 

A: As part of this transition, PACE has taken the opportunity to provide all Institute Faculty with computational and data resources at a modest level.    All academic and research faculty (“PIs”) participating in PACE are automatically granted a certain level of resources in addition to any additional funding they may bring. Each PI is provided 1TB of project storage and compute credits (68) equivalent to 10,000 CPU-hours (per month) on a 192GB compute node. These credits may be used towards any computational resources (e.g., GPUs, high memory nodes) that are available within the Phoenix cluster. In addition, all PACE users also have access to the preemptable backfill queue at no cost.   

Q: Do I need to immediately begin providing funding to use Phoenix beyond the free tier? 

A: Not necessarily. If you had access to paid resources in Rich, you now have access to a refresh CODA20 account with an existing balance, as described to each faculty owner. The number of credits in that account is equivalent in computational power to 5 years of continuous use of your old cluster in the Rich Datacenter  

April 2, 2021

PACE Archive Storage Update and New Allocation Moratorium

Filed under: Uncategorized — Semir Sarajlic @ 3:00 pm

Dear PACE Users,

We are reaching out to provide you a status update on PACE’s Archive storage service, and to inform you about the moratorium for new archive storage user creation and allocations that we are instituting effective immediately.  This moratorium on new archive storage deployments decreases any potential negative impacts on transfer and backups due to the potential for large influx of new files.

What’s happening and what we are doing: Currently, the original PACE Archive storage is hosted on vendor hardware that is at limited support capacity as the vendor has ceased operations.  PACE has initiated a two phase plan to transfer PACE Archive storage from the current hardware to a permanent storage solution.  At this time, phase 1 is underway, and archive storage data is being replicated to a temporary storage solution.   PACE aims to finish the archive system transfer and configuration of this phase by May Maintenance Period (5/19/2021 – 5/21/2021).   The phase 1 is a temporary solution as PACE explores a more cost-efficient solution that will require a second migration of the data to the permanent storage solution, which will be part of the phase 2 of the plan, and we will follow-up with details accordingly.   

How does this impact me:  There is no service impact to current PACE archive storage users.   With the moratorium in effect, new users/allocations requests for archive storage are delayed until after the maintenance period.  New requests for archive storage may be processed starting 05/22/2021.  

What we will continue to do:  PACE team will continue to monitor the transfer of the data to the NetApp storage, and we will report as needed. 

Please accept our sincere apology for any inconvenience that this temporary limitation may cause you.  If you have any questions or concerns, please direct them to


The PACE Team

Powered by WordPress