Limitless scalability, on-demand, and reliable: 5 steps to set up high-performance computing (HPC) in the cloud.

When it’s set up correctly, cloud infrastructure can offer incredible performance and cost-savings for a multitude of projects. But what about those high-performance computing needs in niche and innovative industries?

Strangely, despite the potential benefits, many of the companies in those industries still use hard-to-maintain on-prem solutions – and we’d like to help change that.

In this newsletter we’re going to outline the 5 key steps needed to ensure cloud HPC delivers all the benefits of cloud infrastructure – while still protecting things like priceless intellectual property and industry secrets.

?

Why use cloud high-performance computing (HPC)?

The flexibility of cloud infrastructure means you can create highly intensive compute instances on-demand, with a limitless ability to scale.

From a cost perspective, this has a big advantage compared to running your own HPC on-prem. Instead of investing in machines and paying for maintenance every day, you only pay for the resources you use.

There are other benefits too: easy management, resilient, and secure. You’re also never limited by the physical computing power of your own machine – so your innovation is never held back.

Numerous industries use HPC capabilities to support their project development by running complex, realistic simulations or for sophisticated engineering designs.

?

For example:

• Aerospace
• Advanced weapons systems
• Chip design
• Medical device design
• Materials science
• Technical engineering
• Pharmaceuticals
• Civil
• Smart city design
• Digital twins

?

Of course, many established players in these industries already have sunk investment in their own on-prem solutions.? So, it may seem easier to just ‘stick with what you know,’ despite the hidden costs of this attitude.

However, startups don’t have this problem. They can go immediately to the cloud, and instantly gain nearly infinite computing power at a very low cost. Plus, they don’t have the constant resource drain for maintenance.

So, what’s needed to get this up and running?

5 steps to set up high-performance computing in the cloud

Want to know more? Register for our webinar:

Step 1: Outline your needs

You must clearly define what computational power and processing capabilities are needed for your organization and its typical use cases. This helps make key decisions about system architecture. AWS EC2 can support a wide range of instances using different processor technologies. Each is better for certain tasks. P4d is good for machine learning, C5 instances are good for cost-effective batch processing, Trn1 is good for generative AI or deep learning, and HPC6id is a better choice for finite element analysis. Next to computational power, you need to think about things like storage needs, workload management, and if you need tools like SageMaker or Lambda.

?

Step 2: Set up your AWS environment

This stage makes your cloud HPC easy to manage in the long term. Start by creating a landing zone with all your environments, accounts, and permissions. Then create your network resources like VPC, subnets and security groups. Considering the heavy data flows, select an AWS Region that’s geographically close to your main location and make sure your network setup is not scattered too much around different regions or availability zones. This reduces latency and cost of sending traffic among AWS regions and availability zones. If you’re using AWS Research and Engineering Studio (RES), then a lot of the setup is streamlined with templates but you still need to do certain tasks like setting up provisioning users, increasing service quotas, and deploying RES resources in CloudFormation.

?

Step 3: Security

Using an intranet in your virtual private cloud (VPC) is like ‘virtually airgapping’ your HPC, making it more secure. It also helps you to manage your network effectively. AWS Directory service allows you to apply your on-prem Active Directory (AD) to manage access. Additional tools from AWS add even more security: secrets management (via AWS Secrets Manager), Network Access Control Lists (ACLs), and Security groups, for example.

?

Step 4: Manage resources

You can control resource budgets directly from the AWS Billing console, but you need to activate RES environment tags first. This takes a few days to appear, but once they’re visible you can use these tags to set up budgets for each project and create alerts.

?

Step 5: Create user environments

AWS RES Environment management and Session management allows you to create dedicated virtual desktops for each user. Virtual desktops can be securely accessed via a web browser and have built-in security features that protect your data. These are customized with all the tools each user needs (and is allowed). Each project can have multiple environments.

?

Looking for more in-depth material, visit our blog:

Ok, so now what??

The steps above outline the basic setup that any high-performance computing project will need. For the next part, you really need to look closely at the specific requirements of each project or organization.? Your storage needs, for example, will depend a lot on access patterns, backups, and the security required.

AWS FSx is a good option for most storage requirements as it works with a variety of third-party file storage systems. Because it’s fully managed, your data is also backed up and protected by replication and redundancy across multiple regions.

And for long term storage, AWS S3 offers a cost efficient way of storing data in “cold storage” with their Glacier offering.

In fact, you can find almost everything you need to manage your HPC requirements within the AWS ecosystem. But each situation is different, and this is where expert advice can prove its value by taking the shortest route to the best outcome, while avoiding all the pitfalls along the way.

?

Want to get your high-performance computing set up right from the start? Get in touch to discuss your needs.

?