MATLAB based Massive Catastrophe Modeling in Minimal Time

In the Reinsurance Industry, it’s vital to be prepared for the worst. A massive natural catastrophe can happen anywhere, without prior notification, but some disasters are more predictable than others; the probability of an earthquake, typhoon or tornado depends on the geographic region.

Risk factors for catastrophe models are changing all the time based on the climate change, collected data, and continuous research. Managing the catastrophe risk requires frequent calibration and systematic validation of the simulation models. In general, this means large computations that use large amounts of collected data.

Computing a big catastrophe model is a time consuming project. It may take weeks to perform a single analysis run on a local cluster. Enabling a more interactive computational process would require more computational capacity, but building a new, bigger datacenter to serve the peak loads is often too expensive. And because of the continuously growing business requirements, scalability may still be an issue. The lack of affordable scalability is the main reason why reinsurance companies have chosen to use Techila when processing big compute loads.

This post looks at one of these cases.

MATLAB is a recognized tool, used in building and simulating catastrophe models. It was also used in this case. You can read more about the benefits of MATLAB in catastrophe modeling in this article about a Swiss Re user case: https://www.mathworks.com/company/user_stories/swiss-re-calculates-potential-loss-from-natural-disasters.html

The customer was using MATLAB in their own data center to solve the models. MATLAB’s parfor and MDCS (MATLAB Distributed Computing Server) were used, but they had scalability issues. Techila analyzed the issues and compared the scalability to a similar run that was performed using Techila distributed computing engine and Techila’s cloudfor. In a quick test, the problem in the scalability was already clearly seen.

Techila's solution enabled the existing MATLAB code to be migrated from MDCS to a Techila environment with minimal changes on a couple of code lines by replacing parfor with cloudfor. After the introduction of cloudfor, it was also possible to run the same code in on-premises as well as in the leading cloud environments without any further changes. This benefited the customer who did not want to build a new, bigger datacenter to serve the peak loads, but wanted to enable scaling up the computational throughput on demand by bursting the processing to cloud.

The optimal solution for the customer was a so-called hybrid IT solution, which allowed using resources from a public cloud along with the existing local cluster during peak workloads.

Performing the model calibration uses many years of CPU time. In the local cluster environment this translates to a project that takes several weeks. The goal of the customer was to get the model calibrated in a matter of hours. To reach this goal without building a new, large data center, computational capacity would need to be leveraged from one of the public cloud providers.

The reinsurance company decided to use Amazon Web Services Elastic Compute Cloud (AWS EC2) for the computations. The decision was based on the reliability and stability of the deployment as well as AWS’s ability to respond to their compliance requirements. The scalability of the cloud provider was measured by performing test deployments of 10.000 CPU. AWS was able to respond to the deployment requests consistently and in a reasonable time, and the requested CPU core count was reached even at an on-demand model without problems.

The computation in AWS EC2 was performed by using 1250 c3.2xlarge instances. Each c3.2xlarge has 8 CPU cores. The input data for the computation was stored in AWS S3 cloud storage. The input data set was some gigabytes, and it was split into 100.000 parts to support enhanced data parallelism and scalability of processing.

As soon as the individual parts were processed, the results were transferred from S3 cloud storage to the analyst’s computer for the post-processing. Because Techila enabled real-time post-processing of result data, the analyst was able to interactively observe any potential issues in the process at the earliest possible phase.

Techila distributed computing engine enabled completing the model calibration in the 2 hour target time. As soon as the last computational jobs were completed in the cloud, Techila distributed computing engine’s management features shut down the idling cloud instances one at a time. The customer was very pleased with the scalability and the cost-efficiency of the Techila distributed computing engine solution along with cloud-based processing. It was also recognized that if the data grows, or if there is a need for even faster model calibration, the problem could be solved even faster simply by scaling more cloud computing resources.

The solution was ready to be taken into production.

For more information about Techila Technologies Ltd., please see https://www.techilatechnologies.com/