CMFT - Cloud based Managed File Transfer Securely
File transfer was one of the first integration pattern in the EAI space. Even today, 55 percent of systems integration relies on successful file transfers. File-based data movement is the lifeblood of business and Banking. With the volume and size of files growing year-over-year, customer and supplier needs will drive complex networks and cybersecurity threats more than ever before.
However, studies show that 88% of business have difficulty moving big data quickly and efficiently.
Business faces many challenges? such as data security gaps, lack of visibility when a problem occurs, timely manual recovery from failures and costly SLA fees due to poor performance
Security weaknesses Critical data needs to remain secure and under your control, but FTP was not designed with secure file transfer in mind and SFTP lacks security controls to handle today’s cyber threats. FTP clients are common and free, giving every hacker the tools necessary to attempt to breach your critical systems. – These security weaknesses and other vulnerabilities make it easy to intercept FTP-based file transfers.
Even minor lapses in security can damage your reputation, send your stock value plummeting and result in massive costs
?Complex and IT Overburden?Encryption afterthought requires extra steps and IT expertise, making it difficult, expensive and time-consuming to send files safely.
Lack of control Bank? do have cutt off periods for some of the transaction where as other 24 windows. FTP sends files on a firstcome, first-served basis. you can’t prioritize critical transfers, balance processing windows, or respond to immediate business needs
Blind Spots When a file is delayed or isn’t transferred at all, you need to be notified in real time so you can proactively correct problems before they impact downstream business activities. You can’t fix what you can’t see, and with FTP you only discover failures when you feel the pain.
Manual recovery from failures -?FTP can’t recover a failed connection automatically, so you must restart the process manually. – FTP doesn’t include checkpoint restart, requiring you to resend entire files regardless of how much was previously sent.
Unable to cope up with the Growing demand.? Over time, most organizations realize they are no longer able to scale FTP service.
Managing workloads in multi cloud To deploy more than just a handful of services or applications on cloud, it is tedious and time-consuming to set each one up manually. Not to mention, having to configure each cloud resource by hand also leaves you at a much higher risk of making errors or introducing inconsistencies.
What We need
Cloud Managed file transfer (CMFT)?
Now let us see how CMFT provides a Secure File Transfer that comes with a unified file transfer platform so you can share and track mission-critical information movement within the Bank/Enterprise and across your partner network with peace of mind. Gain the benefit of the most trusted managed file transfer solution.
Cloud Managed file transfer (CMFT) is a technology platform that allows organizations to reliably exchange electronic data between systems and people in a secure way to meet compliance needs. These data movements can be both internal and external to an enterprise and include various types, including sensitive, compliance-protected or high-volume data. It can be offered as software or as a service and may include a single pane for visibility and governance.
MFT is a more reliable and efficient means for secure data and file transfer, outpacing and outperforming applications such as file transfer protocol (FTP), hypertext transfer protocol (HTTP), secure file transfer protocol (SFTP) and other methods.
At its face, we front with SFTP Gateway. its a solution that? provides and configures SFTP access to limitless and reliable cloud storage locations, like S3. It includes a web interface and REST API that simplifies user management, folder permissions and instance administration, whether you're supporting a single user or thousands.
?Infrastructure automation /Infrastructure-as-Code (IaC)?
?This is based on the best practices such as IAS cloud formation. CloudFormation is an infrastructure automation platform for AWS that deploys AWS resources in a repeatable, testable and auditable manner. infrastructure automation or Infrastructure-as-Code (IaC) tool and a cloud automation solution
AWS CloudFormation allows creation and deployment using the console, CloudFormation Designer, and the AWS command line.
It has a concept of CloudFormation Hooks and share details on our own CloudFormation Hook? The HA CloudFormation templates automatically provision the network load balancer, target groups, EC2 auto scaling groups, launch configuration, and RDS instance. In order to implement HA, you will need to use the CloudFormation templates provided.
?SFTP Gateway is an SFTP server that provides real-time access to files stored on cloud storage. Users can move files to and from cloud platforms using an SFTP client like FileZilla or WinSCP. Behind the scenes, SFTP Gateway uses a software layer that translates the SFTP protocol to AWS SDK commands.
?Self Management: CMFT comes with a web admin UI for managing parnter users and their folders. The new folder management feature lets you set up flexible file sharing scenarios.?
?Self managed SSH Keys
Flexible to have partner provided or generation on demand
China Wall between the partners/ tenants
?By default, CMFT creates a /users folder. Each new SFTP user is given a home directory under the "users" folder, which looks like:
/users/partnerA/
/users/partnerB/
SFTP users have permissions to their Home Directory. And they cannot traverse outside their Home Directory, because they are chrooted. The default permissions are read/write, and SFTP Gateway lets you override these permissions (e.g. read-only).
No local disk All files and folders as seen from the SFTP client are objects on S3.There is no local storage on the Linux file system. Everything you see is a live view of S3.
File Integrity checks - MD5 hash validation
As an extra layer of protection, you can use MD5 hash validation to ensure that your file is not corrupted during the SFTP transfer process. SFTP Gateway will run an MD5 hash validation. If it passes, the actual file gets uploaded to S3, and the .md5 file is discarded.
Security and Compliance:
Encrypting local server data at rest
For compliance reasons, you may need to encrypt the data at rest when files are stored locally on the SFTP Gateway instance. These files could reside in the user's local, shared, or downloads directories.
EC2 - mount an encrypted EBS volume onto /home. This involves 4 main steps:
S3 Server Side Encryption
You can encrypt your files at rest when stored on S3. There are three encryption options, but it's highly recommended to use SSE-S3 because it's by far the easiest to configure and manage
Restricted Security Groups
?SFTP Gateway supports both key-based. Data is encrypted in transit and at rest. An independent third-party security audit of SFTP Gateway provides confidence your data is moving safely to the cloud.
?Auto Scaling We? implement an Auto Scaling group (ASG) running behind an Application Load Balancer (ALB) to manage an adequate number of worker EC2 instances to respond to the varying workload. Here we use a CPU utilization based target scaling policy to manage the worker EC2 instances.
Security Group to be associated with the ALB, so that it allows incoming HTTP web traffic on ports
领英推荐
Add and Configure the group size and scaling policies as follows:
Set the desired, minimum and maximum capacity of your Auto Scaling group like so:
? ? Set Desired capacity to 1,
? ? Set Minimum capacity to 1
? ? Set Maximum capacity to 3
Scaling Policy : Configure scaling policy to use a Target tracking scaling policy with Average CPU Utilization as the Metric Type. Set the Target value as 40 and the instance warm-up time as 300 seconds.
CST - CLoud Stress Testing
install the stress utility on the EC2 instance
Invoke the stress test on the instance using the following command:
The instance sees a spike in the CPU Utilization metric which shows up on the EC2 tab of the Monitoring section of the Auto Scaling group within a timespan of 5-10 minutes.
The spike in the CPU utilization triggers the target tracking scaling policy as the monitoring alarm breaches the target value of 40% (set in Task 3). This results in the launch of two additional EC2 instances as seen in the Activity tab. Three EC2 instances can be also be seen under the Running instance state
Serverless workflow orchestration?
During file processing, we need to stitch a number of services? such decryption, validation, conversion, exception handling, retries etc The challenge comes when such a large number of services all need access to various parts of a shared state. To operate these services effectively, the teams must also be able orchestrate the flow of data through all application services in a single place, and this is exactly what AWS Step Functions handles. Step Functions has become a crucial piece of the Serverless ecosystem because of all the state and data management needed to keep Serverless systems working effectively at scale.
The fundamental value of AWS Step Functions lies in the easy orchestration of applications that require interconnecting multiple Serverless functions. If you have business processes that require a combination of multiple decoupled Serverless applications to produce their end result, Step Functions could be the right choice for easy orchestration.
?AWS Step Functions makes the life of a Serverless developer easier by allowing you to quickly create complex sequences of tasks in AWS, while taking on the error handling and retry logic and allowing you to decouple your application’s business logic from its orchestration logic. Below, we go over the specifics of how Step Functions can help you.
Quickly create complex sequences of tasks Orchestrating a sequence of ten individual Serverless applications, managing their retries, and debugging any failures can be extremely challenging. As you add even more functions, the complexity of managing them grows exponentially.
With its graphical interface and built-in operational controls, Step Functions manages the sequencing of the tasks for you and removes a large operational burden from your team.
Manage state between executions of various stateless functions For many Serverless workflows, setting up queues and databases for communication between all the Serverless services can be time-consuming and error-prone, not to mention that not all use cases call for a real database or a real queueing system.
Step Functions makes it easy to set up state management early on, and it continues to work well as your application scales and you add more services into the mix.
?Decouple application workflow logic from business logic Here we have another best practice of Serverless development. Adding workflow logic to applications that should only handle business logic increases the complexity of the applications and can easily generate issues. In addition, managing state separate from business logic allows developers to retain clarity when working on a Serverless system.
More efficient workflows with parallel executions When you’re managing the state of various functions yourself, it can be challenging to obtain high performance from the system. Some orchestration pieces might only be able to process one task at a time, which could slow down the entire application.
Observability
Observability is all about the data. Efficient ingestion and storage of metrics, logs, and traces is the foundation of Elastic Observability and allows you to monitor and visualize your entire AWS ecosystem from infrastructure to applications, accelerating the adoption of cloud. Lay the foundation of your unified observability solution on AWS and then that same observability data can be applied to security use cases.
We stream the logs to an Amazon ES cluster in near-real time, through a CloudWatch Logs subscription. When the logs are streaming to the Amazon ES cluster, you can access the Kibana endpoint to visualize the data.
On your Amazon ES console, after a few minutes, you can see activities in the Key performance indicators section. The following screenshot shows an increase in Indexing rate.
From Discover tab to add specific fields as filters and search for them. In the following screenshot, I selected fields specific to error events logged in CloudTrail to find the issues.
Key Aspirations
Adoption of IaC tools offer a range of benefits that make cloud service deployment and management faster and more efficient.
Deployment speed
This approach leads to much faster deployment than you could achieve if you had to manually set up each deployment by running commands on the CLI or pressing buttons in the AWS console.
Scaling up
By keeping CloudFormation templates on hand, you will know that you can add more virtual machine instances or storage space, for example, at a moment's notice if your applications experience increased traffic and you need to scale your environment up.
Alternatively, when demand decreases and you want to scale down to save money, you can take some of your deployments offline while still retaining the ability to redeploy them quickly using CloudFormation when demand increases.
Seamless Service integration
Graphically Managing multiple services through a Step functions workflow ? makes it easy to integrate AWS services as you build out a complete cloud environment. Orchestrating a sequence of ten individual Serverless applications, managing their retries, and debugging any failures can be extremely challenging. As you add even more functions, the complexity of managing them grows exponentially.
With its graphical interface and built-in operational controls, Step Functions manages the sequencing of the tasks for you and removes a large operational burden from your team.
?Consistency
With CloudFormation templates +? Step function based workflow to define and deploy AWS resources, you can apply precisely the same configuration repeatedly. In this way, CloudFormation ensures that your applications and services will be consistent and identical, no matter how many instances you create.
Security
As long as you design your CloudFormation templates to be secure, you do not need to worry that an engineer who deploys resources will forget to turn on important access control, for example, or leave data exposed to unrestricted, public access.
Easy updates
In addition to deploying new resources, you can apply changes to existing resources with CloudFormation templates. This ability simplifies the process of, for example, adding more storage to a fleet of ec2 instances or changing access control rules.
Auditing and change management
When you use CloudFormation templates +? Step function to manage your infrastructure, you can track changes based on which templates you have applied and how they change over time. Change tracking in CloudFormation means that you will be able to determine how your AWS services and resources have changed over time without looking through logs to reconstruct the timeline of updates.