Modernize your Legacy SOAP Server using Amazon API Gateway and AWS Lambda

Modernize your Legacy SOAP Server using Amazon API Gateway and AWS Lambda

Daniel Abib, Enterprise Solutions Architect, FSI - LATAM

?

In this blog series, you will learn how to build a proxy solution to modernize a legacy server running SOAP protocol as well as a solution to integrate your legacy SOAP client to interact with modern applications using REST interfaces. This is part one of two blogs of the series.

The migration of legacy architectures using monolithic solutions or even solutions hosted in mainframes to modern solutions architectures that use micro services is already widespread in the market and being adopted by many companies of different sizes and business segments. If you are a developer, a software engineer or a solution architect working on enterprises in IT, you may be hearing buzzwords like architecture modernization or digital transformation multiple times a day. This blog will help you modernize a legacy application and allow you to continue in your digital transformation journey.

Some companies that were running businesses since the 90’s may still have some legacy systems running in production because sometimes they cannot modernize these solutions due to costs or maybe they don’t have the time, knowledge or even the source code to do so.

Usually, these legacy applications have deprecated methods to exchange data via flat files or using old (discontinued) APIs standards. The concept behind an API, as a programmable access gateway to request or send data to another application, exists since the early era of computer science, starting with RPC (Remote Procedure Call) in 1981. Though, the ways of implementing them have gone through huge transformations over the last years.

One important API protocol adopted by companies in early 2000’s was SOAP (acronym for Simple Object Access Protocol), a HTTP/HTTPS, JMS or SMTP based protocol for exchanging structured information in the implementation of web services in computer networks, using XML (Extensible Markup Language) as its payload.

The SOAP API is defined by a language called WSDL (Web Service Description Language). Using this language, you can define the endpoints, the required and optional information (parameters), receive, respond and describe the whole processes that can be accomplished to establish the communication of two computers. This flexibility allows the utilization of different programming languages to send and receive data.

More and more often, new technologies like REST, GraphQL and gRPC are been replacing some legacy protocols as SOAP APIs due to being modern, faster, lighter and easier to maintain. What must companies do with their legacy applications that are still important to run their business?

This blog post is about sharing a strategy to support companies that still have servers or clients’ applications using SOAP protocols to exchange data using REST APIs.

The good news is that most of the SOAP APIs and REST APIs are based on HTTP/HTTPS protocols (in fact, SOAP can also use SMTP, JSM and other protocols), and the impact to create the interoperability is not so challenging, but due to the fact that SOAP has limited HTTP/HTTPS methods (POST only), it fails to get the benefit of the wider range of HTTP methods available (such as GET, PUT, DELETE, HEAD and OPTIONS) and the full range of HTTP status codes associated with them.

Another main difference between SOAP and REST is that REST APIs allows different messaging formats (payloads), such as plain text, binary, HTML, JSON, and XML. On the other hand, the SOAP protocol only allows XML structure to share information among servers and clients.

Part I: Create a micro service using Amazon API Gateway and AWS Lambda to Proxy SOAP servers

Regardless whether your company has a server on-premise or in the cloud and this legacy solution just allows data exchange using SOAP, having to expose these services over a modern API pattern as REST, we can use Amazon API Gateway and AWS Lambda to help you on this journey.

?

The follow diagram represents the architecture to a SOAP Proxy Server using the AWS services.

?

In this scenario, we want expose the APIs from the SOAP Server (on the right of the picture). As mentioned before, the SOAP protocol mainly uses HTTP/HTTPS to transport the data int the XML format to encapsule the payload. However, there are considerations to be made:

?

·??????The HTTP/HTTPS Header Content-Type (that is used to indicate the media type of the resource) must be set as “text/xml; charset=UTF-8”;

·??????The payload must be in XML format;

·??????The payload must follow the WSDL format defined by the server.

?

In the proposed solution, the AWS Lambda function has the responsibility to convert the incoming JSON payload into XML to be able to send to the server. Later, the function has to convert the XML received from the server in JSON format to reply back to the calling application.

?

Amazon API Gateway can receive any kind of payload (XML, JSON, binary, etc.), so there is no limitation for using this service to address this scenario, but be sure to check the service quotas defined in the following page: https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

?

Part II: Let’s build the solution using AWS SAM

?

The AWS Serverless Application Model (SAM) is an open-source framework for building serverless applications. It provides shorthand syntax to express functions, APIs, databases, and event source mappings. With just a few lines per resource, you can define the application you want and model it using YAML. During the deployment, AWS ?SAM transforms and expands the SAM syntax into AWS CloudFormation, enabling you to build serverless applications faster.

?

AWS SAM provides you with a command line tool, the AWS SAM CLI, that makes it easy for you to create and manage serverless applications. With AWS SAM CLI, we can automatically create a bucket, API, Lambda Functions via AWS CloudFormation based on your local AWS credentials.

?

Because AWS SAM integrates with other AWS services, creating serverless applications with AWS SAM provides the following benefits:

·??????Single-deployment configuration. AWS SAM makes it easy to organize related components and resources, and operates on a single stack. You can use AWS SAM to share configuration among resources, such as memory and timeouts, and deploy all related resources together as a single, versioned entity.

·??????Extension of AWS CloudFormation. Because AWS SAM is an extension of AWS CloudFormation, you get the reliable deployment capabilities of AWS CloudFormation. You can define resources by using AWS CloudFormation in your AWS SAM template. You can also use the full suite of resources, intrinsic functions and other template features that are available in AWS CloudFormation.

·??????Built-in best practices. You can use AWS SAM to define and deploy your infrastructure as config. This makes it possible for you to use and enforce the best practices such as code reviews. Also, with a few lines of configuration, you can enable safe deployments through CodeDeploy, and can enable tracing by using AWS X-Ray.

·??????Local debugging and testing. The AWS SAM CLI lets you locally build, test, and debug serverless applications that are defined by AWS SAM templates. The CLI provides a Lambda-like execution environment locally. It helps you catch issues upfront by providing parity with the actual Lambda execution environment.

·??????Deep integration with development tools. You can use AWS SAM with a suite of AWS tools for building serverless applications.

?

For this implementation, there are the following pre-requisites:

·??????NodeJS 12.x or higher (we will use NodeJS 14.x in this example)

·??????NPM (node package manager) version 6x or later

·??????AWS SAM version 1.31.0 or later

·??????Docker (it’s optional and required only if you want to test your API locally)

·??????Curl – curl is a computer open-source command-line tool for transferring data using various network protocols. We will use it to test our API locally as well as after the deployment the application on AWS. If you are not familiar with curl, you can use Postman, Insomnia, HTTPie, SoapUI or similar software.

?

In this blog, we are going to use AWS Cloud9 as an IDE (Integrated Development Environment), but you can use the one that you are used to coding. If you don’t know how to create this virtual development environment, there is an explanation in this link.

?

Here is the sequence of steps we have to do:

?

1.????Open Cloud9 service on AWS Management Console

2.????In the terminal or using the menus, create a folder for the project called proxySoapServer with a sub-folder called ./src. The ./src folder will hold the Lambda code required for this solution.

3.????Create the following blank files:

3.1.??./template.yaml

3.2.??./src/app.js

3.3.??./src/package.json

?

Option A: You can use Cloud9 editor – Menu File -> New File

Open B: You can right click on the folder in the left and select New File:

?

Option C: You can use a bash command in the terminal in Cloud9:

?

The project (folder and files) structure must be like this:

? ?

4.????Open the template.yaml file and copy the following content into the file:?

You will notice that the syntax of the template.yml file looks exactly like AWS CloudFormation. The reason is that the AWS SAM templates are an extension of AWS CloudFormation templates. It means that any resource you can define in AWS CloudFormation, can be defined in a AWS SAM template as well, but using AWS SAM will be more agile to develop, test and deploy serverless solutions on AWS.

?

In this SAM template, we are defining a Lambda function called NumberToWords (check the line number 12 for the function name) and a REST API Gateway exposing the /post method (lines 19 to 22). The source code of the Lambda function will be in the folder /src (line 15), in a file called app.js (line 16) with a function named “handler”. From lines 25 to 32, the outputs of our stack are described.

?

Remember to save the file.

?

The next step for this blog is copying the source code of our Lambda function. Open the file app.js in the folder ./src and copy the following code:

?

?

Remember to save the file as well.

?

This Lambda function is a simple proxy application to convert the incoming data from JSON into XML format (lines 49 to 56), the “Application-type” to “text/xml (lines 45 to 47) and call a public SOAP server that converts number into words (line 58). It means, if you send an integer number like “1029” in JSON format, there will be a transformation to XML, calling the legacy server and it will return a string with a value of “one thousand twenty-nine” in XML.

?

The server is going to return the word corresponding to the positive number passed as parameter, limited to quadrillions. You can check their website in the following URL:

?

https://www.dataaccess.com/webservicesserver/NumberConversion.wso?op=NumberToWords

?

This server expects the following SOAP envelop (SOAP 1.1 in the left and SOAP 1.2 in the right):

?

And the responses that the server will provide are in the following format (SOAP 1.1 in the left and SOAP 1.2 in the right):

?

Due to the fact of using external libraries in the Lambda code, we must update the package.json file and run “npm install” command later. Open in the ./src folder the file package.json and copy the following content:?

In the Cloud9 terminal, change the directory to ./src and run the following command to install the external libraries:

You will notice that there are 2 dependencies and node_modules folder was created in the ./src to include these libraries.

Before testing the application, we need to build the artifacts using “sam build” command in the root directory of the project.

The “sam build” command processes your AWS SAM template file, application code and any applicable language-specific files and dependencies. The command builds and copy artifacts in the format and location expected for subsequent steps in your workflow. You can specify dependencies in a manifest file that you include in your application, such as requirements.txt for Python functions, or package.json for Node.js functions.

?

The output of the command should be similar to the following image:

?

Now it’s time to run our application locally to test it. AWS SAM allows you to run your serverless application locally for quick development and testing. When you run the following command in a directory that contains your serverless functions and your AWS SAM template, it creates a local HTTP server that hosts all of your functions.

?

You can use the command “sam local start-api” to emulate the execution of an Amazon API Gateway and receive request locally, but it’s mandatory to have docker running in your local environment.?

**IMPORTANT** If you don’t have docker running locally in your machine, you can ignore this step and deploy the application as defined in Part III of this blog. In Cloud9, docker is running by default.

You can open another terminal (click in the + button in the right of “Immediate tab” and choose New Terminal) to show the execution of the REST API call (POST request method using curl) and kept the first terminal opened to show how the docker instantiates a container to run the Amazon API Gateway and the Lambda function.

You can test your local API using the following curl (post) command:

?

curl -i -X POST localhost:3000/ -H "Content-Type: application/json" -d '{"data": "188"}'?

The output should look like the following image:

In this local execution, the curl command (client application) has sent a REST POST request to Amazon API Gateway running locally in a docker container. This request had Content-Type as Application/json. The API Gateway sent the payload in JSON to Lambda function that “wrapped” this information into an XML format.

?

After this transformation, the Lambda function sent the request to the SOAP server and received an XML with the number transformed into words.

?

To be able to reply this request in REST, the Lambda function decomposed the XML and collected the exact information needed to create the JSON and sent it to the client application (curl).

?

Some important information in the execution:

1)????Image of the AWS Lambda and Amazon API Gateway was loaded in docker to be able to run it locally

2)????AWS Lambda was able to retrieve the correct value from the payload

3)????AWS Lambda called the legacy SOAP Server that returned the number in words

4)????The result of the REST API was successful (200 HTTP status code)

5)????Even though the SOAP server replied with a Text/XML Application-type in the HTTP header, the client (curl) received an Application-type as application/json due to the conversion executed by our solution

6)????The final result in JSON format was the number converted in to words.

?

Part III: Deploying the solution on AWS

?

Now it’s time to deploy this SOAP Proxy solution into AWS. You will see how easy it is to deploy a solution using AWS SAM.

?

Execute the following steps:

?

1)????Press Control+C in the terminal that is running the “sam local start-api” command, use a second terminal or open a new one to run the command “sam deploy -g”?

2)????Fill out the following values when requested:

This command (sam deploy -g) deploys your application into the AWS Cloud. It takes the deployment artifacts that was created, packages it and uploads them to an Amazon Simple Storage Service (Amazon S3) bucket that AWS CloudFormation will use to deploy the application.

?

That’s it… there is no more work to do and your solution is up and running. Simple, isn’t it?

?

This single command has created an API Gateway (with a stage and deployment), a Lambda Function, ?an execution role, a S3 bucket to store the artefacts, called AWS CloudFormation that deploys the solution and all the configuration needed to run this service on AWS on your behalf.

?

As a result, the AWS CloudFormation Outputs informs the URL that your REST API is listening.?

You can use the following command to test your solution:?

(OBS1: you must replace the URL for the one you generated on your deployment)

(OBS2: you can copy the command above in the output session of “sam deploy”)?

Part IV: Clean up

Using AWS Serverless service, customers only pay for what they use. If there is no request for AWS Lambda or Amazon API Gateway, you will not be charged at all.

?

Both services used here have a free tier as defined bellow:

?

But, if you want to clean up your AWS Account, you can simply run an AWS SAM command:?

?

You have to confirm the deletion of the application as well as the artifacts stored on S3. This will be the result:

?

?

Part V: Conclusion

?

Following these few steps, we were able to create a REST service to proxy a legacy SOAP server, using Amazon API Gateway and AWS Lambda. This REST API exposes an endpoint that accepts JSON payload and transforms this data into XML format. Then, we invoked a legacy server using its WSDL standard and finally, with the server’s response, our function had to transform from XML to JSON and replied the result to the client application.

?

There were few challenges as mentioned before:

1) The REST API is required to have Content-Type header of “Application/json”;

2) The SOAP Server is required to have Content-Type header of “text/xml”;

3) Apart from that, the conversion from JSON to XML and XML to JSON is mandatory.

?

You can use this solution and AWS services to keep using your legacy software and continue the modernization journey of your company using modern and more scalable application programming interface (API).

?

If you have other use cases for modernize SOAP legacy architectures and want to share it with me, please reach me at LinkedIn and I’ll try to help. https://www.dhirubhai.net/in/danielabib/

?

Author bio

Daniel Abib is Enterprise Solution Architect at AWS, with over 20 years working with project management, scalable application architectures, systems development & CI/CD, microservices, serverless and security.?He works supporting Enterprise customers, helping them on their journey to the cloud.

https://www.dhirubhai.net/in/danielabib/

?

Suggested tags: [Serverless, AWS Lambda, Amazon API Gateway, AWS SAM, SOAP, Modernization]