Spring RestTemplate to WebClient causes OutOfMemoryError

Spring RestTemplate to WebClient causes OutOfMemoryError

Spring Boot is a highly popular framework for Java enterprise applications. One common method of integration with internal or external applications is through HTTP REST connections. We were upgrading from RestTemplate to the Java NIO-based WebClient, which can significantly enhance application performance by allowing concurrency when calling REST service endpoints. The benefits of? WebClients is as follows:

  • Concurrency: WebClient enables handling multiple connections simultaneously without blocking threads, leading to better concurrency.
  • Asynchronous: Asynchronous programming allows the application to perform other tasks while waiting for I/O operations to complete, improving overall efficiency.
  • Performance: Non-blocking I/O can manage more connections with fewer threads, reducing the resources required for handling concurrent requests.

Although the performance improved, however with the same number of concurrent connections, the WebClient was crashing with OutOfMemoryError. We will be analyzing the WebClient crash issues along with how to troubleshoot and fix them.

Spring RestTemplate to WebClient Upgrade

To harness the benefits of NIO, such as concurrency and asynchronous processing, we upgraded the rest client call from Spring RestTemplate to WebClient, as shown below.

Spring RestTemplate

  public void restClientCall(Integer id, String url,String imagePath) {

    	// Create RestTemplate instance
    	RestTemplate restTemplate = new RestTemplate();

    	// Prepare the image file
    	File imageFile = new File(imagePath);

    	// Prepare headers
   	HttpHeaders headers = new HttpHeaders();
    	headers.setContentType(MediaType.MULTIPART_FORM_DATA);

    	// Prepare the request body
   	MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
    	body.add("file",?new org.springframework.core.io.FileSystemResource(imageFile));

    	// Create the HTTP entity with headers and the multipart body
   	HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body, headers);

    	System.out.println("Starting to post an image for Id"+id);

    	// Perform the POST request
    	ResponseEntity<String> responseEntity = restTemplate.postForEntity(url, requestEntity, String.class);

    	// Print the response status code and body
    	System.out.println("Response Id "+id +":"+ responseEntity.getBody());
   	System.out.println(" Time: " + LocalTime.now());
? }        

To the following Spring WebClient as below:

public void webHeavyClientCall(Integer id,String url, String imagePath) {

	// Create a WebClient instance
	WebClient webClient = WebClient.create();

	// Prepare the image file
??????? File imageFile = new File(imagePath);

	// Perform the POST request with the image as a part of the request body
??????? MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
	body.add("file", new FileSystemResource(imageFile));
??????? System.out.println("Image upload started "+id);
		webClient.post().uri(url).contentType(MediaType.MULTIPART_FORM_DATA).body(BodyInserters.fromMultipartData(body)).retrieve().bodyToMono(String.class).subscribe(response -> {
?????????? System.out.println("Response Id"+id+ ":" + response);
	});
}        

WebClient resulting in OutOfMemoryError

When we ran both the programs in OpenJDK 11. Program that was using NIO based Spring WebClient, resulted in ‘java.lang.OutOfMemoryError: Direct buffer memory’ after a few iterations, whereas the Spring RestTemplate based program completed successfully. Below is the output of the NIO based Spring WebClient program. You can notice ‘java.lang.OutOfMemoryError’ reported.

Starting to post an image for Id0

Starting to post an image for Id1

Starting to post an image for Id2

Starting to post an image for Id3

Starting to post an image for Id4

Starting to post an image for Id5

Starting to post an image for Id6

Starting to post an image for Id7

Starting to post an image for Id8

Starting to post an image for Id9

Starting to post an image for Id10

Starting to post an image for Id11

Starting to post an image for Id12

Starting to post an image for Id13

Starting to post an image for Id14

2023-12-06 17:21:46.730  WARN 13804 --- [tor-http-nio-12] io.netty.util.concurrent.DefaultPromise  : An exception was thrown by reactor.ipc.netty.FutureMono$FutureSubscription.operationComplete()

reactor.core.Exceptions$ErrorCallbackNotImplemented: io.netty.channel.socket.ChannelOutputShutdownException: Channel output shutdown

Caused by: java.lang.OutOfMemoryError: Direct buffer memory

	at java.base/java.nio.Bits.reserveMemory(Bits.java:175) ~[na:na]

	at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[na:na]

	at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318) ~[na:na]

	at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242) ~[na:na]

	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:164) ~[na:na]

	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130) ~[na:na]

	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496) ~[na:na]

	at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:418) ~[netty-transport-4.1.23.Final.jar!/:4.1.23.Final]

	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) ~[netty-transport-4.1.23.Final.jar!/:4.1.23.Final]

	... 18 common frames omitted
        

Troubleshooting ‘OutOfMemoryError: Direct buffer memory’

In order to troubleshoot this problem, we leveraged the yCrash monitoring tool. This tool is capable of predicting outages before it surfaces in the production environment. Once it predicts outage in the environment, it captures 360° troubleshooting artifacts from your environment, analyses them and instantly generates a root cause analysis report. Artifacts it captures includes Garbage Collection log, Thread Dump, Heap Substitute, netstat, vmstat, iostat, top, top -H, dmesg, kernel parameters, disk usage….?

You can register here and start using the free-tier of this tool.

The yCrash server analyzed the Spring Boot Rest Client and provides clear indications of issues with recommendations. Below is the incident summary report that yCrash generated for the SpringBoot WebClient application.You can notice yCrash clearly pointing out the error with necessary recommendations to remediate the problem.

Fig 1: Incident Summary Report from yCrash

Garbage Collection analysis Report

yCrash’s Garbage Collection (GC) analysis report revealed that Full GCs were consecutively running (see screenshot below). When GC runs, the entire application pauses and no transactions will be processed. Entire application would become unresponsive. We observed the unresponsiveness behaviour before the SpringBoot WebClient application crashed with OutOfMemoryError.

Fig 2: yCrash report pointing our Consecutive Full GC problem

Logs analysis reporting OutOfMemoryError: Direct buffer memory

yCrash’s application log analysis report revealed that application was suffering from ‘ java.lang.OutOfMemoryError: Direct buffer memory’ (see the screenshot below) which causing the application to crash.

Fig 3: yCrash log report pointing java.lang.OutOfMemoryError: Direct buffer memory

Why is Spring WebClient suffering from OutOfMemoryError??

Fig 4: RestTemplate Objects Stored in Others Region of Native Memory
Fig 5: WebClient Objects Stored in Direct Memory Region of Native Memory

Spring WebClient is developed based on Java NIO technology. In Java NIO, objects are stored in the ‘Direct Buffer Memory’ region of JVM’s native memory, whereas RestTemplate objects are stored in the ‘others’ region of JVM’s native memory. There are different memory regions in JVM. To learn about them, you may watch this video clip.?

When we executed the above two programs, we had set the Direct Buffer Memory size as 200k (i.e. -XX:MaxDirectMemorySize=200k). This size was sufficient for Spring RestTemplate, because objects were never stored in this region, on the other hand it wasn’t sufficient for the Spring WebClient. Thus Spring WebClient suffered from ?java.lang.OutOfMemoryError: Direct buffer memory.?

Increasing -XX:MaxDirectMemorySize

After identifying this issue, we increased the direct memory size to a higher value using the JVM argument -XX:MaxDirectMemorySize=1000k. ?After making this change the Spring WebClient program worked perfectly fine with no issues.

Starting to post an image for Id0

Starting to post an image for Id1

Starting to post an image for Id2

Starting to post an image for Id3

Starting to post an image for Id4

Starting to post an image for Id5

Starting to post an image for Id6

Starting to post an image for Id7

Starting to post an image for Id8

Starting to post an image for Id9

Starting to post an image for Id10

Starting to post an image for Id11

Starting to post an image for Id12

Starting to post an image for Id13

Starting to post an image for Id14

Starting to post an image for Id15

Starting to post an image for Id16

Starting to post an image for Id17

Starting to post an image for Id18

Starting to post an image for Id19

Response Id11:Image uploaded successfully!

Response Id4:Image uploaded successfully!

Response Id1:Image uploaded successfully!

Response Id18:Image uploaded successfully!

Response Id2:Image uploaded successfully!

Response Id3:Image uploaded successfully!

Response Id6:Image uploaded successfully!

Response Id5:Image uploaded successfully!

Response Id10:Image uploaded successfully!

Response Id13:Image uploaded successfully!

Response Id15:Image uploaded successfully!

Response Id8:Image uploaded successfully!

Response Id17:Image uploaded successfully!

Response Id9:Image uploaded successfully!

Response Id7:Image uploaded successfully!

Response Id0:Image uploaded successfully!

Response Id16:Image uploaded successfully!

Response Id14:Image uploaded successfully!

Response Id19:Image uploaded successfully!

Response Id12:Image uploaded successfully!
        

Conclusion

In this post we discussed the OutOfMemoryError issue we faced when upgrading from Spring RestTemplate to Java NIO-based WebClient. We also shared the diagnostic approach we took and then resolution to the problem. Hopefully you find it useful.

要查看或添加评论,请登录

yCrash的更多文章

社区洞察

其他会员也浏览了