Nginx 502 Bad Gateway: Upstream Server connection refused caused by a a Backend Service.

Last week was a bit of a hassle devwise, it has been long since I posted some short tips of how I get my way around the world of tech for anybody else who may be in the same predicament and also for my future self in case I forget, which I hardly doubt because the scars sink in deep. Without further ado, let dive in the simple fix I had for a connection problem from a passion project.

As a software engineer I believe it is key to understand the bug, especially its nature because you may spend time going a totally wrong direction.

So, in my case, the nature of the bug was a 502 Nginx error, the logs kept flagging the following:

connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: x.x.x.x, request: "GET / HTTP/1.1", upstream: "https://url/ipaddress:portnumber/", host: "x.x.x.x"        

Now my network setup has a Load Balancer where Nginx is installed, then there are of course proxy passes to my endpoints and the endpoint and port was registered in the vhosts completing a "Normal" setup for Nginx. The upstream server in this instance, which is my staging server which was the originating library for my pipeline:

1. Did not have Nginx,
2. I was using Apache Tomcat, which was running on a Cent OS Linux Server,
3. I was using Java and the particular endpoint and the jar were up and running.

What I had not noticed, which is silly really was the fact that since there was a connection refusal of the server it may be attached to the firewall and what is allowed in the service for inbound traffic.

After going through all this I realised that the firewall was up and it was blocking all incoming traffic... I know, how could I be so ignorant right. Hey, I am learning through every hurdle lol.

Well, my fix was very small & short:

Step 1:

systemctl status firewalld        

This notorious task was up, lol,

Step 2:

systemctl stop friewalld        

There, done.. but you know the blackness in me, since there are multiple layers of proxies to the Load Balancer with 2 way handshake on each layer I decided to disable the firewall.

PS: I advise you not to disable the firewall service if you have every reason to use it for your own security.

There are many ways to kill a cat i guess and if anyone has a better fix, please advise, let us learn through sharing information.

If this did not do any help, please get a skimmed perspective here:

https://bobcares.com/blog/502-bad-gateway-nginx/