Apache Load Balancer to AWS Elastic Load Balancer

Recently we ported a B2B web services application from a traditional hosting provider to AWS cloud.

Application’s traffic was originally load balanced using Apache Load Balancer. In the AWS cloud environment, we replaced Apache Load Balancer with AWS Elastic Load Balancer. During this migration, we ran into an interesting hiccup. In this article sharing the problem and solution, as it may benefit others too.

Problem Statement

Application was a B2B (webservices) application.  Its services were consumed by multiple clients, who are geographically distributed all throughout the world. Application maintained the whitelist of IP addresses of its clients. Only the clients in the whitelist can invoke the services. The team had developed a Java ServletFilter in the application layer and using this filter it was authenticating the incoming requests IP address. (Even though it’s questionable whether it’s a secure solution, it is outside the focus of this article).

The requests which were serviced properly under Apache Load Balancer started to fail when we moved to AWS Elastic Load Balancer. In fact, it was failing because of IP address authentication. How can the requests fail IP address authentication when IP filtering was happening in the application layer code? Application layer code remained unchanged in both environments. Isn’t it is puzzling?

Root Cause

Note: Before reading further, please review the above diagram for few seconds. It will help you to understand the below content easily.

Apache Load Balancer preserved Originator’s IP address

Assume that the client from IP address 1.1.1.1 fires an HTTP request to the application. Request first hits the Apache Load Balancer, then Load Balancer send the request to the Application Server. When the request finally hits the application server, client’s IP address was preserved in the HTTP request. i.e. Server would see the originator’s IP address as 1.1.1.1, thus authentication worked correctly without any issues under the Apache Load Balancer.

AWS Elastic Load Balancer not preserving Originator’s IP address

Under AWS Elastic Load Balancer, when client’s HTTP request hits the application server, client’s IP address was NOT preserved. i.e. suppose client sent the request from the IP: 1.1.1.1. When the request hits the Server; Server sees the AWS elastic load balancer’s IP address only (i.e. 20.20.20.20) and not client’s IP address (i.e. 1.1.1.1). Since AWS Elastic Load Balancer’s IP address isn’t white-listed, application started to reject the requests.

Because of this discrepancy in the load balancer behavior, application started to reject all the incoming requests.

Solution

There are a couple of solutions to address this problem:

1. Web ACL

AWS provides a simple but yet powerful ‘Web Access Control List (Web ACL)’ service. Using this service, one can do the IP filtering in the AWS Elastic Load Balancer. Thus team moved the IP Filtering logic from the application layer to the Load Balancer using ‘Web ACL’ service.

In the application layer also maintained a whitelist of Load Balancer’s IP addresses only. So that no one besides Load Balancers can directly shoot the request to the application server.

2. X-Forwarded-For

AWS Elastic balancer can be configured to pass client’s IP address in the X-Forward-For header element. The X-Forwarded-For request header takes the following form:

X-Forwarded-For: clientIPAddress

Example: X-Forwarded-For: 203.0.113.7

Team went with solution #1 – Web ACL, as it’s better to knock down the rogue request at the Load Balancer level itself instead of letting it come all the way to the Application server.