Two Failures Behind One Load Balancer, and Why Both Components Were Right¶
The Auto Scaling Group kept replacing instances. The application kept starting cleanly. Neither of those statements was wrong, and the gap between them took most of an afternoon to close. Then the moment the health check passed, a second failure appeared underneath it that had been invisible until the first one was gone.
Try It Yourself
Both fixes are live in the deployed application. The full four-model deployment, including this one, is documented in the BankApp runbook.
Both failures share a cause that is not in either component: a load balancer changes what the application sees, and an application that was written without one assumes things that stop being true.
The First Symptom Is a Loop, Not an Error¶
The target group showed instances cycling through states without ever settling:
| i-0b30ae2882deb7bf0 | draining |
| i-02dc8b14446e9f184 | unhealthy |
| i-00106fa2c4522af1e | initial |
| i-0d2edf3ca7a790144 | draining |
initial, then unhealthy, then draining, then terminated, then a new instance repeating the sequence. The Auto Scaling Group was doing exactly what it is designed to do: an instance failing its health check is a failed instance, and failed instances get replaced.
That is the trap in this failure. Nothing is broken from the ASG's point of view, so nothing logs an error. The replacement loop is the correct response to bad input.
Ruling Out the Obvious Before the Subtle¶
The first suspicion was the artifact. It was not:
The second suspicion was startup failure, which meant getting onto an instance that was actively being terminated. That is its own small race, and the bastion path made it possible:
The logs were clean:
Spring Boot, Hibernate, the connection pool and MySQL had all initialised. At this point the application and the infrastructure were both reporting success, which is the state that means the question is wrong.
The question that was right: not "is the app up" but "what exactly does the health checker receive".
That is the whole diagnosis in one line, and it is worth noting how much cheaper it was than everything before it.
Ordering Is the Bug¶
302 is a redirect. Spring Security was intercepting the health endpoint as an unauthenticated request and sending it to the login page. The health checker followed the redirect, received an HTML login form, and correctly concluded that this was not a healthy response.
The rule responsible, in SecurityConfig.java:
.authorizeHttpRequests(authz -> authz
.requestMatchers("/register").permitAll()
.anyRequest().authenticated() // caught /actuator/health
)
The fix is one line, and where it goes is the entire fix:
.authorizeHttpRequests(authz -> authz
.requestMatchers("/register").permitAll()
.requestMatchers("/actuator/health").permitAll()
.anyRequest().authenticated()
)
Matchers are evaluated top to bottom and stop at the first match
Placing the health matcher below anyRequest().authenticated() compiles, deploys, and does nothing at all, because the catch-all has already matched. This is the kind of bug that survives code review, since the line is present and looks correct in isolation.
After redeploying, the endpoint returned 200 with {"status":"UP"} and the targets went healthy.
The Second Failure Was Hiding Behind the First¶
With the application reachable at https://bankapp.ibtisam-iq.com, login submitted and bounced straight back to the login page. Repeatedly. Forever.
This failure could not have been found earlier. It requires a browser, a session, and a working health check, and until this point there had never been an instance alive long enough to try.
The architecture is the ordinary one:
- Port
80redirects to HTTPS with a301 - Port
443forwards to the target group on port8000over plain HTTP
TLS terminates at the load balancer. The application only ever sees HTTP. Three separate consequences follow, and all three have to be fixed together.
Spring generated http:// redirects after login. The load balancer sets X-Forwarded-Proto: https, but Spring ignores that header unless told to trust it. So Spring saw an HTTP request and issued a post-login redirect to http://bankapp.ibtisam-iq.com/dashboard. The browser upgraded that to https://, which invalidated the session established on the http:// leg. Back to login.
The session cookie had no Secure flag. Browsers only send Secure cookies over HTTPS. Without the flag the cookie was set and then not transmitted on the next HTTPS request, so Spring Security saw an unauthenticated request and redirected to login.
The session cookie had no SameSite attribute. Modern browsers increasingly drop cookies with no explicit value, which compounds the loss above rather than causing it independently.
Three properties in application.properties, applied together:
# Trust X-Forwarded-Proto from the load balancer so Spring generates https:// redirects
server.forward-headers-strategy=native
# Required for the cookie to be transmitted over HTTPS
server.servlet.session.cookie.secure=true
# Allows the cookie on top-level navigations and the login form POST
server.servlet.session.cookie.same-site=lax
Fixing one of the three looks like no progress
Each property removes one reason the session is lost, and any remaining reason produces the identical symptom. Applying them one at a time and testing between produces three consecutive failures that look identical, which reads as "the fix did not work" rather than "the fix worked and there is another one".
The Diagnosis Order Is the Reusable Part¶
Looking back, the expensive steps were the ones that checked whether components were working. The cheap step was the one that checked what a specific caller received.
That ordering generalises. When infrastructure and application disagree, there is a third thing neither of them is showing you: the actual request and the actual response between them.
A useful sequence for this class of failure:
- Reproduce the exact call the failing component makes. Not a similar call. The health checker requests a specific path with no cookies and no auth header, so
curlit from inside the instance with nothing attached. Testing from a browser you are already logged into hides the entire bug. - Read the status code before reading the body.
302and200are different diagnoses. A body that contains an HTML login form when JSON was expected tells you the same thing more slowly. - Check what the application believes about the request, not just what it does with it. Spring's view of the scheme was wrong, and no amount of reading application logs would have shown that, because from its perspective nothing was wrong.
Health endpoints deserve their own rule, not an exception to yours
The general pattern is that a health endpoint is a public, unauthenticated, cheap route that answers with a status code and nothing sensitive. Treating it as "one more path to add to the allow list" is how it ends up below a catch-all. Treating it as a category means it goes at the top of the matcher chain by default.
Sticky Sessions Are a Fourth Thing, Not a Fifth Fix¶
There is no shared session store. Sessions live in each instance's memory, so a request that lands on a different instance than the one that authenticated has no session there either:
aws elbv2 modify-target-group-attributes \
--target-group-arn $TG_ARN \
--attributes \
Key=stickiness.enabled,Value=true \
Key=stickiness.type,Value=lb_cookie \
Key=stickiness.lb_cookie.duration_seconds,Value=86400
This is worth being honest about: stickiness is a workaround for the absence of a shared session store, not a design choice. It pins a browser to an instance, which means an instance replacement logs those users out. The real fix is externalising sessions, which was out of scope for a deployment comparison where the point was to change only the compute layer.
What Actually Transfers¶
Neither failure is about AWS, and neither is about Spring. Both are about the same structural fact: a reverse proxy in front of an application changes two things the application cannot see.
It introduces a client that is not a user. Health checkers do not log in, do not hold cookies, and do not follow your intent. Any authentication rule that defaults to "deny" will catch them, and the symptom appears in the infrastructure rather than in the application.
It changes the scheme without telling the application. Every framework has a way to trust forwarded headers, and every framework defaults to not trusting them, because trusting them blindly is a spoofing vector. That default is correct and it is also the thing that breaks you, so it has to be turned on deliberately and only where the proxy is trusted.
The version of this in ECS Fargate later was identical, because the cause was never in the compute layer. Same two fixes, same order, no rediscovery needed. That is the useful outcome of writing it down the first time.
Source¶
- The original debugging notes, written while this was happening
- Phase 4: EC2 Auto Scaling runbook
- Phase 5: ECS Fargate runbook, where both fixes carried over unchanged
Series: One Application, Four Ways (Part 2 of 4)