A couple weeks ago we began experiencing slowdowns on our LAN and internet. It wasn’t hard to identify that the email server on our LAN was the culprit. We’ve had user email accounts hacked before, with resulting several hundred thousand emails sent before we noticed the problem. The email server did not show a large number of outbound emails or data from an individual user, but the per-client list in our Balance router showed massive amounts of outbound data coming from the email server’s LAN address. The picture below shows our typical data through the WAN port before the problem began, and after we fixed it:
Looking at inbound/outbound sessions on the Balance Status page, I noticed inbound port 11211 from some countries around the world where we have no communication. Russia, China, etc. I don’t know what port 11211 is so I asked Google. Thats where I learned about the memcache attack:
So what happened that I was exposed to this?
It has always been my practice to set the default inbound firewall to block-all. That just seems like a good practice to me. Some other discussions here, other users are comfortable with the default allow-all firewall. It shouldn’t make any difference because if there is no inbound port forwarding, an inbound attack can’t go anywhere in the LAN. About a month ago I changed the default inbound firewall to allow-all.
After fighting this problem for a few days, I changed the default inbound firewall to block-all. The problem instantly stopped. Life is good again.
I don’t have an inbound firewall that allows 11211 so this should not have happened. How could this be?
It took me a few more days to figure it out, but the reason is because I have IP forwarding set up for the email server. Outbound email goes out through a WAN port which has five public IP addresses. The mail must use a specific outbound IP that is not the default for that port. It has to use that IP for anti-spam verification (receiving servers look to see if our mail is coming from the correct IP. If not our mail is flagged as spam).
I set up that IP forwarding years ago. I had set up IP forwarding both inbound and outbound! Inbound IP forwarding effectively maps all inbound ports to the LAN device. When I changed my default inbound firewall to allow-all, that exposed my mail server to every bad thing the internet has to offer.
My problem was cured by changing the default firewall rule. Since then I also removed inbound IP forwarding (I only really needed it outbound), so I have to layers to keep the bad people out. It was good lesson for me. I hope that by sharing this some other person can learn from this mistake. Use your own choice on the default firewall but don’t mix it with inbound IP forwarding!