Ok so I have 2 WANs coming in to a balance 20:
WAN 1 - Cable 150/50 - /29 static
WAN 2 - Fiber 200/200 - 1 IP only DHCP
I have a rack with 2 Pfsense appliances in High Availability
Each Firewall is connected to two switches for distribution, LAN 1 and LAN 2.
Each server in the rack has dual nics, connected to LAN 1 and LAN 2, configured for failover.
I am tasked with providing redundancy as much as possible. So far everything from the Pfsense appliances down is working. Can kill power or drop an ethernet cable on either firewall or either switch without dropping packets to the WAN side of the firewalls.
I need help with the Peplink side. So far I am testing at another location (away from the peplink).
The High Availability on the firewalls requires minimum of 3 static IPs from each WAN in order to have clean failover. The challenge is that we cannot get statics from the Fiber ISP. Otherwise I would be inclined to eliminate the peplink.
Options / Questions:
-
Should I get a second Balance 20? what advantages would that give us beyond hardware redundancy of that stage?
-
Should I eliminate the Balance 20?
-
Can the Balance 20 forward 3 static IPs from WAN1 to 2 of its LAN ports?
-
Can the Balance 20 NAT the DHCP IP from WAN2 to 3 statics, and assign those to 2 of its LAN ports?
-
I need to setup a few highly available VPN tunnels into the network. If I set these up on the Pfsense firewalls, how easily will these pass through a multi-WAN implementation on the Balance 20?
Thanks!
Oooh - finally an interesting challenge! 
My first suggestion would be to let the Cable router assign IPs directly to the PFsense HA pair, then slap the Fiber router into the Balance 20’s WAN and use NAT to give you the required 3 static IPs on its LAN. Pop the the Balance LAN interface into a VLAN so that the PFsense can be assigned IPs from it on the same physical WAN ports as the existing Cable IPs. Then assign the pfsense HA VIP as a 1:1 NAT Mapping on the Balance 20.
Then go get a cup of tea and have a rest.
1 Like
Thanks Martin for your response!
Let me get it straight. You want the cable WAN to totally bypass the Peplink hardware, and attach it directly to the PFsense appliances. Then you want the Fiber to do a one:many NAT to turn its DHCP address into a trio of simulated Statics, dropped onto a Vlan into the Pfsense.
-
I was hoping to use the Peplink as switch to distribute the cable and fiber connection to the two firewalls, avoiding a pair of small switches in between. Primary reason being to eliminate more devices, more power supplies that can fail.
-
My pfsense hardware has multiple interfaces, so I don’t have to use “the same physical WAN ports” as you mentioned. Right now I have it setup for WAN1 and WAN2 on separate interfaces.
-
It seems like your solution would take Peplink’s (very good) multi-wan features out of the equation, using the balance only for the NAT. Would it not be better to keep it in the link?
-
It’s important that the VPN connections (have not yet decided openVPN or IPSEC) failover smoothly when one of the providers goes down. I’m not too worried about load balancing or saturating a link, the reliability of the link is higher priority.
Making tea in the interim. 
Ahha the plot thickens…excellent.
Some observations in no particular order:
-
The Balance 20 is rated at 150Mbps throughput - you have a total of 350/250 so it would become a bottleneck if used for load balancing both WANs.
-
Balance routers do an epic job of Load balancing in Multi-WAN environments and PepVPN is astonishingly good at coping with all the usual challenges to be found with IPsec when multi-WANs are in play. If there is a choice I always use PepVPN/SpeedFusion over IPsec .
-
I suggested using the Balance 20 just for NAT because 1) It doesn’t have enough throughput to cope with the available WAN bandwidth across both WANs, and 2) since you have HA on every other element I liked the idea of two separate perimeter routers with their own roles. That said, a pair of Balance routers in HA is a better configuration - but that will require a pair of switches on the WAN of the PFsense appliances, and another pair on the WAN of the Peplinks (if you are aiming for proper redundancy) and you’re a touch limited with only having a single IP on your Fiber connection.
-
So another configuration option would be to put the cable modem into WAN 1 on the balance and enable drop in mode, then plug the Fiber connection into WAN2 and use NAT. Then add another Balance for HA later if you’d like to. That way the pfsense HA pair gets a public IP from the cable modem (through the balance that becomes transparent to the PFsense pair) , and if the cable modem goes down, the balance will act as an ARP proxy and forward traffic to the fiber WAN as a backup.
Oh man, that’s exciting Martin.
That’s exactly as I envisioned.
In the spec for the Balance 20 it says it can’t do HA. Can you comment on that?
I think the throughput will not be an issue in the beginning. But as the application scales, it may become a factor. At that point though, the budget will allow a more high-end solution. I have left space in the top of the rack for some fancier edge hardware. Hopefully I can keep it compact by cramming 2 small switches into 1U above and below the balance pair (4U total)
For now it sounds like we can put together something that works quite well with the balance 20. Can I route WAN1 through to LAN 1 and 2, and WAN2 through LAN 3 and 4, avoiding the switches between the balance and the firewalls? Or are you suggesting I setup the PFsense HA as single-WAN and consider the Balance as the WAN?
The Balance has been in place for 2 years and runs very well, so I feel ok trusting it until we start saturating the throughput.
Now, for the IPSEC. The endpoints are the company owner’s home office, my home office, a second location which will be equipped with an identical rack, and an Amazon VPC which hosts customer-facing apps. The way I read the documentation, you can only use PepVPN if you have pep hardware at both ends. So unless there is a Peplink AWS appliance, can I assume PepVPN won’t work for that tunnel? For the other tunnels, I am not opposed to supplying Peplink hardware if that is the way to go.
I really appreciate your assistance!
No worries.
Rats, you’re right - no HA on B20 and even worse no Drop In Mode - sorry its after midnight here and I’ve had a long day…, I should have checked.
But assuming you got your hands on a balance that could do both - yes I was suggesting the PFsense HA Pair has single WAN. You’d set the pfsense VIP to be one from your /29 block, each device will have its own IP from that range also. Normally you would plug the pfsense WANs and the Balance LAN into a switch so the physical WANs of the pfsense can all route via the balance.
If you only have a B20 you’re really limited. You can do NAT of course from both the fiber and cable modems and then have two WANs on the pfsense and use it for load balancing, but i would likely fail back to my previous suggestion of cable modem directly connected and B20 just doing the fiber modem NAT role. You could always add a USB LTE connection to the B20 later too if you like. That way, your PFsense gets a public IP so no nat in the way to cause IPSEC issues.
1 Like
Also- Peplink does have a virtual appliance you can run in AWS. Its called Fusionhub
That type of star topology with a FusionHub in AWS alongside core Apps and remote sites connecting in is what it was designed for,
1 Like
Ok I understand.
So what If I take the B20 and install it at the owner’s home office. Then I can replace it with some more beefy Peplink hardware. What do you suggest? Ideally something that can handle 1GB/s max throughput (combined), and will PEP-VPN with the remote B20 and with Fusionhub (wow I had no idea you guys had that!).
1 Like
I think I answered my own question here. What I need is a pair of Balance 305’s. I’ll put in a request
1 Like
You’re right - the B305 (load balancing only) or B380 (with SpeedFusion) is where you need to start. You’ll really like PepVPN - its sooo easy Good luck!
1 Like
Ok Martin. Moving target incoming.
I called and bitched out the fiber ISP. They buckled and gave me a /29.
So now I have 5 static IPs from each provider, so I can setup the multi-WAN HA properly in pfsense without any fakery.
Boss still wants a third fall-back to 4G through a USB modem on the balance 20. So what do you think now? We can pass the 5 statics through the B20, and use the 4G as disaster backup. Is the lack of a drop-in mode a dealbreaker here?
Cheers!
How many NICs on the pfsense appliances?
If you can give them 3 wans, there is the way I would want to do it and a cheaper way…
First the cheapest way.
Pfsense is really very good (I prefer opnsense for the enhanced ui if you haven’t played with it do…) and it can happily do active passive failover and load balancing, and it supports IPSec which is what you’re currently using at your sites so you could just treat the balance 20 as an additional failover circuit used for backup only.
If it was me - I would Peplink SD-WAN the whole thing. Fusionhub in AWS, Pair of B380s in your rack in HA, balance 20 at the bosses house wiith 4G failover, balance ones everywhere else. The central management with InControl, the ease of PepVPN and the seamless packet level failover capabilities of SpeedFusion all add up to a really powerful easy to manage highly available solution - but you’d need to justify it commercially and there is the challenge.
1 Like
The pfsense boxes are Supermicro 5018A-FTN4s. Each has a 2 port 10Gbe card. Each feeds into a ubiquity US-48 over the 10Gbe interfaces.
So they have 4 Gb interfaces left over. WAN1, WAN2, PFSYNC, AND MGMT. I keep the all the IPMI from all the systems on the separate management interface with only a tunnel for me.
I love your suggestion with the peplink and I think that’s a good upgrade path, but boy is it expensive. The beauty is that if we come to need that solution, there will be budget for it.
Will the B20 pass through the static ip’s from the fiber ISP?
When both ISPs fail and we are left with only 4G failover, what happens to the tunnels? What can I put in place to ensure they stay up or rebuild quickly after the failover?
No. As it doesn’t support drop in mode.
It can certainly look expensive. However cost savings through ease of management (especially with incontrol and remote web admin) and the high availability of SpeedFusion VPN eliminating truck rolls to remote sites and potentially keeping your company online and working with ease over cellular and fixed lines still makes it very cost effective in the long run.
You’re stuffed generally. making IPsec work inbound over cellular (with MNO NAT in play typically) is one of the trickiest pieces of config you’ll ever do. Its one of the reasons why Peplink developed PepVPN in the first place - to make VPN configurations easy/possible in complex NAT/ Multi-WAN deployment scenarios,
Outside of PepVPN, my other goto VPN technology which I have had some success with when using cellular connectivity is TINC VPN. I’ve not used this in multi-WAN scenarios (not sure that it supports multi-wan) but its is very simple and very clever.
I’m honestly not sure what to advise you do. You either take the least disruptive path and put the Balance 20 on the shelf -and just use your existing pfsense pair, or you spend the cash and adopt Peplink’s world. Anywhere in between seems like too much of a compromise or adding complexity for the sake of trying to fit all the existing hardware you currently have into an end solution design.
1 Like
