The Ubiquiti Diaries: A Site-to-Site VPN Story

Miscellaneous Considerations and Concluding Remarks

The site-to-site VPN setup created with the help of ZeroTier is able to provide easy access to systems in both sites without involving a relay in the middle. Remote management of systems connected to the UDM from the US is straightforward - essentially dealing with something in the local LAN. The need for port forwarding and other overheads is done away with. Performing offsite backups is also simplified. A NAS connected to the UDM is equivalent to a NAS connected to the USG Pro 4 as far as backup jobs go, as all the routing is handled transparently within the USG Pro 4 and the UDM. The two remaining aspects I set out to achieve - seamless utilization of OTT services on either side (particularly, Indian ones on the US side) and transparent extension of the Indian Wi-Fi network in the US were achieved only on a per-device basis (by configuring a proxy). My next step was to address that by creating a wireless network on the USG Pro 4 that would defer to the UDM as the gateway.

Policy-Based Routing on the USG Pro 4

The first step involved the configuration of a new network in the web UI, followed by creating a new Wi-Fi SSID and allocating the new network to it. In my case, the new network was allocated VLAN ID 5. The USG was allowed to provision itself after these changes before moving on to the next step. Using the same config.gateway.json scheme adopted for task scheduling, a new routing table was created with the default route pointing to the ZeroTier IP of the UDM deployed in India.

The next step involved creation of new firewall rules with the modify tag. All VLANs except for the newly created one with ID 5 were relegated to the usage of the main routing table. The new VLAN alone (singled out using the source address subnet) was allocated the newly created routing table. Most importantly, source validation was disabled.

Finally, the new firewall rules were allocated to the newly created VLAN.

Upon provisioning the new configuration, the new Wi-Fi SSID became equivalent to the one on the UDM side. For all practical purposes, clients connecting to that SSID appeared to be accessing the web from an Indian IP address. This method of VPN access also circumvents checks put in by some OTT apps, wherein streaming would get disabled upon the recognition of any VPN profile on the device.

Future Work

The satisfaction of getting the site-to-site VPN working aside, I am not particularly happy that I had to involve a third-party service in the mix. Eventually, I would like to get either manual IPSec or WireGuard working. Both are suppoed to not have trouble with the public WAN IP side acting as a server and the machine behind the CGNAT acting as a peer as long as the required ports are open. However, I have not had much luck (as described in detail in the IPSec section). WireGuard debugging is also complicated by the fact that the USG Pro 4 is a relatively closed system (enabling kernel level debugging of WireGuard using the recommended scheme doesn't work).

Given that I am not a professional network administrator, it is entirely possible that there is a much simpler solution for my requirements here - experienced readers are welcome to provide suggestions in the comments.

Final Words

Ubiquiti's UniFi lineup is immensely popular among prosumers, and enjoys a cult following. It is easy to see why - simple aspects are easy to configure over the web UI, while SSH access is available for the more esoteric things. At the same time, the relationship is a bit frustrating too (the propensity of the gateways to corrupt themselves at the drop of a hat in the event of a power failure, and the UDM being silent about unsuccessful NTP synchronization attempts are cases in point). On balance, I would still recommend Ubiquiti equipment, and upgrades to my primary deployment of the USG Pro 4 and host of Wi-Fi 5 APs are likely to be UniFi-based in the future.

Back in 2017 (when I decided to go with UniFi for my residential installation), the concept of economical managed SDN solutions for prosumers was in its infancy, with Ubiquiti's offerings being the only game in town. Since then, we have seen other vendors step in with their own suite of products - Netgear's Insight-managed business networking products, TP-Link's huge Omada line-up with support for both cloud-based and local management controllers, and QNAP's set of products supporting QuWAN are examples. In light of the increased competition, Ubiquiti would do well to address some of the pain points outlined in this article. In particular, the absence of a guaranteed upgrade path for customized USG Pro 4 installs like the one above to one of the newer AArch64-based gateways is a cause of concern. There are also features taken for granted (like the NTP server functionality in the USG Pro 4) that are not present in the newer gateways. Considering that Ubiquiti has opted for a container-based architecture in the UDM and subsequently released UniFi gateways, they might as well create an official curated app store to enable additional functionality.

Moving forward, I hope to address other interesting aspects of Ubiquiti's lineup of products in this series of articles. Suggestions from readers are welcome.