Linking OpenStack to a private network with OPNsense / pfSense / VyOS / FortiGate
Estimated time to read: 12 minutes
This guide explains how you can setup an OpenStack network with an attached custom router to handle traffic to specific subnets. You can then, for example, tunnel traffic using a VPN or IPsec to connect your cloud servers with servers on infrastructure such as VMware outside of Fuga Cloud. In the guide we will cover setting up an instance with FortiGate, pfSense, OPNsense or VyOS. You can however use any routing/firewall software capable of this functionality or even a bare linux instance that routes the traffic through a VPN service like WireGuard or OpenVPN!
During this guide you will use the OpenStack CLI to create and configure the necessary resources, so make sure you have that installed and configured to follow along with the tutorial. It also assumes you already know how to configure your desired routing software once installed, if not don't hesitate to contact us for additional support with migrating to Fuga Cloud!
Setting up the network
First up you will need to create a new custom network in OpenStack. To do this we will run the following command:
openstack network create custom-network \
--description "Network with additional custom router for internal traffic" \
--disable-port-security \
--internal
Write down the network ID from the results. Then create a subnet for this network using the following command:
This custom network will the network assigned to any compute instance or EMK-cluster that should be using the tunnel. Instead of their own fixed external IP they will get an IP on our custom subnet. You can then make services on these instances public via either the OpenStack router you will configure or via the custom router we will setup. Another option is to also assign the instance to the public network, or to associate a floating IP with the instance. In case of the last option it is advised to enable port security on the instance so you can use security groups to manage access from this floating IP.
Setting up the default router
Next up, you will create a router and associate it with your network. This router will handle by default all traffic on your private network, you will also configure it to route certain subnets to a custom router later in the tutorial. Create a new router with the following command:
openstack router create default-router \
--description "Default gateway for the custom network, routes private traffic to custom router" \
--external-gateway public
This will create a router that uses the public network as default gateway. Again write down the IP of the router we just created. To associate this router with your private network we will run the following command:
Now your default router and your custom network should be attached to each other. The next step will be setting up a custom router on a compute instance.
Setting up a custom router
Fuga Cloud by default only provides images of Ubuntu, Debian, CentOS and Rocky Linux, in order to use a custom router OS you will need to supply your own image. The steps below describe how you can setup your own images for these specific operating systems.
Creating a custom image
Creating an image for OPNsense is relatively straightforward. Although OPNsense does not provide a dedicated cloud image, we can use their nano image to quickly spin up an instance with OPNsense. This image contains a preinstalled version of OPNsense that will automatically adapt to the size of the disk it is installed on.
First up, download the nano image (AMD64) from the OPNsense website. This will download a compressed version, in order to use it with OpenStack we will need to decompress it. To do this run the following command:
This will replace the compressed file with an uncompressed version. After this is done, upload it to Fuga Cloud with the following command:
Once succesfully uploaded, you have an image which you can use to create instances with OPNsense!Installing pfSense on an instance requires a bit more effort than installing OPNsense. pfSense does not supply a cloud or nano image, so you need to install pfSense onto a volume using the ISO installer. After this you can turn this volume into an image, and use it to create instances that run pfSense.
First up, download the ISO (AMD64) from the pfSense website. Once downloaded upload it as image using the following command:
openstack image create pfsense-installer-iso \
--file pfSense-CE-2.7.2-RELEASE-amd64.iso \
--disk-format iso \
--min-disk 10
Once the ISO is uploaded, create a server using the following command:
openstack server create pfsense-installer-instance \
--image pfsense-installer-iso \
--network public \
--flavor s3.medium
You'll need to attach a volume where pfSense will be installed. An installation of pfSense requires at least 8GB of storage, in this tutorial we chose to use 10GB. Use these commands to create and attach a volume:
openstack volume create pfsense-volume \
--size 10 \
--bootable
openstack server add volume pfsense-installer-instance pfsense-volume \
--device /dev/vda
To follow the installation process, use the following command to open up a webconsole to the instance:
Now follow along the instructions and install pfSense to the created volume. After this is done you can exit the instance and delete it:
The next step is to create an image from this volume, so you can use that for subsequent use when creating pfSense instances:
After this is done you have an image which you can use to create instances with pfSense!
Creating an OpenStack compatible image for VyOS LTS is fairly straightforward. Go to this page and make sure you are logged in with the account that holds your VyOS subscription so you can download the most recent version. VyOS also provides legacy LTS releases for free, beware however that these releases no longer receive security updates and bug fixes. It is also possible to build an LTS release from source without a subscription, however that is beyond the scope of this tutorial.
Once you are ready to download VyOS, make sure you download the Qcow2 image for KVM, this image with work out of the box with Fuga Cloud so no need to install it to a volume and then turn that volume into an image! To upload the image run the following command:
openstack image create "VyOS LTS" \
--file vyos-1.3.6-10G-qemu.qcow2 \
--disk-format qcow2 \
--min-disk 10
This is all you need to do, now you are ready to create an instance with VyOS LTS!
Besides their subscription based LTS builds, VyOS also provides nightly builds for free. This builds have limited testing and are built daily using the most recent sources. These builds are only provided as an ISO installer, so we will need to install VyOS onto a volume and then turn that volume into an image.
First up, download the most recent ISO from their GitHub. Once downloaded upload it as image using the following command:
openstack image create vyos-nightly-installer-iso \
--file vyos-1.5-rolling-202402210022-amd64.iso \
--disk-format iso \
--min-disk 10
Once the ISO is uploaded, create a server using the following command:
openstack server create vyos-installer-instance \
--image vyos-nightly-installer-iso \
--network public \
--flavor s3.medium
You'll need to attach a volume where VyOS will be installed. VyOS requires at least 10GB of storage. Use these commands to create and attach a volume:
openstack volume create vyos-nightly-volume \
--size 10 \
--bootable
openstack server add volume vyos-installer-instance vyos-nightly-volume \
--device /dev/vda
To follow the installation process, use the following command to open up a webconsole to the instance:
Now follow along the instructions and install pfSense to the created volume. After this is done you can exit the instance and delete it:
The next step is to create an image from this volume, so you can use that for subsequent use when creating pfSense instances:
After this is done you have an image which you can use to create instances with VyOS Nightly!
Like VyOS and OPNsense, FortiGate provides preinstalled and ready to use image of FortiOS. This makes the process of creating an OpenStack image fairly straightforward. First you will need to download a KVM image from their website. Then you can upload this image with the following command:
openstack image create "FortiGate FortiOS" \
--file fortios.qcow2 \
--disk-format qcow2 \
--min-disk 10
This is all you need to do, now you are ready to create an instance with FortiOS!
Creating an instance with your custom image
Next step is to create an instance with your newly uploaded image. The image should be a member of both the public network and the custom-network you created. Below are some guidelines for setting op the following operating systems:
Run the following command to create your OPNsense instance:
openstack server create custom-router \
--image OPNsense \
--network public \
--network custom-network \
--flavor s3.medium
Once created and ready, open up a console using the following command:
You can login with root : opnsense as credentials, it is recommended to change these for better security. After logging in proceed to the menu where you can configure the network interfaces. Once there you'll want to configure the interface associated with the public network as WAN and configure the interface associated with your custom network as WAN. You can use the MAC addresses for matching the interfaces with the right network, to see which MAC address is associated with which network you can run the following command using the OpenStack CLI:
After setting up the interfaces, configure the IP-addresses associated with the networks in the console. After this OPNsense should be setup, and it is up to you to further customize firewall rules and other settings. A common next step is to configure remote access to the OPNsense GUI, this tutorial on the Netgate docs discusses various options for configuring this. Although it targets pfSense, most options are also applicable to OPNsense, due to its roots as a pfSense fork.
Run the following command to create your pfSense instance:
openstack server create custom-router \
--image pfSense \
--network public \
--network custom-network \
--flavor s3.medium
Once created and ready, open up a console using the following command:
You can login with root : pfsense as credentials, it is recommended to change these for better security. After logging in proceed to the menu where you can configure the network interfaces. Once there you'll want to configure the interface associated with the public network as WAN and configure the interface associated with your custom network as WAN. You can use the MAC addresses for matching the interfaces with the right network, to see which MAC address is associated with which network you can run the following command using the OpenStack CLI:
After setting up the interfaces, configure the IP-addresses associated with the networks in the console. After this pfSense should be setup, and it is up to you to further customize firewall rules and other settings. A common next step is to configure remote access to the pfSense GUI, this tutorial on the Netgate docs discusses various options for configuring this.
Run the following command to create your VyOS instance:
openstack server create custom-router \
--image VyOS \
--network public \
--network custom-network \
--flavor s3.medium
Once created and ready, open up a console using the following command:
You can login with vyos : vyos as credentials, it is recommended to change these for better security. To start configuring the network interfaces, run the configure command. Next write down the MAC addresses associated with the eth0 and eth1 interfaces, to see them run the following command:
Match these with the information provided by OpenStack to determine which interface belongs the the public network and which belongs to your custom network. Use the following command in the OpenStack CLI to see this:
For this example we'll use eth0 as the WAN interface and eth1 as the LAN interface. We'll also assume that 172.10.0.189 is the external IP address on the public network, and that 10.240.10.189 is the IP address on the custom network. You can use the output of the above command to find these values for your environment. Configure the interfaces as follows:
set interfaces ethernet eth0 address '172.10.0.189/32'
set interfaces ethernet eth0 description 'OUTSIDE'
set interfaces ethernet eth1 address '10.240.10.189/32'
set interfaces ethernet eth1 description 'INSIDE'
While you're there you can configure any firewall rules or other settings you need in your environment. To save these changes run the following two commands:
You'll now have a basic VyOS router up and running that can act as a gateway for designated routes in your custom network.
Run the following command to create your FortiOS instance:
openstack server create custom-router \
--image "FortiGate FortiOS" \
--network public \
--network custom-network \
--flavor s3.medium
Once created and ready we need to configure our WAN network interface so we can access the FortiGate GUI, open up a console using the following command:
You can login with as the user admin without a password. Next step is to configure the WAN interface, this example will use 172.10.0.189 as the assigned public IP address. Adjust the following command to your environment and execute it:
config system interface
edit <INTERFACE_NAME>
set mode static
set ip 172.10.0.189 255.255.255.255
next
end
After this we will need to configure the default gateway for the WAN interface. Retrieve the default gateway from Fuga Cloud using the following command in the OpenStack CLI:
Write down the value at gateway_ip. Now in the FortiOS console, execute the command below and replace
The last step in the FortiOS console is to configure default DNS servers. For this example we will use the Google DNS servers. Execute the following command:
You will now have access to the GUI by going to the public IP address in your browser. The credentials are the same as in the console. Here you can configure an admin password and setup your LAN interrface correctly.
Routing a specific IP/subnet via the custom router
Now that you have configured your network and routers, the next step is to configure the default router to route your private traffic to the custom router.
Using the OpenStack CLI, execute the following command:
In this route all traffic to IP-addresses in the 10.360.x.x subnet will be passed to the gateway 10.240.10.189, which in this example hosts the custom router on the private network. Adjust these values to your environment.
Deploying an EMK cluster to this network
When using kubectl to deploy your cluster, you can specify what OpenStack network and router it should use for external networking. You can do this by specifying their ID under the provider configuration. You should also specify a separate local subnet for your cluster. EMK can manage internal networking between the workers without interfering in your OpenStack network, which in turn is used for external connectivity to and from your cluster. The result would look something like this:
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
namespace: # garden-<Name of your project>
name: hybrid-cluster
spec:
provider:
type: openstack
infrastructureConfig:
apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1
kind: InfrastructureConfig
networks:
id: # <Network ID of your private network>
router:
id: # <ID of your OpenStack router (not custom router!)>
workers: 10.250.0.0/16
floatingPoolName: public
controlPlaneConfig:
apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1
kind: ControlPlaneConfig
loadBalancerProvider: amphora
zone: fra-c
workers:
- name: worker-name
minimum: 2
maximum: 3
maxSurge: 1
zones:
- fra-b
- fra-c
- fra-a
cri:
name: containerd
machine:
type: emk1.small
image:
name: gardenlinux
version: 934.11.0
cloudProfileName: fugacloud
kubernetes:
version: 1.27.9
enableStaticTokenKubeconfig: false
hibernation:
schedules: []
networking:
nodes: 10.250.0.0/16
type: calico
secretBindingName: # <Insert your secret binding>
purpose: production
region: fra
This will deploy a cluster that uses your private network for networking, and thus will route traffic to IP-addresses in the 10.360.x.x subnet via your custom router. Make sure to adjust the values to what matches your environment.