lugh.ch

as nerdy as needed.

Provisioning infrastructure with Terraform on Infomaniak's OpenStack cloud


This is neither a full guide nor to be considered best practice. It's a point-in-time experiment I did late 2023 to up my Terraform (OpenTofu) skills and gather OpenStack experience. It probably also contains serious fallacies.

About Infomaniak

This is not a paid post 😅

Infomaniak is a Swiss web hosting company based in the Romandie. I got to know their OpenStack offering by chance because I used their DNS registrar offering at a time, when WHOIS privacy was still a marketable service for .ch domain names.

What's the goal?

  • Get experience with OpenStack
  • Deepen Terraform knowledge
  • Have a new, economical home for my Nextcloud data

High level access view

I'vem omitted a few things here for the sake of readability. They will be covered by later diagrams.

Networks, subnets and routers

Routers connect networks, which contain subnets to segregate workloads. My setup is using project/self-service/tenant networks.

Networks and subnets

We only allow public access to production-grade services, therefore the production subnet inside the the service network is merely allocated to production instances.

resource "openstack_networking_network_v2" "net-services" {
  name           = "net-services"
  description    = "Network for subnets holding services that are going to be used from the outside"
  admin_state_up = true
  external       = false
}

resource "openstack_networking_subnet_v2" "subn-services-prod" {
  name            = "subn-services-prod"
  description     = "PROD services subnet"
  network_id      = openstack_networking_network_v2.net-services.id
  cidr            = "192.168.aaa.0/24"
  ip_version      = 4
  dns_nameservers = ["1.1.1.1"]
}
resource "openstack_networking_subnet_v2" "subn-services-dev" {
  name            = "subn-services-dev"
  description     = "DEV services subnet"
  network_id      = openstack_networking_network_v2.net-services.id
  cidr            = "192.168.bbb.0/24"
  ip_version      = 4
  dns_nameservers = ["1.1.1.1"]
}

The DMZ internal and external network and subnet is configured the same way as the infra network:

resource "openstack_networking_network_v2" "net-dmzext" {
  name           = "net-dmzext"
  description    = "External facing DMZ net"
  admin_state_up = true
  external       = false
}

resource "openstack_networking_subnet_v2" "subn-dmzext" {
  name            = "subn-dmzext"
  description     = "External facing DMZ subnet"
  network_id      = openstack_networking_network_v2.net-dmzext.id
  cidr            = "192.168.ccc.0/24"
  ip_version      = 4
  dns_nameservers = ["1.1.1.1"]
}

resource "openstack_networking_network_v2" "net-dmzint" {
  name           = "net-dmzint"
  description    = "Internal facing DMZ net"
  admin_state_up = true
  external       = false
}

resource "openstack_networking_subnet_v2" "subn-dmzint" {
  name            = "subn-dmzint"
  description     = "Internal facing DMZ subnet"
  network_id      = openstack_networking_network_v2.net-dmzint.id
  cidr            = "192.168.ddd.0/24"
  ip_version      = 4
  dns_nameservers = ["1.1.1.1"]
}

Routers and router interfaces

As their name suggests, these connect networks to the Internet. Router interfaces join subnets to a certain router. To be completely transparent here: I don't know how the heck to explain this. OpenStack's routing/port security parts felt like black magic wizardry to me. I've watched hours of intoductory OpenStack networking videos and read blog posts. I just made it work somehow and tested all my positive and negative use cases.

This actual network topology diagram from Horizon might help.

OpenStack network topology

  • You can access the service of inst-nextcloud via its floating IP over the net-services network only
  • Management access to the bastion instance only is possible via the net-dmzext network over a separate floating IP. Only the bastion host can reach all instances on net-dmzint
  • The floating IP addresses are obtained from the ext-floating network managed by Infomaniak

The HCL below ensures connectivity between the subn-services-prod, subn-dmzext and subn-dmzint subnets and sets ext-floating1 as the gateway.

resource "openstack_networking_router_v2" "router-dmzext2dmzint" {
  name                = "router-dmzext2dmzint"
  description         = "Connects DMZ external, DMZ internal and service networks"
  admin_state_up      = true
  external_network_id = "0f9c3806-bd21-490f-918d-4a6d1c648489" # ext-floating1
}

resource "openstack_networking_router_interface_v2" "iface-dmzext" {
  router_id = openstack_networking_router_v2.router-dmzext2dmzint.id
  subnet_id = openstack_networking_subnet_v2.subn-dmzext.id
}

resource "openstack_networking_router_interface_v2" "iface-dmzint" {
  router_id = openstack_networking_router_v2.router-dmzext2dmzint.id
  subnet_id = openstack_networking_subnet_v2.dmzint.id
}

resource "openstack_networking_router_interface_v2" "router-iface-service" {
  router_id = openstack_networking_router_v2.router-dmzext2dmzint.id
  subnet_id = openstack_networking_subnet_v2.subn-services-prod.id
}

Of course this has to be secured. More on this in the Security groups and rules part below.

Floating IPs and associations

Floating IP addresses can be mapped to fixed internal IP addresses. The official documentation explains this well:

Each instance has a private, fixed IP address and can also have a public, or floating IP address. Private IP addresses are used for communication between instances, and public addresses are used for communication with networks outside the cloud, including the Internet.

In my case, this is used to offer access to services and the bastion host from the public Internet. In OpenStack lingo, you associate a floating IP to an instance.

# Bastion
resource "openstack_networking_floatingip_v2" "float-dmzext-1" {
  description = openstack_compute_instance_v2.inst-bastion.name
  pool        = "ext-floating1"
}

resource "openstack_compute_floatingip_associate_v2" "assoc-dmzext-1" {
  floating_ip = openstack_networking_floatingip_v2.float-dmzext-1.address
  instance_id = openstack_compute_instance_v2.inst-bastion.id
}

# Nextcloud
resource "openstack_networking_floatingip_v2" "float-services-1" {
  description = openstack_compute_instance_v2.inst-nextcloud.name
  pool        = "ext-floating1"
}

resource "openstack_compute_floatingip_associate_v2" "assoc-services-1" {
  floating_ip = openstack_networking_floatingip_v2.float-services-1.address
  instance_id = openstack_compute_instance_v2.inst-nextcloud.id
}

Security groups and rules

One can assign security groups to instances. Security groups can contain multiple rules. The actual rules define what ingress/egress traffic is allowed. By default, everything not allowed is denied.

Security groups

For instances behind the bastion host

This security group can be assigned to any instance behind the bastion host. The naming is a bit misleading.

resource "openstack_networking_secgroup_v2" "sg-bastion2internal" {
  name        = "sg-bastion2internal"
  description = "Rules to add to instances behind the Jumphost"
}

resource "openstack_networking_secgroup_rule_v2" "sg-rule-internal-1" {
  description    = "SSH from Jumphost"
  direction      = "ingress"
  ethertype      = "IPv4"
  protocol       = "tcp"
  port_range_min = 22
  port_range_max = 22
  remote_group_id   = openstack_networking_secgroup_v2.sg-world2bastion.id
  security_group_id = openstack_networking_secgroup_v2.sg-bastion2internal.id
}

resource "openstack_networking_secgroup_rule_v2" "sg-rule-internal-2" {
  description = "ICMP from Jumphost"
  direction   = "ingress"
  ethertype   = "IPv4"
  protocol    = "icmp"
  remote_group_id   = openstack_networking_secgroup_v2.sg-world2bastion.id
  security_group_id = openstack_networking_secgroup_v2.sg-bastion2internal.id
}

resource "openstack_networking_secgroup_rule_v2" "sg-rule-internal-3" {
  description       = "DNS TCP"
  direction         = "egress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 53
  port_range_max    = 53
  security_group_id = openstack_networking_secgroup_v2.sg-bastion2internal.id
}

resource "openstack_networking_secgroup_rule_v2" "sg-rule-internal-4" {
  description       = "DNS UDP"
  direction         = "egress"
  ethertype         = "IPv4"
  protocol          = "udp"
  port_range_min    = 53
  port_range_max    = 53
  security_group_id = openstack_networking_secgroup_v2.sg-bastion2internal.id
}

Bastion security group rules

Mind the direction parameters. The ingress rule allows traffic towards the bastion host. The egress rule permits traffic from the bastion host towards other systems in the internal DMZ subnet.

resource "openstack_networking_secgroup_v2" "sg-world2bastion" {
  name        = "sg-world2bastion"
  description = "SSH access to/from bastion for selected networks"
}

resource "openstack_networking_secgroup_rule_v2" "sg-rule-bastion-1" {
  description       = "<my-home-ip-addr>"
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = "nn.nn.nn.nn/32"
  security_group_id = openstack_networking_secgroup_v2.sg-world2bastion.id
}

resource "openstack_networking_secgroup_rule_v2" "sg-rule-bastion_2" {
  description       = "Bastion DMZ ext to DMZ int"
  direction         = "egress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = openstack_networking_subnet_v2.subn-dmzint.cidr
  security_group_id = openstack_networking_secgroup_v2.sg-world2bastion.id
}

Nextcloud security group rules

This simply allows HTTP/HTTPS traffic from everywhere.

resource "openstack_networking_secgroup_v2" "sg-nextcloud" {
  name        = "sg-nextcloud"
  description = "Nextcloud"
}

resource "openstack_networking_secgroup_rule_v2" "sg-rule-nextcloud-1" {
  description       = "Allow HTTP"
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 80
  port_range_max    = 80
  remote_ip_prefix  = "0.0.0.0/0"
  security_group_id = openstack_networking_secgroup_v2.sg-nextcloud.id
}

resource "openstack_networking_secgroup_rule_v2" "sg-rule-nextcloud-2" {
  description       = "Allow HTTPS"
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 443
  port_range_max    = 443
  remote_ip_prefix  = "0.0.0.0/0"
  security_group_id = openstack_networking_secgroup_v2.sg-nextcloud.id
}

Instances

This shows how instances are erected and to which subnets they are connected.

Bastion instance

Is powered off by default. Which saves a little money, and also increases security.

Multi-layer defense

resource "openstack_compute_keypair_v2" "key-bastion" {
  name       = "key-jumphost"
  public_key = "ssh-ed25519 [...]"
}

resource "openstack_compute_instance_v2" "inst-bastion" {
  name              = "inst-bastion"
  image_id          = "c397fe03-a5cb-4d22-9965-6ebf859ddf9f"
  flavor_name       = "a1-ram2-disk20-perf1"
  key_pair          = "key-bastion"
  security_groups   = ["sg-world2jumphost"]
  power_state       = "active" # shutoff or active
  availability_zone = "dc3-a-10"

  metadata = {
    application = "SSH Bastion"
  }

  network {
    name = "net-dmzext"
    uuid = openstack_networking_subnet_v2.subn-dmzext.network_id
  }

  network {
    name = "net-dmzint"
    uuid = openstack_networking_subnet_v2.subn-dmzint.network_id
  }
}

Nextcloud instance

Has connectivy to the services network (to get a floating IP) and to the DMZ internal network. This way, only the exposes service is reachable from the Internet, but management access has to pass the bastion instance.

resource "openstack_compute_keypair_v2" "key-nextcloud" {
  name       = "key-nextcloud"
  public_key = "ssh-ed25519 [...]"
}

resource "openstack_objectstorage_container_v1" "s3-nextcloud-prod" {
  name         = "s3-nextcloud-prod"
  content_type = "application/json"
}

resource "openstack_compute_instance_v2" "inst-nextcloud" {
  name            = "inst-nextcloud"
  image_id        = "c397fe03-a5cb-4d22-9965-6ebf859ddf9f"
  flavor_name     = "a1-ram2-disk20-perf1"
  key_pair        = "key-nextcloud"
  security_groups = ["sg-bastion2internal", "sg-nextcloud"]
  power_state     = "shutoff" # shutoff or active
  # depends_on : https://github.com/terraform-provider-openstack/terraform-provider-openstack/issues/775

  metadata = {
    application = "Nextcloud"
  }

  network {
    name = "net-services"
    uuid = openstack_networking_subnet_v2.subn-services-prod.network_id

  }
  network {
    name = "net-dmzint"
    uuid = openstack_networking_subnet_v2.subn-dmzint.network_id

  }
}

What did I learn

Instance creation can fail due to API rate limits. This only happened during extensive trial/error phases throughout Terraform setup, so not a big deal.

OpenStack networking is complex, at least for me. The official networking documentation is more geared towards operators of OpenStack installations, which of course is fine. But mere users have to get to know the difference of networks (provider vs. self-service [which OpenStack providers might call differently]), ports, subnets, routers and so on. Which itself is not a deal, you want to know these conceptual items anyways. I just often asked myself "is this something I can use as an OpenStack end user, or is it used for administration purposes?"

Some issues I had

Billing UX quite sucks

The billing docs refer to a openstack rating dataframes get command that simply does not work: parse error: Invalid numeric literal at line 1, column 10. This must be an effect of openstack: 'rating dataframes get' is not an openstack command. But even if it worked, this is cumbersome. I'd rather have a scheduled report sent to me as PDF/JSON/whatever. Having to rely on a system with OpenStack credentials and certain Python packages to get intermediate billing reports is a bit too low-level for my taste.

I cannot remember the command, but once I got it working, I did fail to filter costs by exact month boundaries, e.g. previous/current month.

The overhead of the virtual currency is also annoying. OpenStack CloudKitty lists no hard requirement to maintain a virtual currency.

Infomaniak docs lack some context or are sometimes outdated

Their public cloud documentation is a bit dated. Most parts work, but several areas needed experimentation because it was missing context.

I can imagine a kind of community feedback functionality could help to improve documentation quality.

How did the experiment end?

During this whole process, I was also onboarding my family to the Proton ecosystem. As soon as they announced a photo backup feature, I stopped the OpenStack project, as this feature was the last missing piece to replace Nextcloud.

I learned a lot and I could easily tear down the whole infrastructure thanks to terraform destroy ;-)


Similar posts