Setting up an OpenStack Private Cloud on an IBM X3850 X5 Server

So, you just scored yourself a used IBM X3850 X5 server and want to setup your own private cloud? It’s definitely a journey where the slightest mistake means that you need to wipe your machine and start again. So I’ve decided to take note of every step that I took so that you don’t have to go through as many iterations trying to get this up and running as I have.

Tips

1. Don’t try and install Ubuntu or Debian. I tried and as soon as the install CD booted, the PCIe and NVM Error lights (on the light path diagnostics panel) came on and the system crashed

2. This is one of the reasons I selected OpenStack for my private cloud (despite its steeper learning curve and difficulty in getting it up and running). OpenStack runs natively on Centos, whereas my preferred option (Proxmox) only runs on Debian.

Preparation and OS Installation

3. Get some hard drives if you haven’t already. Mine didn’t come with any so I bought 2 x 1.2tb SAS, installed them and set them up in RAID 1 via the RAID configuration tool in BIOS

4. Burn the latest version of the Centos 7 ISO to a CD or DVD

5. Boot up the server and insert the Centos 7 installation CD. It’ll take some time to validate memory and run system checks but eventually you’ll be shown a screen that gives you an option to enter BIOS (F1) or select a drive to boot from (F12). I of course pressed F12 and selected the Centos 7 CD

6. The system booted up in to the Centos 7 setup process with no issue. I ensured my ethernet cable was plugged in to my router (although I did have some difficulties when I plugged it in to one of the ethernet ports on the main board — but my server came with a PCIe ethernet card with 4 x 1gbps ports, so I used one of those instead)

7. I selected “Australia” as my location and clicked Continue

8. I then selected the Installation Target option. Within that I found my 1.2tb RAID virtual drive, which I selected and chose the auto-partition option — then clicked Done

9. Back in the main setup dashboard I then selected “Networking”. I chose the ethernet interface that my cable was plugged in to and clicked “Enable”. After a few seconds, my router’s DHCP server assigned it an IP address in the relevant range. So I clicked Done to return to the setup dashboard.

10. I then clicked “Continue / Install”

11. As the system was installing the files I was presented with two options — ROOT PASSWORD and USER CREATION. I clicked the first option and set my root password. I then clicked the second option and entered the credentials for my main administrator user.

12. Setup then continued until it was finished and I was presented with a “Reboot” button. I removed the installation CD and clicked “Reboot”.

Disable SELinux

1. Once booted in to the freshly installed OS, run the following commands from the terminal:

2. sestatus

3. setenforce 0

4. nano /etc/sysconfig/selinux

5. Update SELINUX variable from “enforcing” to “disabled” and save the file

Open Up Relevent Firewall ports

1. firewall-cmd –permanent –add-port=5901/tcp

2. firewall-cmd –permanent –add-port=5901/udp

3. firewall-cmd –permanent –add-port=80/tcp

4. firewall-cmd –permanent –add-port=443/tcp

5. firewall-cmd — reload

Installing the GNOME Desktop

1. I find things much easier to do with a desktop — like run multiple tasks in parallel. So once I started up my system and logged in as root, I installed the Gnome desktop.

2. Firstly, I enabled the EPEL repo — yum -y install epel-release

3. From the command line I entered one simple command:

yum -y groups install “GNOME Desktop”

4. From Gnome, I opened a Terminal and entered “systemctl set-default graphical.target” to ensure that the system would boot in to the GUI on every startup

Enable IOMMU

1. Nano /etc/default/grub

2. Append “irqpoll” and “intel_iommu=on” to end of GRUB_CMDLINE_LINUX:

3. GRUB_CMDLINE_LINUX=”nofb splash=quiet console=tty0 … irqpoll intel_iommu=on

4. Run “grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg”

5. Note: IOMMU is for OpenStack PCI passthrough during virtualization. IRQPOLL is because I was alerted to a problem with IRQ 22 when I initially just added intel_iommu=on and there wasn’t enough verbocity in the error to explain why the problem existed.

6. Reboot

Installing TigerVNC

1. Firstly I installed my favourite text editor — yum -y install nano

2. sudo yum install tigervnc-server

3. su — darren

4. vncpasswd

5. Enter a password of at least 6 digits in length

6. cp /lib/systemd/system/[email protected] /etc/systemd/system/vncserver@:1.service

7. nano /etc/systemd/system/vncserver@\:1.service

8. Ensure the following lines are present in this config file:

[Unit]

Description=Remote desktop service (VNC)

After=syslog.target network.target

[Service]

Type=forking

ExecStartPre=/bin/sh -c ‘/usr/bin/vncserver -kill %i > /dev/null 2>&1 || :’

ExecStart=/sbin/runuser -l darren -c “/usr/bin/vncserver %i -geometry 1280x1024”

PIDFile=/home/darren/.vnc/%H%i.pid

ExecStop=/bin/sh -c ‘/usr/bin/vncserver -kill %i > /dev/null 2>&1 || :’

[Install]

WantedBy=multi-user.target

9. systemctl daemon-reload

10. systemctl start vncserver@:1

11. systemctl status vncserver@:1

12. systemctl enable vncserver@:1

13. Now test doing a remote desktop session in to your server from another computer

Installing GParted

· I just installed this so I can ensure my partitions are all setup correctly

Installing OpenStack

5. yum -y install centos-release-openstack-rocky

6. yum -y install openstack-packstack python-pip

7. yum update

8. ip a (to discover the name of my working ethernet interface)

9. packstack — allinone — provision-demo=n — os-neutron-ovs-bridge-mappings=extnet:br-ex — os-neutron-ovs-bridge-interfaces=br-ex:enp151s0f0 — os-neutron-ml2-type-drivers=vxlan,flat

10. Once installed

a. Get password from the rc file in the home folder of the machine you installed OpenStack on to

b. Navigate to the IP address of the installed instance and verify its running

c. Reboot computer

11. Ensure you have one or two qcow2 formatted OS cloud images (I had one for Centos 7 and another for Ubuntu 18.04) — You can work with ISOs but it’s a lot harder (you have to attach two volumes to the instance you create — one for the iso and one for the HDD you are installing to. And sometimes it fails to detach the ISO image once installed)

12. Go to your Openstack admin, login, go to images tab and create an image for reach of these — ensure this works

Further Network Configuration

1. systemctl stop NetworkManager.service

2. systemctl disable NetworkManager.service

3. nano /etc/sysconfig/network-scripts/ifcfg-enp151s0f0

4. Comment out BOOTPROTO

5. service network restart

Further Network Configuration

1. Open shell and “sudo su” to get root

2. cd ~/

3. . keystonerc_admin (sets environment variables to allow you to use Openstack CLI)

4. neutron net-create external_network — provider:network_type flat — provider:physical_network extnet — router:external

5. Check / limit the range your existing LAN router utilises for DHCP. For instance mine was from 192.168.1.2–192.168.1.255. I changed it to 192.168.1.2–192.168.1.100

6. neutron subnet-create — name public_subnet — enable_dhcp=False — allocation-pool=start=192.168.1.101,end=192.168.1.200 — gateway=192.168.1.1 external_network 192.168.1.0/24

7. openstack project create — enable industryswarm

8. openstack user create — project industryswarm — password castle — email [email protected] — enable darren

9. neutron router-create router1

10. neutron router-gateway-set router1 external_network

11. neutron net-create private_network

12. neutron subnet-create — name private_subnet private_network 192.168.100.0/24

13. neutron router-interface-add router1 private_subnet

Further Openstack Configuration

1. Navigate to Openstack URL (IP of your machine) in your browser and login with admin credentials

2. Go to Project > Network > Security Groups

3. Click Create New Security Group

4. Call it HTTP, HTTPS and SSH and click Create

5. Once created, select it and choose Manage Rules

6. Delete both existing rules in there

7. Add 4 new rules — types HTTP, HTTPS, SSH and All ICMP.

8. You now have a working security group for your instances

9. Now choose Floating IPs

10. Click Allocate IP to project, and then simply confirm when presented with dialog. Repeat to get a second floating IP

Create Two Test Instances

1. First to to Project > Compute > Key Pairs

2. Click Create New Key Pair. Once created it will automatically download the private key. Save this somewhere safe and chmod it to 600

3. Go to Project > Compute > Instances

4. Click “Launch Instance”

5. Settings are as follows (everything else is left default):

a. Name: Ubuntu

b. Source

i. Volume Size: 20gb

ii. Allocated: Push up your ubuntu image to be allocated to this instance

c. Flavour

i. Push m1.small up to be associated with this instance

d. Networks

i. Push both private_network and external_network up to be allocated to this instance

e. Security Groups

i. Push the HTTP, HTTPS and SSH security group that you created up to be allocated to this instance

f. Key Pair

i. The key pair you created previously should automatically be selected

6. Click “Launch Instance”

7. Then click “Associate Floating IP” and select one of the ones you had created (if you’re not quick enough then you may not be able to SSH in with the key you generated)

8. Ensure instance is up and running

9. Open a terminal on a separate computer on your LAN and ensure you can successfully ping the floating IP address of the instance (it should work)

10. Then: ssh -I Openstack.pem [email protected] (the floating IP I chose) and confirm that you are logged in and get the Ubuntu prompt.

11. From within the instance prompt — use vi to edit /etc/resolv.conf and set nameserver IP to 8.8.8.8

12. And then attempt to ping www.google.com and a few of your LAN network IP addresses. They should all work.

13. Now repeat the above (steps 3 to 12) for the Centos image

Image Your Machine

1. If you have managed to start instance(s) from within OpenStack then you’re now in a good state. Its probably taken a while to get here, and any additional changes have the ability to mess with your OpenStack setup and prevent it from working at all

2. Therefore, I believe that now is a good time to take a snapshot image of your machine’s current state that you can restore to, should any further changes put your machine in a state that’s hard to revert from

3. I initially tried Clonezilla, but it appears to be based on a Linux distro that isn’t compatible with the IBM X3850 server.

4. So I burnt a live Centos CD (one that boots in to a live Gnome environment instead of installing the OS) and booted in to that.

5. I attached a 4tb external USB HDD that was pre-formatted to the system and it auto-mounted and showed as available

6. I discovered the path that it was mounted to by right clicking on properties in the file manager and made a note of this

7. I then made sure that I ejected / unmounted all of the partitions that Centos had auto-mounted through the file manager

8. And then opened a command prompt

9. From here I typed “fdisk -l” to get a list of drive devices and partitions. It was immediately obvious that my 1TB RAID drive for the system was /dev/sdc

10. I then ran the command “dd if=/dev/sdc of=/var/run/media/[GUID for the USB drive]/backup.img bs=64k” — this will copy the entire 1TB RAID device across to an image file that you can restore from later, if the need arises.

11. Be prepared for this to take some time — probably hours as the entire 1TB drive is copied across to an image on the USB

Anyway, that’s it for now. I’m going to write some follow up posts about my journey installing the remainder of the OpenStack environment (including Zun for containers, Qinling for FaaS — Function as a Service — based applications; and generally as many components that Openstack has to offer.

Stay tuned.

Originally published on the Blog of Darren Smith