CEPH Why? & How?

Ceph is an Open Source Distributed storage solution and it puts together to simplify the highly scalable object, file-based storage and block under one whole system.

Ceph supports both scale-up and scale-out using economical commodity hardware. Clusters of Ceph are constructed with the help of an algorithm called CRUSH (Controlled Replication Under Scalable Hashing) in order to run commodity hardware.

Architecture

Ceph is based on RADOS (Reliable Autonomic Distributed Object Store) and it provides multiple storage interfaces in a single unified storage cluster.

Why Ceph?

Ceph becomes more popular for multiple reasons.

1. Open source: As Ceph is an open-source software there are no licensing fees involved for using Ceph.
2. Ceph storage is software-defined and hence there is?no need to depend on expensive hardware. Using Ceph, commodity hardware can be used to build a storage product that is just as scalable and flexible as same as any SAN storage system.
3. Scalability:?If you want more capacity there’s no need to purchase additional licenses for Ceph, you can just add more servers.
4. The exclusive aspect of Ceph is that it grants multiple interfaces to access the storage.

Also, Ceph storage offers common features found in any enterprise-grade storage product like snapshots, replication, thin provisioning, and self-healing.

Specific Use Cases

• Large bulk block storage

As Ceph has the ability to design and build cost-effective OSD nodes, Ceph helps us to build large high-performance storage clusters that are very cost-effective compared to all other available storage options.

• Object storage with custom application

We can connect to the in-house application to directly talk to the underlying Ceph RADOS layer using librados. This feature will simplify the development of your application and grant you direct access to highly performant reliable storage.

• Distributed filesystem – web farm

A farm of web servers all need to access the same files so that they can all serve the same content regardless of which one the client connects to.

Commonly, a high-availability NFS solution would be used to provide distributed file access but it has limitations at scale. CephFS can implement a distributed filesystem to store the web content and allow it to be mounted across all the web servers in the farm.

The Ceph Metadata Server is necessary to run Ceph File System clients.

Note

It is a best practice to have a Ceph Manager for each Monitor, but it is not necessary.

• Monitors: A Ceph Monitor (ceph-mon) maintains maps of the cluster state, including the monitor map, manager map, the OSD map, the MDS map, and the CRUSH map. These maps are critical cluster state required for Ceph daemons to coordinate with each other. Monitors are also responsible for managing authentication between daemons and clients. At least three monitors are normally required for redundancy and high availability.
• Managers: A Ceph Manager daemon (ceph-mgr) is responsible for keeping track of runtime metrics and the current state of the Ceph cluster, including storage utilization, current performance metrics, and system load. The Ceph Manager daemons also host python-based modules to manage and expose Ceph cluster information, including a web-based Ceph Dashboard. At least two managers are normally required for high availability.
• Ceph OSDs: An Object Storage Daemon (Ceph OSD, ceph-osd) stores data, handles data replication, recovery, rebalancing, and provides some monitoring information to Ceph Monitors and Managers by checking other Ceph OSD Daemons for a heartbeat. At least three Ceph OSDs are normally required for redundancy and high availability.
• MDSs: A Ceph Metadata Server (MDS, ceph-mds) stores metadata for the Ceph File System. Ceph Metadata Servers allow CephFS users to run basic commands (like ls, find, etc.) without placing a burden on the Ceph Storage Cluster.

Ceph stores data as objects within logical storage pools. Using the CRUSH algorithm, Ceph calculates which placement group (PG) should contain the object, and which OSD should store the placement group. The CRUSH algorithm enables the Ceph Storage Cluster to scale, rebalance, and recover dynamically.

CEPH Lab Setup:

Nodes Setup

1. Create 5 nodes with 1 GB RAM and 1 core CPU each. Let's denote them as follows:

 * ceph1 as Monitor
 * ceph2 as Controller
 * ceph3 as Compute0
 * ceph4 as Compute1
 * ceph5 as Client        

2. Attach 3 disks of 20 GB each to ceph-compute01 and ceph-compute02.

Hostnames Configuration (Execute below commands on the respective nodes)

3. On ceph-monitor:

 hostnamectl set-hostname ceph1        

4. On ceph-controller:

 hostnamectl set-hostname ceph2        

5. On ceph-compute0:

 hostnamectl set-hostname ceph3        

6. On ceph-compute1:

 hostnamectl set-hostname ceph4        

7. On ceph-client:

 hostnamectl set-hostname ceph5        

Create FQDN Entry

8. Create FQDN Entry of hostname on all nodes with their respective IP:

 echo '192.168.xxx.xxx  ceph1.in  ceph-controller' >> /etc/hosts
 echo '192.168.xxx.xxx  ceph2.in ceph-compute01' >> /etc/hosts
 echo '192.168.xxx.xxx  ceph3.in ceph-compute02' >> /etc/hosts
 echo '192.168.xxx.xxx  ceph4.in ceph-monitor' >> /etc/hosts
 echo '192.168.xxx.xxx  ceph5.in ceph-client' >> /etc/hosts        

Replace 192.168.xxx.xxx with the respective IP addresses.

Basic Node Configuration (Execute these commands on all nodes)

9. Disable Firewall:

 systemctl stop firewalld && systemctl disable firewalld        

10. Install chrony for time synchronization:

yum install chrony -y
chronyc sourcestats        

11. Create a user for Ceph deployment and disable its password:

useradd cephadm && echo "cdac" | passwd --stdin cephadm        

12. Allow cephadm user to run sudo commands without a password:

echo "cephadm ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadm
chmod 0440 /etc/sudoers.d/cephadm        

13. Disable SELinux:

sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux        

14. Install the EPEL repository:

sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm        

15. Reboot each node:

reboot        

Chrony Configuration (Execute these commands on all nodes except ceph-controller)

16. Modify the chrony configuration:

nano /etc/chrony.conf        

Comment all available servers and add the line server ceph-controller.

17. Restart chrony:

systemctl restart chronyd        

Ceph Controller Configuration

18. Install the Ceph release package and Ceph deployment tools:

sudo rpm -Uvh https://download.ceph.com/rpm-mimic/el7/noarch/ceph-release-1-1.el7.noarch.rpm
yum update -y && sudo yum install ceph-deploy python2-pip  -y        

19. Generate SSH keys for the cephadm user:

su - cephadm
ssh-keygen        

Hit enter to accept the defaults

20. Copy the SSH keys to the other nodes:

ssh-copy-id cephadm@ceph3
ssh-copy-id cephadm@ceph4
ssh-copy-id cephadm@ceph1
ssh-copy-id cephadm@ceph5        

21. Create an SSH configuration file to specify the username to use when SSHing to the other nodes:

nano ~/.ssh/config        

Add the following lines:

Host ceph-compute0
   Hostname ceph3
   User cephadm
Host ceph-compute02
   Hostname ceph4
   User cephadm
Host ceph-monitor
   Hostname ceph1
   User cephadm
Host ceph-client
   Hostname ceph5
   User cephadm        

22. Adjust the permissions on the SSH configuration file:

chmod 644 ~/.ssh/config        

23. Test the SSH configuration:

ssh cephadm@ceph1
ssh cephadm@ceph3
ssh cephadm@ceph4        

Ceph Cluster Configuration (Execute these commands as the cephadm user on the Ceph controller)

24. Create a new directory for the Ceph cluster and navigate into it:

mkdir ceph_cluster
cd ceph_cluster        

25. Create a new cluster:

ceph-deploy new ceph1        

26. Edit the Ceph configuration file:

nano ceph.conf        

Add the line public network = 192.168.xxx.xxx/24, replace 192.168.xxx.xxx with your local network.

Please note, you will need to replace placeholders like 192.168.xxx.xxx with your actual values.

27. Install Ceph on the nodes:

ceph-deploy install ceph2 ceph3 ceph4 ceph1        

Initialize the Monitor and Create the Ceph Storage Cluster

28. Initialize the monitor:

ceph-deploy mon create-initial        

29. Deploy the Ceph client admin keyring:

ceph-deploy admin ceph2 ceph3 ceph4 ceph1        

30. Create Ceph manager daemons:

ceph-deploy mgr create ceph3 ceph4        

31. List the disks available:

ceph-deploy disk list ceph3 ceph4        

32. Prepare and activate the OSDs (Replace /dev/sdb, /dev/sdc, and /dev/sdd with the appropriate disk names):

  ceph-deploy osd create --data /dev/sdb ceph-compute01
  ceph-deploy osd create --data /dev/sdc ceph-compute01
  ceph-deploy osd create --data /dev/sdd ceph-compute01

  ceph-deploy osd create --data /dev/sdb ceph-compute02
  ceph-deploy osd create --data /dev/sdc ceph-compute02
  ceph-deploy osd

 create --data /dev/sdd ceph-compute02        

33. Install Ceph on the client and distribute the admin keyring:

ceph-deploy install ceph5
ceph-deploy admin ceph5        

here above ceph5 is denoted as a client node.

Verify the Ceph Storage Cluster

34. Check the health of the Ceph cluster and its detailed status:

ceph health
ceph health detail
ceph -s        

35. Create a Ceph storage pool:

ceph osd pool create rbd 50 50        

Ceph Client Configuration

36. On the client, create a new block device, disable features that are not compatible with the kernel RBD driver, and map the block device:

  rbd create disk01 --size 4096
  rbd ls -l
  modprobe rbd
  rbd feature disable disk01 exclusive-lock object-map fast-diff deep-flatten
  rbd map disk01
  rbd showmapped        

37. Create a filesystem on the new block device, create a mount point, and mount the block device:

  mkfs.xfs /dev/rbd0
  mkdir -p /mnt/mydisk
  mount /dev/rbd0 /mnt/mydisk
  df -h        

VM: ceph1 Monitor

┌──────────────────────────────────────────────────────────────────────┐
│                 ? MobaXterm Personal Edition v23.1 ?                 │
│               (SSH client, X server and network tools)               │
│                                                                      │
│ ? SSH session to [email protected]                                │
│   ? Direct SSH      :  ?                                             │
│   ? SSH compression :  ?                                             │
│   ? SSH-browser     :  ?                                             │
│   ? X11-forwarding  :  ?  (disabled or not supported by server)      │
│                                                                      │
│ ? For more info, ctrl+click on help or visit our website.            │
└──────────────────────────────────────────────────────────────────────┘

Last login: Sun Jul 16 21:12:39 2023
[root@ceph1 ~]# vim /etc/chrony.conf
[root@ceph1 ~]# systemctl restart chronyd
[root@ceph1 ~]# ceph-deploy install ceph-controller ceph-compute01 ceph-compute02 ceph-monitor
-bash: ceph-deploy: command not found
[root@ceph1 ~]# ceph-deploy mon create-initial
-bash: ceph-deploy: command not found
[root@ceph1 ~]# vim /etc/hosts
[root@ceph1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.239.154 ceph1.in
192.168.239.155 ceph2.in
192.168.239.156 ceph3.in
192.168.239.157 ceph4.in
192.168.239.158 ceph5.in
192.168.239.159 ceph6.in
[root@ceph1 ~]# systemctl restart chronyd
[root@ceph1 ~]#        

For other VMs terminal command history reference, visit to the given below github link.

https://github.com/siddheshkachkure/SBM/tree/master/Ceph%20Storage%20/Nodes