Task -Integrating aws services with Kubernetes using EKS.
Deploying Drupal with Mysql database using completely managed service-Eks...explained from very basics.
Before going to the Task let us learn some things to get an idea regarding my Task.
What is a Content Management System (CMS)?
A Content Management System [CMS] is a software platform that lets its users create, edit, archive, collaborate, report, publish, distribute and inform. Its Graphic User Interface (GUI) makes interacting with a website's database user friendly.It allows users to manage their content from an internal user interface or dashboard after one click installation.
Now , you may get what i am talking about ..yes it is Drupal.
Drupal offers limitless potential with native features and module extensions -- it’s a platform for the next disruptive technology, without disruption to your business.
Now I'm going to deploy my drupal using Kubernetes and Aws commonly by EKS.
What is Kubernetes?
Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem.
Why Kubernetes..?
Containers are a good way to bundle and run your applications. But in a situation if a container goes down, another container needs to start. Wouldn't it be easier if this behavior was handled by a system? That's how Kubernetes comes to the rescue! Kubernetes provides you with a framework to run distributed systems resiliently. It takes care of scaling and failover for your application, provides deployment patterns, and more.
Next , we want to manage our resources and Kubernetes by someone,then we go for our public cloud say ,AWS where we get Kubernetes As A Service.
What is Amazon EKS?
Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that makes it easy for you to run Kubernetes on AWS without needing to stand up or maintain your own Kubernetes control plane. Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications.
Now , let's get into the task.
Task Description:
> We are going to going create a kubernetes cluster using Eks and by using kubernetes we deploy two deployments say, drupal (For our CMS site..) and Mysql database(to store ) to store your drupal sites data.
> Next we integrate our EKS cluster with further aws services such ELB , EFS to make to balance the load and make our Deployment's data persistent.
> Next we are going to create a serverless architecture provided by EKS with kubernetes.
?> These entire things are done using CLI mode.
Prerequisites to perform task
- AWS user account - with Administration access.
- Client softwares to be installed : eksctl , kubectl , awscli.
Now , lets start our task.
Firstly ,we are going to create an AWS IAM user :
Now , Using the access key and Secret key generated to this user login to aws CLI:
Before creating the cluster , there are no resources and clusters found.
Now , we have to create EKS-cluster using the below code:
cluster.yml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: mycluster-eks
region: ap-south-1
nodeGroups:
- name: ng1
desiredCapacity: 2
instanceType: t2.micro
ssh:
publicKeyName: mykey1122
- name: ng2
desiredCapacity: 1
instanceType: t2.small
ssh:
publicKeyName: mykey1122
- name: ng-mixed
minSize: 2
maxSize: 5
instancesDistribution:
maxPrice: 0.017
instanceTypes: ["t3.small", "t3.medium"]
onDemandBaseCapacity: 0
onDemandPercentageAboveBaseCapacity: 50
spotInstancePools: 2
ssh:
publicKeyName: mykey1122
run the command: eksctl create cluster -f cluster.yml
We observe the following ,after the cluster has been created:
Now we have to update the kube-config file with our cluster created to do everything from CLI.
Verify worker nodes of cluster using CLI:
Create a Namespace (Workspace)and set it to default:
Now , we are done with the cluster part , now let's deploy our applications.Before going to deploy we need to create a persistent storage source for apps.So here we are going to EFS.
Why EFS ...why not EBS ??
Let's create EFS :
EFS should be created within the same vpc where your EKS cluster is running.
Add tag.
EFS created and EFS id and DNS name generated , which are used to create EFS provisioner.
Next , let's deploy our applications with one kustomization file.
kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
secretGenerator:
- name: mysql-pass
literals:
- password=rootpass
- userpass=vamsipass
resources:
- create-efs-provisioner.yml
- create-rbac.yml
- create-storage.yml
- deploy-mysql.yaml
- drupal-cms.yaml
The above file runs all the required files one by one.Let's learn about those files.
create-efs-provisioner.yml - to create efs provisioner.
kind: Deployment apiVersion: apps/v1 metadata: name: efs-provisioner spec: selector: matchLabels: app: efs-provisioner replicas: 1 strategy: type: Recreate template: metadata: labels: app: efs-provisioner spec: containers: - name: efs-provisioner image: quay.io/external_storage/efs-provisioner:v0.1.0 env: - name: FILE_SYSTEM_ID value: fs-b6b83267 - name: AWS_REGION value: ap-south-1 - name: PROVISIONER_NAME value: lw-course/aws-efs volumeMounts: - name: pv-volume mountPath: /persistentvolumes volumes: - name: pv-volume nfs: server: fs-b6b83267.efs.ap-south-1.amazonaws.com path: /
create-rbac.yml - this additional security and power to the provisioner.
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nfs-provisioner-role-binding
subjects:
- kind: ServiceAccount
name: default
namespace: eksns
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
create-storage.yml - here we create a storage class using above efs provisioner ans claim storage for our pods.
kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: aws-efs provisioner: lw-course/aws-efs --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: efs-drupal annotations: volume.beta.kubernetes.io/storage-class: "aws-efs" spec: accessModes: - ReadWriteMany resources: requests: storage: 2Gi --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: efs-mysql annotations: volume.beta.kubernetes.io/storage-class: "aws-efs" spec: accessModes: - ReadWriteMany resources: requests: storage: 2Gi
deploy-mysql.yaml - using this file we create mysql database for drupal.
apiVersion: v1 kind: Service metadata: name: drupal-mysql labels: app: drupal spec: ports: - port: 3306 selector: app: drupal tier: mysql clusterIP: None --- apiVersion: apps/v1 kind: Deployment metadata: name: drupal-mysql labels: app: drupal spec: selector: matchLabels: app: drupal tier: mysql strategy: type: Recreate template: metadata: labels: app: drupal tier: mysql spec: containers: - image: mysql:5.6 name: mysql env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-pass key: password - name: MYSQL_USER value: vamsi - name: MYSQL_PASSWORD valueFrom: secretKeyRef: name: mysql-pass key: userpass - name: MYSQL_DATABASE value: my-drupal-db ports: - containerPort: 3306 name: mysql volumeMounts: - name: mysql-persistent-storage mountPath: /var/lib/mysql volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: efs-mysql
drupal-cms.yaml - this creates our main deployment Drupal.
apiVersion: v1
kind: Service
metadata:
name: drupal-service
labels:
app: drupal
spec:
ports:
- port: 80
selector:
app: drupal
tier: frontend
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: drupal
name: drupal
spec:
selector:
matchLabels:
app: drupal
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: drupal
tier: frontend
spec:
initContainers:
- name: init-sites-volume
image: drupal:8.6
command: ['/bin/bash', '-c']
args: ['cp -r /var/www/html/sites/ /data/; chown www-data:www-data /data/ -R']
volumeMounts:
- mountPath: /data
name: vol-drupal
containers:
- image: drupal:8.6
name: drupal
ports:
- containerPort: 80
volumeMounts:
- mountPath: /var/www/html/modules
name: vol-drupal
subPath: modules
- mountPath: /var/www/html/profiles
name: vol-drupal
subPath: profiles
- mountPath: /var/www/html/sites
name: vol-drupal
subPath: sites
- mountPath: /var/www/html/themes
name: vol-drupal
subPath: themes
volumes:
- name: vol-drupal
persistentVolumeClaim:
claimName: efs-drupal
Note: Before running these files, you have to install amazon-efs-utils client software in all the worker nodes,use remote login via ssh command i.e ,
ssh -i <<eks-course.pem>> ec2-user@<<ec2-workernode>> "sudo yum install -y amazon-efs-utils"
Now let's run the kustomization file to built entire architecture.
cmd:kubectl create -k .
Now , everything fine and i get a DNS address given by ELB to access my site.
Getting my Drupal site..and configuring done step by step..
Now configure your site details..
Finally, my site is here...
Let's create some content and publish...
Read more..
Thus , how we managed to complete our main task .Let me discuss one more thing i.e, till now what we did in a cluster ,which we pre planned to construct.But what if some service there to plan everything for us?? For example you have a cluster but you don't want to worry about the worker nodes and where and how many they are running but should save my resources and whenever i require something it should be created on the fly by the cluster.
Is it possible , yes it is .It is known as Serverless architecture provided by amazon ECS in the name of Fargate Cluster.Let's see what exactly it is.
Amazon ECS on AWS Fargate
AWS Fargate is a technology that you can use with Amazon ECS to run containers without having to manage servers or clusters of Amazon EC2 instances. With AWS Fargate, you no longer have to provision, configure, or scale clusters of virtual machines to run containers. This removes the need to choose server types, decide when to scale your clusters, or optimize cluster packing.
Let's quicky launch a Fargate-cluster and run a pod over there.
Create the cluster by running this simple code...of fargate.yml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: f-lwcluster
region: ap-southeast-1
fargateProfiles:
- name: fargate-default
selectors:
- namespace: kube-system
- namespace: default
run using cmd: eksctl create cluster -f fargate.yml
Even though the cluster created , we observe no worker nodes running on EC2.
Now update the kubeconfig file to access cluster using cli.
Now if you see , this cluster has two worker nodes running on their internal datacenters.
Now let's run a pod with some image , to check whether the cluster is working or not.
That's all guys....Let us destroy both clusters as amazon charges.
Deleting eks cluster..
Deleting Fargate cluster..
Note: You can't be able to launch this fargate cluster in all regions , it work on only few regions. check the link for fargate information.
Challenges faced during the task:
> When i started with the task , i found Drupal which is similar to Wordpress , and i just tried implementing with on docker first ,next on kubernetes using minikube but the thing is that ,the official Drupal image doent't support any environment variables , hence i set up one time installation above.
>Next thing is that i read many blogs and visited many sites to know about the folder that makes Drupal set up data persistent, finally i managed somehow and find out some folders and with the help Init containers resource ,I completed my task successfully.
Okay, that's all about the Task.
My journey of EKS :
Firstly , i'm very grateful to Vimal sir ,for this program which is very advanced for making free for all us who were in Hybrid_Multi_Cloud training. Even though we are learning a lot in Multi cloud but you are very concern about us and made this training available for us,
In this two day journey of EKS , I learnt many advanced concepts and with the task provided made me to practice everything.As we all know setting up a multi node cluster by ourselves is a tough job but with the help of AWS we are going to set this just with some clicks.
I also learnt , how to integrate the aws services such as EBS, EFS,ELB with EKS by the wordpress and Mysql example .Also many real use cases i learnt within just a span of two days.
Finally , I once again thank Sir,Vimal Daga for his motivation and efforts in Making India Future Ready.
Thanks for reading...
Github code :
Any queries or feedback connect me on LinkedIn:
Please do like my work, if u believe.Thank You.....onceagain for viewing.