Getting started with secure SSL protected micro-services with JHipster, Keycloak, Nginx and Kubernetes - Part 5: Deployment
Links
Part 1: Intro and architecture
Part 2: Kubernetes and Nginx
Part 3: Keycloak
Part 4: Spring Boot development with JHipster
Part 5: Deployment <-- This article
Deployment to Kubernetes
Deployment concept
- We have already seen how our Keycloak was deployed to Kubernetes. Now, we are going to deploy our micro-services there also.
- The deployment we do here will try to emulate production deployment as much as we can.
- Firstly, let us visit the concept behind Kubernetes deployment. We have already deployed Keycloak using Helm. Fortunately, JHipster comes with a sleuth of its own tools for Kubernetes deployment. In a real CI/CD set up, source code would be pushed to a source code repository. From there we would build it and submit it to Docker Hub (https://hub.docker.com). From Docker Hub, we would then deploy the binaries into a Kubernetes cluster.
- In our tutorial set up, we do not have all the CI/CD components. Nonetheless, we will still push, from our development machine, to Docker Hub, before deploying to our Kubernetes cluster.
Deployment setup
- To avoid port conflict, please shut down both banking and gateway micro-services
- Firstly, we need to change the issuer-url to point to Keycloak from inside Kubernetes. For both $PROJECTS/ebanking/gateway/src/main/resources/config/application.yaml and $PROJECTS/ebanking/banking/src/main/resources/config/application.yaml
- Find the properties
security:
oauth2:
client:
provider:
Customise it as per below:
security:
oauth2:
client:
provider:
oidc:
issuer-uri: https://keycloak-http.default.svc.cluster.local:8090/auth/realms/ebanking
registration:
oidc:
client-id: ebankingclient
client-secret: <client secret>
- Notice that issuer URI is now pointing to the internal (to Kubernetes) Keycloak URL. Save the files.
- To start deployment setup, we need a Docker Hub id. Go to https://hub.docker.com. Register for an account.
- Then, fire up a command line console in the $PROJECTS/ebanking/k8s folder. Run the command
> jhipster kubernetes
- The choices presented are:
- Which * type *- choose Microservice application
- Enter the root directory - put your $PROJECTS/ebanking directory
- When asked which application do you want to include - choose both banking and gateway
- Make sure you enter the registry admin password
- For Kubernetes namespace - choose default
- For base Docker repository - use your Docker Hub username
- To push docker images - choose docker push
- For istio - set to No
- For Kubernetes service type for edge service - choose NodePort
- Once successful you will see the screen below
WARNING! Kubernetes configuration generated, but no Jib cache found If you forgot to generate the Docker image for this application, please run: To generate the missing Docker image(s), please run: ./mvnw -ntp -Pprod verify jib:dockerBuild in $PROJECTS/ebanking/banking ./mvnw -ntp -Pprod verify jib:dockerBuild in $PROJECTS/ebanking/gateway WARNING! You will need to push your image to a registry. If you have not done so, use the following commands to tag and push the images: docker image tag banking <your Docker Hub id>/banking docker push <your Docker Hub id>/banking docker image tag gateway <your Docker Hub id>/gateway docker push <your Docker Hub id>/gateway INFO! Alternatively, you can use Jib to build and push image directly to a remote registry: ./mvnw -ntp -Pprod verify jib:build -Djib.to.image=<your Docker Hub id>/banking in $PROJECTS/ebanking/banking ./mvnw -ntp -Pprod verify jib:build -Djib.to.image=<your Docker Hub id>/gateway in $PROJECTS/ebanking/gateway You can deploy all your apps by running the following kubectl command: bash kubectl-apply.sh -f [OR] If you want to use kustomize configuration, then run the following command: bash kubectl-apply.sh -k Use these commands to find your application's IP addresses: kubectl get svc gateway -n ebanking
Push image to Docker Hub
- Go to your $PROJECTS/ebanking/banking folder and fire up a command line console. Run the command below. This will compile your code (if need be) and deploy it to Docker Hub.
>./mvnw -ntp -Pprod verify jib:build -Djib.to.image=<your Docker Hub id>/banking
- Next, go to your $PROJECTS/ebanking/gateway folder and fire up a command line console. Run the command below. This will compile your code (if need be) and deploy it to Docker Hub.
>./mvnw -ntp -Pprod verify jib:build -Djib.to.image=<your Docker Hub id>/gateway
- Lets verify that both docker images have been pushed to Docker Hub . Go to Docker Hub (go to hub.docker.com or use Docker Desktop menu) and log in.
- You should be able to see your Docker images.
Download image from Docker Hub to Kubernetes
- Now it is time to pull back those Docker images into our Kubernetes container. In the $PROJECTS/ebanking/k8s folder, run the command below:
> bash kubectl-apply.sh -f
- To see if your images are pulled and deployed properly, run the command line:
> kubectl get pods
- You will get the output below:
NAME READY STATUS RESTARTS AGE banking-bbdf47c5b-stfg9 1/1 Running 0 133m banking-postgresql-646f5f8d5c-xnwgm 1/1 Running 0 133m gateway-77697476b7-wbm2q 1/1 Running 0 133m gateway-postgresql-8687dd66b5-dcgdf 1/1 Running 0 133m jhipster-registry-0 1/1 Running 0 133m jhipster-registry-1 1/1 Running 0 133m
- You can see all our components are there. Keycloak, JHipster registry, banking and gateway. Make sure the status is ‘Running'
- We have now completed our deployment to Kubernetes
Setup Ingress for micro-services
- Lastly, we need to setup Ingress for our micro-services. Firstly, recall that we already setup an Ingress for our Keycloak. We will edit that Ingress and add a path for our Gateway
- Go to our $PROJECTS/ebanking/k8s folder. In there, open up our file my-ingress.yaml. Put in the content below
- The hosts must contain the common name of the certificate set in the paragraph - Ingress Setup and Security.
- The secretName property must contain the name of our secret created in the paragraph - Ingress Setup and Security.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/server-snippet: |
location /keycloak-auth/ {
proxy_pass https://keycloak-http.default.svc.cluster.local:8090/auth/;
}
spec:
tls:
- hosts:
- localhost
secretName: my-gw-certs
rules:
- http:
paths:
- path: /
backend:
serviceName: gateway
servicePort: 18080
- path: /auth
backend:
serviceName: keycloak-http
servicePort: 8090
- Notice that we have added the gateway service:
path: /
backend:
serviceName: gateway
servicePort: 18080
- Save the file
- Delete the old ingress
> kubectl delete ingress my-ingress
- In the same folder run :
> kubectl apply -f my-ingress.yaml
- Once the command is successful, you are now done with SSL ingress setup for both Keycloak and micro-services.
Testing micro-services running in Kubernetes
- Firstly, lets get our access key. Run the command:
> curl -kX POST -vu client:ebankingclient https://localhost/keycloak-auth/realms/ebanking/protocol/openid-connect/token -H "Accept: application/json" -d "username=donald.duck&password=<password>&grant_type=password&scope=read&client_id=ebankingclient&client_secret=<client secret>"
- Make sure
- You change <password> to what ever password that was set up for the user donald.duck in the paragraph 'Install and configure Keycloak on Kubernetes - Create roles and users in Keycloak’.
- You change <client secret> to what we copied from Keycloak before
- Again, you will get a long response:
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to keycloak-http.default.svc.cluster.local (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=localhost
* start date: Mar 31 04:06:30 2020 GMT
* expire date: Mar 29 04:06:30 2030 GMT
* issuer: CN=localhost
* SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Server auth using Basic with user 'client'
* Using Stream ID: 1 (easy handle 0x7fc09380ec00)
> POST /auth/realms/ebanking/protocol/openid-connect/token HTTP/2
> Host: keycloak-http.default.svc.cluster.local:8443
> Authorization: Basic Y2xpZW50OmViYW5raW5nY2xpZW50
> User-Agent: curl/7.54.0
> Accept: application/json
> Content-Length: 143
> Content-Type: application/x-www-form-urlencoded
>
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< cache-control: no-store
< set-cookie: KEYCLOAK_LOCALE=; Version=1; Comment=Expiring cookie; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/ebanking/; HttpOnly
< set-cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/ebanking/; HttpOnly
< pragma: no-cache
< content-type: application/json
< content-length: 2604
< date: Tue, 31 Mar 2020 09:13:15 GMT
<
* Connection #0 to host keycloak-http.default.svc.cluster.local left intact
{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJDRk1Gc3JreDhRRjF6bm5RM2JzSHo1VUxGeE9HNkxKbkJsWGV4c09pb1ZjIn0.eyJqdGkiOiJmOTk1ZDhlYi0xYjU1LTQ1ZWYtYTVkMy1hNjY4ZjE0YzE3NzAiLCJleHAiOjE1ODU2NDYyOTUsIm5iZiI6MCwiaWF0IjoxNTg1NjQ1OTk1LCJpc3MiOiJodHRwczovL2tleWNsb2FrLWh0dHAuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbDo4NDQzL2F1dGgvcmVhbG1zL2ViYW5raW5nIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6ImY0NTY2MTUzLTM5NDEtNGYzNC04MWNlLWIxMTcwYTc1NzlhZSIsInR5cCI6IkJlYXJlciIsImF6cCI6ImViYW5raW5nY2xpZW50IiwiYXV0aF90aW1lIjowLCJzZXNzaW9uX3N0YXRlIjoiN2U2NWY3NTktM2E4Ny00YWVlLWE3ODktZjBiMGNjYWYwN2YwIiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyJodHRwOi8vbG9jYWxob3N0OjE4MDgwIl0sInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJST0xFX1VTRVIiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJwcm9maWxlIGVtYWlsIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJuYW1lIjoiRG9uYWxkIER1Y2siLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJkb25hbGQuZHVjayIsImdpdmVuX25hbWUiOiJEb25hbGQiLCJmYW1pbHlfbmFtZSI6IkR1Y2siLCJlbWFpbCI6ImRvbmFsZC5kdWNrQGNvbXBhbnkuY29tIn0.EF4KxcCvOqje4NFfE4-Zqz4QV4QmzwG_bQF2iotkRZgp-m1F1LMVq_ZyFVMM7okAmmMpeZAdFMykRMf4afPNH5vuF3Oxih7CW6_7oYMjf6mjqMskA0Yi_qALf7Bdg8KIDnAVyHvKHTYQWLRagg-W0lxo8Pf7FL0SSyo9Xfaxybp1rQMP4Pc7TIhrQdHourV485xASxGcHsPeNM92cbW7SoSPEPdw_AQ7U7M2T0fklFUIXNpfRBHyAQYrmsdjzd-MiAUzmu10l3jwxTcnKJDMF-xtd4BgBQLLzJC9ulDF74jrvVHCunZJ22R6_1R1fe4WwVAcFcEvAdko7U--YUvwJg","expires_in":300,"refresh_expires_in":1800,"refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI4ZjQzZmYxZS1lNTkwLTRlMTUtOTZhYi0yZTJkZTQ1ZjM3OWIifQ.eyJqdGkiOiJkYTI4ZjRkZS01ZDI3LTQ2ZTUtYmE5MS0wNjNhZWFhNWQxYmEiLCJleHAiOjE1ODU2NDc3OTUsIm5iZiI6MCwiaWF0IjoxNTg1NjQ1OTk1LCJpc3MiOiJodHRwczovL2tleWNsb2FrLWh0dHAuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbDo4NDQzL2F1dGgvcmVhbG1zL2ViYW5raW5nIiwiYXVkIjoiaHR0cHM6Ly9rZXljbG9hay1odHRwLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw6ODQ0My9hdXRoL3JlYWxtcy9lYmFua2luZyIsInN1YiI6ImY0NTY2MTUzLTM5NDEtNGYzNC04MWNlLWIxMTcwYTc1NzlhZSIsInR5cCI6IlJlZnJlc2giLCJhenAiOiJlYmFua2luZ2NsaWVudCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6IjdlNjVmNzU5LTNhODctNGFlZS1hNzg5LWYwYjBjY2FmMDdmMCIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJST0xFX1VTRVIiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJwcm9maWxlIGVtYWlsIn0.S2a2GL1kx5kvEfk4xP1a1vNUZFhLItXInMAj0pQHfpY","token_type":"bearer","not-before-policy":0,"session_state":"7e65f759-3a87-4aee-a789-f0b0ccaf07f0","scope":"profile email"}
- To add data, run the command below. Make sure
- <access token> is replaced with the access token obtained above
- You do not forget the single quote in the end of the command below
curl -X POST \
https://localhost/services/banking/api/customers \
-H 'Authorization: Bearer <access token>' \
-H 'Content-Type: application/json' \
-H 'Postman-Token: da78b930-e79b-45cc-ad4d-0b6930b4c825' \
-H 'cache-control: no-cache' \
-d '{
"cifNumber" : "ABCD123456789",
"firstName" : "Kiana",
"lastName" : "Bosco",
"icNumber" : "111111-11-1111",
"phoneNumber" : "+6012345678"
}
'
- If all goes well, the command will return you the data created - with id populated as per below:
{"id":1051,"cifNumber":"ABCD123456789","firstName":"Kiana","lastName":"Bosco","icNumber":"111111-11-1111","phoneNumber":"+6012345678"}
- To read back data, use the command below. Make sure
- <access token> is replaced with the access token obtained above
> curl https://localhost/services/banking/api/customers -H "Authorization: Bearer <access token>"
- If all goes well, you should be able to see the data we saved just now:
[{"id":1051,"cifNumber":"ABCD123456789","firstName":"Kiana","lastName":"Bosco","icNumber":"111111-11-1111","phoneNumber":"+6012345678"}
- Congratulations! You have developed, tested, deployed and run your secure micro-services successfully.
Tips and tricks
Deleting your work in Kubernetes
- If you have made a mistake and accidentally uploaded the wrong container to Kubernetes, you can simply delete that deployment
- Note: You do not just delete a pod. A pod is ephemere and will be restarted by the system. You need to delete a deployment instead
- To get all deployments in the “default” namespace
> kubectl get deployments
- To delete
> kubectl delete deployment <deployment name>
Problem connecting to the database
- In a real production environment, we would recommend database to be set up outside of the cluster. In a cloud environment, it is better to use cloud based database services such as RDS
- This is because, there are times when connectivity between services to database is problematic. One way to get over this is to delete the persistence volume of the database. Use the command below to see all persistence volume in the “default” namespace
> kubectl get persistentVolumeClaim
- Use the command below to delete. Make sure deployment is deleted first
> kubectl delete persistentVolumeClaim <persistence volume name>
- You can then recreate the persistence volume by going to $PROJECTS/ebanking/k8s and running
> bash kubectl-apply.sh -f
My Ingress not working
- If you have created your Ingress before you setup your Ingress controller, your Ingress will not work
- Delete the Ingress and recreate it
Uninstall using Helm
- If you have installed something using Helm, you can also use Helm to uninstall
> helm uninstall <the name of chart installed>
Troubleshoot my URL transformation
- Sometimes you need to see how your URL or request transformation happens.
- You can install an echo service that will return you the exact thing you send to it https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/customization/external-auth-headers/deploy/echo-service.yaml
> kubectl apply -f https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/customization/external-auth-headers/deploy/echo-service.yaml
- To integrate it to your ingress, just add another path in my-ingress.yaml file. Delete the old ingress and apply the new one:
- path: /echo
backend:
serviceName: echo-service
servicePort: 80