Episode 6: Redefining the Hiring Process: AI-Powered Recruitment Using Natural Selection and Genetic Algorithms
From Idea to Reality – App Development!
Introduction
Welcome back to the final episode of our AI-powered recruitment series!
In this episode, we bring everything together and mark a major milestone—by developing and deploying the core functionality of GeniMatch AI, we’ve successfully proven our concept.
We’ll dive into the full development and deployment process, starting with ingesting resumes, analyzing them, processing job postings, and intelligently matching candidates using AI and genetic algorithms. Then, we’ll cover deploying the app on Kubernetes and automating the deployment using ArgoCD and GitHub Actions while saving money using AWS Graviton instances.
We’ll break down the tech stack, the challenges we tackled, and how we transformed this idea into a working solution. Let’s wrap up the series with a deep dive into how GeniMatch AI is now up and running!
Let’s get started!
Content:?
Developing the application
Now, let’s break down how we turned GeniMatch AI from an idea into a working solution!
Resumes Module: Managing Candidate Profiles
A core aspect of our system is handling resumes efficiently. We implemented a module that allows users to:
Instead of storing full resumes as-is for matching, we focused on extracting only the most relevant information using AI. These extracted insights are then stored in a vector database, and used later during the AI matching process.
Example: Resumes Listed in the App
Jobs Module: Defining Job Profiles
To effectively match candidates, we needed a robust job management system. This module allows:
Since GeniMatch AI is a multi-tenant service, it was essential to consider company-specific preferences when managing job postings. Each company can set custom requirements, which influences how candidates are ranked and selected.
Example: Job Management Interface shows subset of the job description fields
Example: Job Lists
Settings Module: Customizing the AI Matching Process
To allow flexibility in how genetic algorithms operate, we introduced several parameters:
For this proof of concept (PoC), I didn’t develop a UI-based settings module. Instead, we passed these parameters as environment variables, enabling quick experimentation. If we decide to productize this, we’ll build a proper interface to allow users to configure these parameters dynamically.
Candidate Matching Process: The Core of GeniMatch AI
This is where things get exciting! The matching process consists of three major steps:
Step 1: Candidate-Job Profile Population Creation
Once a user selects a job, they can trigger the matching process, which kicks off population creation. The system:
Step 2: Mutation – Enhancing Diversity in Selection
In genetic algorithms, mutation introduces diversity into the population, helping explore non-obvious but potentially valuable candidates.
In our system, mutation works by:
The goal of mutation is to avoid tunnel vision—sometimes, great candidates don’t have an exact keyword match but still bring valuable skills to the table.
Example: Candidate Pool After Mutation
Step 3: Refinement & Final Selection
After mutation, users can:
At this stage, only the best candidates will move forward for further evaluation.
Future Steps & Productization
With the proof of concept completed, the next step is turning GeniMatch AI into a fully developed product. Future developments may include:
领英推荐
The foundation is now in place—it's just a matter of expanding and refining the system!
Deploying the application
In the last article we provisioned the infrastructure for kubernetes and all of the components that make it useful. In this week's episode we will connect all of this together, and deploy a fully functioning application with a single command.?
We're deploying the application and transitioning to Graviton ARM instances for a more cost-effective solution. We also made these instances spot instances, and made two of them available in different availability regions. This shift introduced some challenges since GitHub currently doesn't offer ARM builders for private repositories on free accounts.
Switching to Graviton instances
ARM64 Graviton instances are cheap, and performance is good for the low cost price. I converted the instance type to spot to save even more money, and increase the node count to two. This way in the case of a spot deallocation event, the other availability zone should be fine.
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 20.0"
cluster_name = local.name
cluster_version = "1.31"
cluster_endpoint_public_access = true
cluster_endpoint_public_access_cidrs = ["${chomp(data.http.public_ip.response_body)}/32"]
create_cloudwatch_log_group = false
cluster_enabled_log_types = []
cluster_addons = {
coredns = {}
eks-pod-identity-agent = {}
kube-proxy = {}
vpc-cni = {}
}
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
eks_managed_node_groups = {
genai = {
ami_type = "AL2_ARM_64"
instance_types = ["t4g.medium"]
min_size = 2
max_size = 2
desired_size = 2
capacity_type = "SPOT"
labels = {
"node-lifecycle" = "spot"
}
}
}
tags = local.tags
depends_on = [
module.nat
]
}
Setting up the ArgoCD to deploy the application
In order for our application to deploy using GitOps we need to authenticate the ArgoCD application with GitHub. We create a secret in AWS Secrets manager to hold this information. Using this information
{
"username": "username",
"password": "github token",
"url": "https://github.com",
"project": "default"
}
Within our infrastructure chart, we define an ExternalSecret to retrieve the secret and generate a Kubernetes secret with the correct ArgoCD credentials. A proper label is required to inform ArgoCD it is a repository secret.?
./charts/infrastructure/argocd-secret.yaml
kind: ExternalSecret
metadata:
name: private-repo
namespace: argocd
spec:
refreshInterval: 1h
secretStoreRef:
name: aws
kind: SecretStore
target:
name: private-repo
creationPolicy: Owner
template:
metadata:
labels:
argocd.argoproj.io/secret-type: repository
type: Opaque
data:
- secretKey: url
remoteRef:
key: my-argo-repo-secret
property: url
- secretKey: username
remoteRef:
key: my-argo-repo-secret
property: username
- secretKey: password
remoteRef:
key: my-argo-repo-secret
property: password
- secretKey: project
remoteRef:
key: my-argo-repo-secret
property: project
Now that we have the credentials in order to authenticate with github, we can now create our ArgoCD application and deploy it. We create this inside the infrastructure chart.
./charts/infrastructure/app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: smarthiring
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/repo
targetRevision: main
path: charts/app
helm:
values: |
filesBucket: {{ .Values.filesBucket }}
vectorBucket: {{ .Values.vectorBucket }}
aws:
postgres_secret_keyname: {{ .Values.postgresSecretName }}
db:
url: {{ .Values.postgresURL }}
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: {{ .Values.appRole }}
destination:
server: https://kubernetes.default.svc # Uses in-cluster Kubernetes
namespace: smarthiring
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
The role gets passed into the helm chart which gets appended on the service account where the proper annotations get applied. This ensures our role can authenticate to AWS services. The postgres secret is also required for accessing the AWS Secret that terraform created.?
I have introduced a few more variables that we are passing in from terraform. We need a few bucket names, along with the postgres secret name. The updated terraform code to deploy the infrastructure chart is:
resource "helm_release" "infrastructure" {
name = "infra"
chart = "../../charts/infrastructure/"
values = [
<<-EOT
roleDNS: ${aws_iam_role.dns.arn}
roleSecrets: ${aws_iam_role.secrets.arn}
region: ${local.region}
dnsZone: ${aws_route53_zone.zone.name}
appRole: ${aws_iam_role.app.arn}
postgresURL: ${split(":", aws_db_instance.postgres.endpoint)[0]}
postgresSecretName: ${aws_secretsmanager_secret.db.name}
filesBucket: ${local.name}-files-${random_string.name.result}
vectorBucket: ${local.name}-vector-${random_string.name.result}
EOT
]
depends_on = [
aws_eks_access_policy_association.self_admin,
helm_release.argocd,
aws_db_instance.postgres,
module.eks
]
}
Once we have this deployed, argocd will start to deploy the helm chart, and thus our application.?
Creating the helm chart for the application
The helm chart is used to deploy the application, and allows us to configure the app in a dynamic way in case the environment changes. We can create a simple sample helm chart by typing helm create app and that will create a default helm chart with a lot of great best practices.?
We can edit the values.yaml and add a few properties our application requires.
db:
url: FillIn
env:
POSTGRES_USERNAME: admindude
aws:
enabled: true
postgres_secret_keyname: postgres-db-password-tLzR
s3:
vectorBucket: FillIn
filesBucket: FillIn
Our application requires access to the Postgres database, along with S3 buckets. We will need to modify the default helm chart to add these values in. Here I am adding a default environment area so that I can easily add a new environmental variable at a later time without modifying the helm chart.?
I am also creating an environmental variable based on a secret.
./charts/app/templates/deployment.yaml
containers:
imagePullPolicy: {{ .Values.image.pullPolicy }}
env:
- name: CV_S3_BUCKET
value: {{ .Values.s3.filesBucket }}
- name: VECTORDB_S3_BUCKET
value: {{ .Values.s3.vectorBucket }}
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "app.fullname" . }}-pgsql-passwd
key: secretValue
Since we are in a dynamic environment we need to read in the postgres secret from AWS Secrets Manager to create a kubernetes secret.
./charts/app/templates/secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: {{ include "app.fullname" . }}-pgsql-db
labels:
{{- include "app.labels" . | nindent 4 }}
spec:
refreshInterval: 60m
secretStoreRef:
name: aws
kind: SecretStore
target:
name: {{ include "app.fullname" . }}-pgsql-passwd
data:
- secretKey: secretValue
remoteRef:
key: {{ .Values.aws.postgres_secret_keyname }}
I have configured the ingress via values.yaml for now. Because we have specified annotations in this ingress object, the environment will create us a TLS certificate, along with a DNS A record.?
?This is the magic that will ensure our application will be reachable in our browser, after the terraform applies.?
ingress:
enabled: true
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
external-dns.alpha.kubernetes.io/hostname: "hire.linuxstem.com"
acme.cert-manager.io/http01-edit-in-place: "true"
hosts:
- host: smarthire.host.com
paths:
- path: /
pathType: ImplementationSpecific
tls:
- secretName: app-tls
hosts:
- smarthire.host.com
Creating a bucket for the application
This application requires two buckets. This module will create a bucket along with the proper permissions to ensure the bucket is not public.?
module "s3_vectordb" {
source = "terraform-aws-modules/s3-bucket/aws"
version = "4.2.2"
bucket = local.s3_bucket_vectordb
force_destroy = true
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
control_object_ownership = true
object_ownership = "BucketOwnerPreferred"
expected_bucket_owner = data.aws_caller_identity.current.account_id
server_side_encryption_configuration = {
rule = {
apply_server_side_encryption_by_default = {
sse_algorithm = "AES256"
}
}
}
}
in a similar way, "s3_files" bucket is provisioned
We need an IAM policy to ensure our role can write to the buckets. This policy only gives access to the specific buckets we are creating for this app.?
data "aws_iam_policy_document" "s3" {
statement {
sid = "AllowS3ReadWriteAccess"
effect = "Allow"
actions = [
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket"
]
resources = [
"arn:aws:s3:::${module.s3_vectordb.s3_bucket_id}",
"arn:aws:s3:::${module.s3_vectordb.s3_bucket_id}/*",
"arn:aws:s3:::${module.s3_files.s3_bucket_id}",
"arn:aws:s3:::${module.s3_files.s3_bucket_id}/*"
]
}
}
resource "aws_iam_policy" "s3" {
name = "s3-for-app-${local.name}"
policy = data.aws_iam_policy_document.s3.json
tags = local.tags
}
resource "aws_iam_policy_attachment" "s3" {
name = "s3-for-app-attachment
roles = [aws_iam_role.app.name]
policy_arn = aws_iam_policy.s3.arn
}
Creating a GitHub action to build an ARM64 image
With this we will have a helm chart for infrastructure, and the application! Now how do we build an ARM64 image? The answer is QEMU, and cross platform building. QEMU is a virtualization stack that also allows you to emulate binaries from other architectures. We can set up a GitHub action pipeline to build an ARM64 image, and push that to AWS ECR!
name: Build and Push to AWS ECR
on:
push:
branches:
- main
workflow_dispatch:
permissions:
contents: write
jobs:
build-and-push:
name: Build and Push Docker Image
runs-on: ubuntu-latest
steps
- name: Checkout Repository
uses: actions/checkout@v3
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v2
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Build Docker Image for ARM64
run: |
cd dynamic-website/smart_hiring/
docker buildx build --load --platform linux/arm64 -t local:$GITHUB_SHA .
- name: Tag Docker Image
run: |
docker tag local:$GITHUB_SHA 024848472643.dkr.ecr.us-east-1.amazonaws.com/smarthiring:$GITHUB_SHA
- name: Push Docker Image to ECR
run: |
docker push 000000000000.dkr.ecr.us-east-1.amazonaws.com/repo:$GITHUB_SHA
The important bits for building ARM images are the following two steps. The first one sets up the QEMU binaries, and the second ensures docker is setup for BuildX.
??? - name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
After we have the build environment configured, we can specify that we want to build using the linux/arm64 platform. Which will use the QEMU emulation layer. This takes a little bit longer to build, but it allows you to build anywhere, on anything.?
?????????docker buildx build --load? --platform linux/arm64 -t local:$GITHUB_SHA .
After the application is built, and pushed to AWS ECR, we need to tell ArgoCD that the image has changed. We do this by changing the image tag in the values.yaml file for the helm chart.?
update-image-tag:
name: Update image tag
needs: build-and-push
runs-on: ubuntu-latest
env:
CHART_PATH: 'charts/app'
steps:
- name: Checkout Repository
uses: actions/checkout@v3
- name: Install yq
run: |
wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq
chmod +x /usr/local/bin/yq
- name: Update values.yaml with New Image Tag
run: |
yq e -i '.image.tag = env(GITHUB_SHA)' $CHART_PATH/values.yaml
echo "Updated image.tag in values.yaml to $GITHUB_SHA"
- name: Commit and Push Changes
run: |
git config --global user.name "github-actions"
git config --global user.email "[email protected]"
git add $CHART_PATH/values.yaml
git commit -m "Update image tag to $GITHUB_SHA [skip ci]" || echo "No changes to commit"
git push origin main || echo "No changes to commit"
This GitHub action will install “yq”, which is a command line utility that allows us to easily change yaml files. We set up the Git environment, change the values.yaml, and then push the commit. The commit message has a specific text to not fire off another build. This will tell ArgoCD that the image has changed, and result in a new deployment.?
Next Step?
This journey has been an incredible opportunity to explore the collaboration between two experts from different domains—DevOps and AI.?
There’s so much more we can share from this collaboration! We’re considering writing a detailed article covering:
We hope this inspires others looking for the right teammate to collaborate with!
Authors:?
Noor Sabahi and Ryan Young
?
#AIRecruitment #GeneticAlgorithms #NaturalSelection #HiringInnovation #AIinHiring #RecruitmentTech #FutureOfHiring #MachineLearning #SmartHiring #AIForRecruitment #AIDrivenApp #LLM #GenerativeAI #RecruitmentRevolution #DataDrivenHiring #AIJobMatching #ArtificialIntelligence