Inferring Internal Job Levels from Public Listings Using ChatGPT

One of my pain points as a job seeker is that sometimes it is not clear from the job listing whether it is at a job level I am happy with, even if I meet the job requirements. This is because job requirements can sometimes be overinflated or understated by the person/department who posted the job.

Very often, I need to eyeball other job listings posted by the company to guess the job level, which I find quite mentally taxing. Sometimes sites like levels.fyi and glassdoor.sg help, but they aren’t always descriptive and not all companies have job levels listed on the site.

Well, now my eyeballing can be done with an LLM like ChatGPT too. Using DBS as an example because I happened to find many job listings on linkedin and it is one of the biggest employers in Singapore with a wide range of job functions, ChatGPT managed to do the following:

1)?????Produced a job levelling system for DBS from 7 benchmark jobs selected by me and extrapolated to job levels unseen in the benchmark jobs provided (with varying consistency)

2)?????Explained the rationale of the job levelling based on the benchmark jobs and described in broad terms its understanding of the organizational structure of DBS

3)?????Made the job levelling system specific to a particular function within DBS

While the immediate use case for this little experiment is to reverse engineer job listings, this is also an example of how one might go about using an LLM to produce internal job levels. In this task, ChatGPT is also implicitly comparing jobs from different functions and different parts of the organization, which I find quite interesting from an org design perspective.

As usual, the lengthy details, JDs, prompts, replies, etc, can be found in the appendix. I will cover point 1 in this article, but leave point 2 and 3 for a future blog post. However, feel free to take a peek at the outputs logged in the appendix. I've organized the data according to the conversations I've had with ChatGPT.

Obtaining rough job levels from public job listings

This is used a 3 step process:

1. Select benchmark jobs
2. Get ChatGPT to summarize the benchmark jobs
3. Prompt ChatGPT with job summaries

Step 2 is optional if you have enough money.

Step 1: Select benchmark jobs

The selection of the benchmark jobs was not random, I chose based on a diversity of job functions and tried to cover the maximum job level range, but it was not very rigorous either as it took about 10mins of filtering job posts on LinkedIn’s system.

Here are the jobs in the order I presented them to ChatGPT:

• VP/AVP, Data Translator, Analytics Center of Excellence, Group Transformation
• Associate, Site Reliability Engineer, Middle Office Technology, Technology & Operations
• Analyst / Senior Officer, Customer Service Officer, Customer Centre, Technology and Operations
• Data Centre Batch Management Operator, Technology & Operations
• SVP / VP, Equity Advisory, Private Banking, Consumer Banking Group
• Head of Credit Risk Monitoring & Reporting
• Executive Director, Relationship Manager, Private Banking (NRI Market), Consumer Banking Group

Step 2: Get ChatGPT to summarize the benchmark jobs

For now, the free version of ChatGPT that I am using does not have an infinite context window, so I can’t just give the raw text of the benchmark jobs into the model. Instead, I first get ChatGPT to summarize the jobs into 50 words each. I assume this will not be necessary on better LLMs.

Step 3: Prompt

This is simply copy and pasting the job summaries into the prompt window and providing some context about DBS. There can also be a few specifications done within the prompt, one of the cool things was explicitly asking the model to predict the missing job levels that are not part of the benchmark jobs.

The full model output is too verbose to copy here, but here is a side-by-side comparison between job levels without benchmarks and job levels with the benchmark jobs:

My Thoughts

It seems that based on the knowledge that the job levelling system is for a bank, the LLM already has some understanding of what titles a bank might have, and proposed 5 broad bands of jobs. When provided with benchmark jobs, it gives greater details at each job level and seems to be picking up specific keywords from the benchmark jobs such as “transformation initiatives” and “end-to-end solution delivery” that are not present in the generic descriptions. It is also picking up on the years of experience requirements, although it is worth nothing that it has chosen years of experience bounds that are different from that in the benchmark jobs.