Game Over for BAs?

(I started writing this as a short essay on what ChatGPT and AI means for BAs. But it has already snowballed into at least 3 posts. For all I know it may not even end there :) And like I always say, every story is told from the point of view of that narrator).

Since of late, ChatGPT has been the main topic of discussion when it comes to anything regarding technology. This has become so associated with AI, that now it has almost become synonymous with AI and ML. Be it a boardroom discussion, a team meeting or a pub chat, you can be sure that one way or the other ChatGPT will make an appearance.?

One of the biggest fears of ChatGPT has been its uncanny ability to do some tasks that humans do at least on par or sometimes even better. Many questions have been raised around what this means to the workforce and recently Washington Post ran an article about how this fear has now become a materialised threat where many who are in the creative writing professions have been made redundant as ChatGPT can do a more financially efficient job than employing people. The question of job replacements has not escaped the tech landscape either, many tech forums where newbie software developers chat have been abuzz with whether AI alternatives such as Co-Pilot or ChatGPT itself, could replace their jobs. My one field of work, Business Analysis has not been completely immune to this, but we were also taking it a bit lightly; “You guys can’t figure out what the requirements are with a human, try doing it with a machine ;)”

On a recent vlog/podcast that I did with a friend of mine about what BAs do and their role in the software development cycle, I ran an interesting experiment on this. As BAs our primary task is to be a bridge between the client and the development team. We are to get the requirements from the client, document them and explain to the development and the testing team what the requirements are, become a proxy to the client and eventually help the team deliver a solution to the client. In this process of doing our job, we use different tool sets and standards.These include the ubiquitous UML and sequence diagrams, db schemas, mindmaps, fishbone diagrams as well as “ways of writing” requirements that use certain wordings and formats. In order to check up on what impact ChatGPT would have on these practices, I asked the following question (I will call this Example 1 for the sake of referencing):

Please write BDD acceptance criteria in Given When Then format for the following requirement: A batch schedule in the system consists of the parameters: Type, Scheduled Time, Actual Trigger Time, Number of Retries and Status. The system has multiple types of users, however only Admin users can interact with the batch schedules. The Admin users will be able to set up batch schedules which are automatically run based on the Scheduled time. Once run, the batch schedule should record the Status and the Actual Trigger Time. If the schedule fails, then it should be automatically rerun for a number of times that is defined in Number of Retries.?        

And I got the following answer which we will analyse piece by piece.

Certainly! Here's the acceptance criteria for the given requirement in Given When Then format using Behavior-Driven Development (BDD):

Scenario 1:Admin sets up a batch schedule for automatic execution
Given an Admin user is logged into the system
When the Admin user sets up a batch schedule with the following parameters:
- Type: [Type value]
- Scheduled Time: [Scheduled Time value]
- Number of Retries: [Number of Retries value]
Then the batch schedule should be created with the provided parameters and saved in the system.

Scenario 2: Batch schedule is automatically executed based on Scheduled Time
Given a batch schedule is set up in the system with the following parameters:
- Type: [Type value]
- Scheduled Time: [Scheduled Time value]
When the Scheduled Time is reached
Then the batch schedule should be automatically executed, and the following should occur:
- The Status of the batch schedule should be set to "Running"
- The Actual Trigger Time should be recorded

Scenario 3: Batch schedule execution is successful
Given a batch schedule is set up and is in "Running" status
When the batch schedule execution is successfully completed
Then the Status of the batch schedule should be updated to "Success"

Scenario 4: Batch schedule execution fails
Given a batch schedule is set up and is in "Running" status
When the batch schedule execution fails
Then the Status of the batch schedule should be updated to "Failed" and the following should occur:
- The batch schedule should automatically be retried for the specified number of times (Number of Retries)
- After each retry, the Status should remain "Failed" until successful execution

Scenario 5: Batch schedule execution retries exhausted
Given a batch schedule has failed execution [Number of Retries] times
When the batch schedule execution continues to fail after all retries
Then the Status of the batch schedule should be updated to "Failed" and no further automatic retries should occur.

Please replace the placeholders [Type value], [Scheduled Time value], [Number of Retries value] with specific values relevant to your application and business logic. These acceptance criteria describe the behavior of the system according to the given requirement and outline the expected outcomes for different scenarios.        

What I found extremely interesting in the answer was how the AI was able to infer certain information which I would’ve expected any BA to make. If we take Scenario 1, we can see that while I have mentioned 5 properties to be present in a batch schedule, the scenario captures only setting up of 3. However if you read Scenario 2, then you get the full picture. While I have specified 5 parameters in the batch schedule, it only makes sense to record Actual Trigger Time and Status while a batch is running or after it is run. Why would you ever set up Actual Trigger Time and Status of a batch at the time of setting it up? It makes no sense. While it makes no sense for us, it is interesting to see that ChatGPT also seemingly has caught up with the same thinking pattern and proposing the same solution.

Some more interesting derivations can be seen in Scenarios 3, 4 and 5. While I have never mentioned the actual state transition of the batch, ChatGPT has assigned some meaningful statuses. If it is running, ChatGPT has derived that it should be in “Running” status. Once done it should go into “Success” status. If it fails, it will go into “Failed” status.

One of the most interesting derivations turned up in the 4th and 5th scenarios. While I have not explained in detail how a retry should happen, the AI model has been able to derive that if the execution fails, it should retry but also keep the status in “Failed” status without updating it until it becomes successful and that there are eventually two end scenarios - one where the batch is eventually successful and one where even after the number of retries it fails.

While my initial expectation was to see whether it can even at a very basic level replicate a particular format of requirements documentation (i.e. Business Driven Development and Acceptance Criteria), it looks like the AI went a few steps ahead and started inferring the requirements that were not explicitly specified by the user.

In the course of requirements gathering, we BAs follow various techniques to draw out the actual requirement from the client. This includes having interviews with the client, doing process flow diagrams, analysing them and then possibly rerunning the interviews and diagrammings again a few times to ensure that all hidden requirements were extracted. Processes like BDD try to make it simpler and clearer by adding acceptance criteria to spell out what will be considered as the criteria to accept a particular piece of functionality is delivered.?

In the above example, the client may have started with “We need a batch schedule mechanism where we can set up a time. If it fails, it should auto rerun a few times.” The BA’s job then would have been to ask more questions around the requirement and give a more detailed requirement to the development team by way of acceptance criteria. The BA would’ve gone into the details of the setting up a batch schedule, what statuses it can be in and how the rerunning etc should be done. There would likely be a small sequence or a flow diagram that would show how the batch would transit through various steps.?Finally if you are following a BDD model, you would've written the acceptance criteria covering all the scenarios.

What ChatGPT showed us is that not only can it derive those acceptance criteria out of a stated requirement, but it can come up with unspecified requirements to fill in the blanks. Isn’t this a very dire indication that AI is slowly replacing the BA work?

When I talked about this with my colleague who is heading the Software Engineering division, his immediate response was that I had actually done the BA’s work already when I fed the question to ChatGPT. I.e. it wasn’t at the level that a client would specify at a very high level. I had already analysed it and had defined some intermediate parts already. I had included which fields should be recorded at the setting up of it, which are to be updated when it is run etc. So I decided to test it again (we will call this Example 2):

Please write the BDD acceptance criteria for the following requirements: We should be able to set up batch schedules in the system to execute them at a predefined time. The schedule should auto rerun a number of times if it fails the first time. We need an audit record of when the batch was run.        

The answer this time contained 8 scenarios as opposed to the 5 before.

Feature: Batch Schedule Management

Scenario 1: Setting up Batch Schedule with Auto-Rerun
Given the user is logged in to the system??
When the user navigates to the batch schedule setup section??
And provides the necessary batch details and predefined execution time??
And specifies the number of auto-reruns??
Then the system should save the batch schedule successfully??
And confirm the successful creation of the batch schedule??
And ensure that the batch is set to execute at the predefined time??
And set up the auto-rerun mechanism as specified

Scenario 2: Batch Execution with Auto-Rerun
Given a batch schedule is set up with auto-rerun enabled??
When the predefined execution time is reached??
And the batch execution fails??
Then the system should automatically initiate a rerun of the batch??
And decrement the auto-rerun counter??

Scenario 3: Successful Batch Execution
Given a batch schedule is set up with auto-rerun enabled??
And the predefined execution time is reached??
And the batch execution succeeds??
Then the system should record the execution timestamp in the audit log??
And indicate the successful execution status in the batch schedule

Scenario 4: Exhausted Auto-Reruns
Given a batch schedule is set up with auto-rerun enabled??
And the predefined execution time is reached??
And the batch execution fails for all auto-reruns??
Then the system should record the final execution timestamp and failure status in the audit log??
And indicate the exhausted status of auto-reruns in the batch schedule

Scenario 5: Audit Record of Batch Execution
Given batch schedules exist in the system??
When the user navigates to the batch execution audit log??
Then the system should display a chronological list of batch executions??
And include the timestamp, batch details, execution status, and any relevant error messages

Scenario 6: Editing Batch Schedule
Given a batch schedule exists in the system??
When the user accesses the batch schedule for editing??
And makes necessary changes to the batch details, execution time, or auto-rerun settings??
Then the system should update the batch schedule accordingly??
And confirm the successful modification of the batch schedule

Scenario 7: Deleting Batch Schedule
Given a batch schedule exists in the system??
When the user selects the option to delete the batch schedule??
Then the system should prompt for confirmation??
And if confirmed, the system should remove the batch schedule and its associated settings??
And confirm the successful deletion of the batch schedule

Scenario 8: Batch Schedule Validation
Given the user is setting up a batch schedule??
When mandatory batch details, execution time, and auto-rerun settings are not provided??
Then the system should display appropriate validation messages??
And prevent the user from creating an incomplete batch schedule        

This time, given the broader phrasing of the requirement (probably - we would never quite know), the answers have also expanded to the include scenarios which cover setting up, editing and deleting a batch. It has also gone on to state that when setting up, there have to be certain validations that are performed. Clearly the non-specificity of the problem statement hasn't stopped ChatGPT from coming up with a comprehensive answer; it has inferred some "common sense" scenarios from a generic statement. If a BA was to write the requirements, these are the scenarios I would expect at the very least in the first round of reviews.

So with that, let me end this post with a cliffhanger. What does this really mean to BAs and the BA profession? Are we on the way to get replaced by AI? What should we be doing?

(But of course the discussion doesn’t end here. Up until my next post on this “story”, carry out the conversation in comments below ;)?)