To Generate or not to generate code using Generative AI (or GenAI in short)

As a 1st step I asked this question to ChatGPT , considering trends and long term it recommended to use it. In principle I believe that code generation using GenAI models will be the norm in future. Few companies have already adopted.

Questions that arise during and after evaluation

Will it help in productivity? How much? How do I measure? Companies have battled with productivity KPIs in the past and continue to do so.

I see plenty of articles which say GenAI accelerates productivity by X%. Without context it can be misleading. A project has typical phases as shown below

|Requirements|Feasibility|Architecture|Design|Implementation|Verification|Release|

To map, Implementation is coding followed by Verification. Teams which adopt Test Driven Development (TDD) go parallel to coding phase

In summary AI can be applied to all these phases and GenAI suits for Implementation and Verification phases.

A developer doesn't end up writing code, there is review, rework, debugging, refactor, security countermeasures, optimization and bug fixes. Not all team members have same skills and competencies. This would mean each developer and test engineer will be at different levels and speed.

When applying GenAI the following questions need to be answered but not limited to

1) License costs with features offered for variants

2) Contract and Service agreement with terms and conditions

3) Security risks involved as there will be company SW IPs involved

4) Ownership of generated code

5) Where does the data/code reside

Assuming you get answers for above, lets see where & how GenAI for code can be applied and used.

1) Engineers will go through a learning curve for GenAI. It requires prompt engineering skills. Developers with domain knowledge and open mind can learn basics in few weeks

2) Generated code needs to be reviewed, prompts have to change. Context has to be given to GenAI

3) Integration into current process/workflow for software development

4) Programming language and environment matter a lot as GenAI model may not have got trained/test data/code. There could be limitations.

Recently, I have explored Github Copilot following capabilities for code generation

1) Context aware code generation

2) Unit tests for generated code

3) Code boiler plates + comments + documentation

4) Bug fixing

5) Refactoring

6) Optimization

7) Security counter measures

and for Verification

1) Identifying test cases

2) Test specification

3) Test implementation

However it still requires in certain formats and not fully matured. I expect in coming months we will see a drastic change.

You can apply Balanced Score Card to derive metrics. Remember that project has multiple phases and we are talking about 2 phases in a project.

My thought would be that a team estimates as usual in a project. Introduce GenAI to team and ask them to use it for development. Check the end results. This would be more realistic measure.

I have collected data with few teams but would refrain from posting here and it will vary based on work/role/task.