Code Synthesis
GPT-4 is already the fourth generation of OpenAI's code generation capable AI system (after GPT-3, Codex, and GPT-3.5). For the first time, it needs to face serious, albeit not-fully developed, competition from PaLM-powered Google Bard.
It can generate code in a dozen or so programming languages (list below). In most cases, the code can be run with no or minimal modifications, provided that you follow some rules and are aware of the some limitations.
Table 1 Key Programming Languages Supported by GPT-4
?The code generation or code synthesis procedure employed by Large Language Models (LLMs) like GPT-4 is quite similar to natural language generation, which initially attracted the most attention. During training, the model "browses" through a gigantic amount of text, which can include source code, and tries to recognize the relationships between different parts of the text, as well as the relationships between source code and their functionality contained in documentation and comments.
When prompted for source code creation, the model first tries to "understand" what functionality the user is requesting, match this abstraction of functionality against learned code functionality, and finally put together the pieces of known code into the most likely solution for the question.
Here we approach the first problems this procedure must overcome. The level of detail in the description of the desired code functionality depends on the problem's complexity and uniqueness. Since models have a shallow understanding of code, uncommon tasks require more explanation. In my experiments, LLM models (a bit of tautology) had trouble generating correct code for some standard financial mathematics calculations that went beyond the scope of standard formulas – e.g. corporate bond pricing between coupon payments. On the other hand, GPT-4 suggested a valid simplification of a TensorFlow code using ragged tensors. This differences results from limited industry-specific coverage in the datasets used for training the LLM models (list below).
Table 2 Code Training Datasets
1. GitHub: GitHub hosts millions of repositories with code in various programming languages, which provides a rich and diverse set of examples for me to learn from. It helps me understand how different languages, libraries, and frameworks are used in real-world projects, as well as the collaboration and development practices employed by developers.
2.?Stack Overflow: As a Q&A platform, Stack Overflow offers numerous code snippets and solutions to common programming problems. It helps me learn how to solve specific coding challenges, understand common issues and pitfalls, and become familiar with the conventions and best practices for different programming languages.
3.?Programming documentation: Official documentation and tutorials are vital for learning the correct usage of programming languages, libraries, and frameworks. They help me grasp the syntax, semantics, and standard practices associated with various programming technologies, ensuring that the code I generate aligns with recommended guidelines.
4.?Online code repositories and databases: Additional code repositories and databases, like GitLab, Bitbucket, and SourceForge, further enrich my training data with diverse code examples. These sources expose me to various programming styles and problem-solving approaches, allowing me to generate code that can be more easily adapted to a specific project or context.
5.?Code from open-source projects: Open-source projects are valuable for understanding real-world applications of programming languages, libraries, and frameworks. They showcase best practices, design patterns, and efficient solutions to common problems, which can be instrumental in generating high-quality code.
While LLM models can review code for syntax errors and simple semantic errors, more complex ones are currently beyond their abilities. The inability to execute the code is another difficulty. The latter problem stems from both security concerns and the challenge of "teaching" a model to create appropriate program feasibility tests. However, it is worth mentioning that some experimental third-party tools allowing model local code execution have been introduced recently – for example, Auto-GPT.
Next, there is a limit to how much information a model can process at once. The context window of the standard version of GPT-4 is around 8,000 tokens (the "extended" version currently available through the API only has a context window of around 32,000 tokens). This translates into 150-220 lines of code. However, it is further reduced by the need to share the context window between the input prompt and the output code.
All this means that models like GPT-4, in the context of code generation, are best used for creating relatively small, well-defined, and preferably common programming tasks – like defining classes, generating functions, or their parts. However, there is a pretty long list of code development supporting tasks GPT-4 can be helpful with (list below).
Table 3 GPT-4 code assistance beyond code synthesis (according to GPT-4 itself)
1.?Interactive help desk and reference
???-??Provide real-time assistance for programming queries
???-??Offer guidance on best practices and code optimization techniques
???-??Help users understand language-specific syntax and concepts
2.?Code review for error identification and source improvements
???-??Detect and suggest fixes for syntax errors and typos
???-??Identify logical errors and potential bugs
???-??Recommend refactoring opportunities for better readability and maintainability
3.?Code analysis for code functionality understanding
???-??Explain the purpose and functionality of specific code segments
领英推荐
???-??Trace data flow and dependencies within the code
???-??Visualize code structure, hierarchy, and relationships
4.?Creation of unit tests / function test cases
???-??Automatically generate test cases for individual functions or modules
???-??Suggest edge cases and various input scenarios for more robust testing
???-??Help integrate tests into existing test frameworks
5.?Code translation between programming languages
???-??Convert code from one language to another while preserving functionality
???-??Suggest idiomatic ways to write code in the target language
???-??Help users transition between languages or update legacy codebases
6.?Code documentation generation
???-??Automatically create documentation for functions, classes, and modules
???-??Suggest improvements in existing documentation to enhance clarity and completeness
???-??Generate examples and usage guides to improve understanding
Additional suggestions:
7.?Assist in debugging
???-??Help identify the root cause of issues during runtime
???-??Offer suggestions for debugging strategies and tools
???-??Provide guidance on interpreting error messages and exception handling
8.?Code completion and suggestion
???-??Predict and suggest code snippets based on context and user intent
???-??Assist in faster coding by offering relevant suggestions and auto-completion
9.?Design pattern and architectural guidance???
???-??Suggest appropriate design patterns and architecture styles based on project requirements
???-??Help users implement design patterns and best practices effectively
To conclude, LLM models like GPT-4, or perhaps in the not-too-distant future Google Bard, cannot replace developers yet. You may also check the GPT-4 Codeforce rating (percentile below 5th) presented in GPT-4 technical paper. ?The LLM models may, however, assist developers in their work and help non-developers create Excel macros or simple VBA scripts.