The Future of Software: Code Generation - Future Trends

Finally, the end of this thread!  We have beat code generation into a pulp with the exception of the whole premise of this series, to determine the future of software development.  So let's get to the future of code generation.

In the last post, I did my best to convince readers that code generation is what has driven software development well beyond anything else in our industry.  And then moved on to what I think is a fact, we have not made an advancement in code generation in over 15 years.  I backed that statement up with a pretty elaborate argument including an analysis of the most pervasive new languages invented in the last 10 years.  So what do we do about it?

I think there are 2 paths you can go down to pursue future advances:

1. Invent a radical departure from what has happened in the past.  The approach is new, unproven, and not based on previous improvements.  This is the revolutionary approach.
2. Analyze the past, determine what has and has not worked, and invent the next improvement based on what has happened in the past.  This is the evolutionary approach.

To be thorough, let's examine both approaches.

Revolutionary Approach to Code Generation

For path number 1, I was pretty certain several years ago that computers would write code themselves.  Writing code is so error-prone that I think it should be completely automated.  Humans, quite frankly, do not write good code, at least not very many humans write good code.  I have met some people in my career, very few and very extraordinary people, who write nearly perfect optimized code.  They don't quite understand why we all can't do what comes so naturally to them.  They secretly re-write our code in their spare time because they just can't stand to see imperfect code.  I've been humbled by them when I see my best effort as compared with their rewrite of my code.  When you find one of these people, do everything you can to work with them and keep them happy!  But unfortunately there are so few of them out there you may never even meet one in your whole career.  So for the rest of us mere mortals, let's get back to doing what computers do best, automation.

The assembler/compiler/object runtime evolutions each introduced significant amounts of automated code generation at each step.  A small population of extremely talented coders created these technologies and wrote extremely good code generators.  But we never got to the point where entire applications were written automatically.  The 4GLs we discussed last time were the closest step in that evolution and they never rose above object-oriented languages in popularity.  4GLs are still out there, they are still being used, but to my great disappointment it isn't what is driving a majority of the software development in the industry.

Given that the 4GL evolutionary approach to code generation is not working, what revolutionary approach would work?  I was convinced years ago that Artificial Intelligence (AI) techniques would write the code for us.  The computer would just learn what good code is and what bad code is, and it simply wouldn't generate bad code.  And with all the recent advancements in AI, I think that is becoming something within the realm of possibility today.  However, the quality of the code is not what makes an application great.  The absence of defects in code is also not what makes an application great.  An extremely popular and successful software vendor taught us that several years ago. A great software application meets the requirements of the user and does so with a cost and artistic quality so that users will want to use and perhaps even enjoy using the software.  Software does not need to work perfectly to be used!  It just has to work well enough and cost low enough that people deem using the software is better than any other alternative. 

Now when you apply this situation to automated application generation, how do you feed a set of application and artistry requirements into a code generator?  I'm not sure, but if I had to guess as to a revolutionary path, that would still be my guess today.  I really think someone is going to come up with a revolutionary approach to automated code generation involving some set of AI technology.  I've seen so much bad code out there that I really want the computer to take this off our hands.  If anyone out there is pursuing that type of path to software development, please let me know!

I'd love to hear your ideas as well for a revolutionary approach to code generation.  Fifteen years is far too long to get stuck in the mud here.

Evolutionary Approach to Code Generation

From the last article in the series we have a pretty good idea what worked and what did not in the evolution of code generation.  It all comes down to code generation with minimal or manageable side effects.  Each successful step had a order of magnitude productivity improvement (that is, the coder could write a good 10 times more code of the same quality after switching technologies).  The first step, moving to assembly language, had no side effects.  The remaining evolutions managed side effects largely through the development of tools.  Unfortunately 4GLs had as many side effects as they did productivity improvements, so in my opinion that is why they never became mainstream.

The 4GL approach generated code for 2 significant challenges that were not previously addressed with assemblers/compilers/runtimes:

1. User interface code
2. Data persistence code
3. Glue code to link user interface and persistence

Let's briefly take a look at these code generation technologies.

User Interface Code Generation

I'm not sure who came up with the original idea of user interface code generation, maybe it was Microsoft, maybe it was Xerox PARC, but I have to commend whoever it was.  I am addicted to user interface code generation technology and simply do not have enough patience to generate a user interface without it.  The most prevalent tool of its class when this first became popular was Microsoft Visual Basic.  I tried to write Microsoft Windows user interface code on my own, without a user interface IDE (Interactive Development Environment), and it was just too tedious for me.  But wow, you could create elaborate Microsoft Windows users interfaces in literally minutes using Visual Basic.  That technology evolved into Visual Studio and I think it is still by far the best user interface IDE in the world.  Visual Studio won Jolt awards every year I can remember.  Unfortunately in my opinion there are 3 issues that affect its use today:

1. Microsoft is losing the battle as a platform for running software.  Cloud systems generally run Linux and portable devices generally run Apple iOS or Google Android.  Microsoft acquired Xamarin and is aggressively adding its technology into Visual Studio to address cross platform mobile device development.  So there could be interesting things to come in the future here, even if Microsoft ultimately loses the platform battle.
2. It is quite difficult to write code that runs across different user interface technologies and runtimes.  Microsoft has the notion of universal applications but their definition of universal is not so universal.  In the past the very first selection you made when you created a Visual Studio project is what user interface technology you are writing the software for (Windows Form application, Web application, command line application).  I just want to write an application once and then have the IDE deploy the application to a web container to run a web-based user interface, create a Windows application, an Android app, or an iOS app.  Hey, I can dream, can't I?
3. The .NET runtime is not as stable or scalable as the Java runtime in my experience.  Most of the code generation capabilities for Visual Studio work only for .NET runtimes, although per #1 above that is changing.

I really love using Visual Studio, I have never been as productive writing applications in any other IDE.  The Visual Studio team is an amazing team and I wish them great success in their efforts to address the issues above.

Apple Xcode is the second best IDE I've used.  But you know how Apple works, you can only write iOS or Mac applications on Xcode.  And it is quite difficult to write an application that works across both iOS and MacOS.  In my opinion, Objective-C and Swift leave a lot to be desired compared to Java and in particular the Java runtime.  And there is also that issue where Xcode only runs on a Mac, and you want a pretty fast/expensive Mac too.  If you just need to write an iOS app, I'm very thankful for Xcode and Swift, it makes that job a whole lot easier.  But I would much prefer to use Visual Studio if I could.

Most recently I've been coding Android applications.  I've done a lot of work with Java over the years, wrote a good portion of a published book and some published articles, so it was great fun getting back into the language and applying it to mobile application development.  Unfortunately I think Android Studio is a big step down from Xcode and Visual Studio, I was very disappointed.  The IDE doesn't actually help you develop user interfaces, it just organizes and builds Java and XML files that you code yourself.  The code generation is basically wizard based with extremely limited round trip engineering capabilities.

When you look at all 3 major user interface IDEs that millions of developers around the world use on a regular basis, here are the issues left to be resolved:

A. Cross platform development.  I think the future of IDE technology is the ability to define an application once and generate the code automatically for use in a web-based user interface, a desktop user interface, or a mobile device.  Now this has actually been possible for quite some time if you run emulation technology.  For example you could write applications in JavaScript, HTML5, or Adobe Flash and run them on tons of devices.  But Apple won't allow that as a native application, emulators aren't allowed in their app store.  Emulators also have significant limitations on device access, feature set, and performance, making them quite difficult to use even with vendors that allow them. 

Xamarin is the closest technology I have found where you can design and implement your application in one technology and generate native applications across platforms.  I've tried it, the technology is incredibly complex and it exceeds my patience.  I've actually found it easier to recode the application in Xcode, Visual Studio, and Android Studio (yes, 3 different languages/platforms/APIs).  I feel for the folks at Xamarin and their competitors trying to make this job a whole lot easier.  The platform vendors (Apple/Google/Microsoft/...) often go out of their way to lock in developers to their particular platform.  And while that is a great business choice for them, it is very detrimental to the software industry and our customers.  Almost every consumer-oriented corporation today has to fund web-based application development, iOS application development, and Android application development of the exact same application.  They need to employ experts in each technology.  And it is extremely difficult to share the code between the technologies.  Maybe Xamarin/Microsoft will eventually get so good with their code generation technology that they can do this automatically.  Or maybe the platform vendors will change their strategy for the good of our industry.  But I have to think something will change here as it is just too hard and way too expensive to write the same application 3 times!

As a result of this tremendous expense and waste of time, I think the future of software for user interface technology will take these paths:

• The code generation technology will get so good it can handle the platform differences and still implement most of the applications people need to write.  Unfortunately this approach will never work for all the applications people need to write. Unless there is standardization there will be a need to code directly to a platform.  Today there is practically no standardization.
• The platform vendors will agree upon and implement a cross platform standard with proprietary extensions for platform specific features.  This can and has happened for many of our software technologies (SQL, Java runtime on pretty much every hardware platform you can think of).  This is entirely possible but it requires the cooperation of the vendors and I don't see that any time soon.  We need an industry leader in this area to step up and develop a cross platform user interface standard for web, mobile, and desktop applications!

Data Persistence Code Generation

I have watched and mentored many college graduate coders in my career and they tend to underestimate the complexity of 2 significant areas of coding:

1. Multi-threaded programming
2. Data persistence

For now multi-threaded programming is off-topic, but we do need to discuss data persistence in the context of the 4GL evoluationary approach.  It is incredibly difficult to implement code that correctly loads data from a persistence store, manipulates that data with the correct set of business rules, and stores that data correctly with ACID properties.  Most coders don't know what ACID properties are, and for some coders that do, they assume the database just takes care of it.  Unfortunately almost none of the NoSQL databases that the latest generation of coders love support ACID.  ACID is an exercise left to every single user of the database.  Even the good old traditional relational databases require you to implement some sort of optimistic or pessimistic locking control on top of their transactional support for ACID, which again is incredibly difficult to implement.  And then on top of all of this complexity, we have the issues of efficiency, scaling, and performance.  Any application with a moderate or higher number of concurrent users will require caching to avoid loading redundant data repeatedly.  Coders love to create caches, but as soon as you cache data you have the responsibility to know when the data in the cache is stale and to invalidate it.  Rarely do I find a coder that does that, they hardly ever write the required refresh logic when the data goes stale.  Finally, the more data you store, the slower the database gets so you need to be very familiar with the optimization (indexing, partitioning, ...) features of your particular database implementation to scale the implementation without unacceptable performance issues.  Databases don't have the same optimization features, even when they conform to relational and SQL standards.

To address this issue, many organizations released data persistence libraries and frameworks over the years.  I even wrote a very rudimentary framework myself.  Unfortunately I don't think a library/framework approach is good enough.  It isn't a code generation approach, it isn't the way the 4GLs were designed.  Libraries and frameworks, while incredibly valuable, require far too much coding and expertise to use properly.  The sheer amount of code leads to massive volumes of defects and maintenance burden.  The level of expertise required is just not prevalent in the industry and leads to even more defects. 

I think the future here is just like the future of user interfaces, we have the same 2 evolutionary paths:

• Code generation needs to cross data persistence technologies to correctly and efficiently map data to objects on any desired persistence platform.  We have a semblance of this technology in the 4GLs we discussed, but with far too many side effects and very limited support for data persistence platforms.  Most of the code generation approaches I were aware of died and were displaced by the greatly inferior framework/library approach.  And I understand why, it is that pesky ratio of productivity improvement to side effect.  Frameworks/libraries manage the side effects by requiring those that use them to code themselves out of situations the library can't handle.  For those setting out on the code generation path, it is the better path but only if you aggressively remove the side effects along the way.
• Vendors across data persistence technologies agree to and implement a standard like SQL making it a lot easier to construct the code generation technology above.  I plan to discuss NoSQL databases and a next generation design for persistence storage in subsequent posts, lots to discuss and improve upon here!

Glue Code Generation

Once we have a great code generator for user interfaces and a great code generator for the persistence layer, then it becomes a much easier task to glue them together.  Without the glue code, an application displays the wrong or inconsistent data, it stores or changes the wrong data in the persistence store, or it misses or applies business rules on the data inconsistently.  4GLs have proven for 20 years now that automated glue code generation is possible and efficient.  I don't see the problem here, the problem is with the user interface and persistence layer code generators.  However, because those problems have not been solved, it is up to us to write the code ourselves.  I find glue code quiet annoying and a big waste of time.  As it is so tedious, it is wide open for massive amounts of defects. 

I think the future is clear here, once the first 2 problems are solved, the automated generation of glue code will be an obvious and much easier problem to solve.

Summary

I believe programming language development has stalled and diverted from its roots as code generators.  To get back on track, instead of providing languages that optimize the manual writing of lines of code for a given platform or use, we need to get back into revolutionary or evolutionary approaches to generating significant amounts of theoretically perfect code.  All of the most prevalent recently released programming languages do not change the game, they just make it a bit easier to manually write code.

For a revolutionary approach to solving this problem, one conceivable path is to apply AI techniques to the problem of code generation.  I think this path has merit and hope to see promising research and products in this area.

For an evolutionary approach to solving this problem, I believe the 4GL approaches of the past were the correct path but were derailed because of lack of focus.  Most people worked on libraries and frameworks instead. Proprietary platforms lock developers into their platform for obvious commercial reasons, making it difficult to create a single code generation technology that crosses the platforms we need to run our applications.  The 4GL code generation approach however can be resumed if technologists:

1. Put pressure on the vendors to standardize
2. Always keep an eye on the side effects introduced by their code generators

Greatly looking forward to your feedback and ideas!

   Todd