The extensibility story

This is the story of one of the major steps towards creating a cloud-based ERP system: How we changed X++ from being a customizable language to an extensible language. This lifted the product I’m most passionate about into the modern age. I know many others share this passion and are curious about how this happened. I’m frequently asked questions about this, which inspired me to write the story. If you are curious, this story is for you.?The story is told from my point of view, an engineer in the SCM team.?For full visibility other key players should tell their story too.

To respect privacy, I’ve excluded names in this story, unless the individual already is a public Microsoft figure, for example a Blogger. Would you like your name included, just reach out to me.

What success can look like

It is January. 2019. We are in a packed conference room in Bellevue, WA. About 100 people gathered for the yearly ISV Summit. I’m on stage giving an update on Extensibility: What’s new, what have we accomplished since last, and announce break-out sessions later in the day. The usual deal. About 5 minutes into the talk a gentleman from the audience interrupted me. He wanted to thank me and my team for having solved the problem, and I’m greeted with a spontaneous applause. I’m happy my team is standing in the back of the room to hear this unexpected appraisal. I was unsure how to react, if I remember correctly, I completed the rest of my presentation quickly.??

Later in the day, we hosted two break-out sessions on extensibility: Bring your issues, and we will guide you.?No one showed up. Apparently, there were no more issues or significant concerns. We had made it. A 3-year journey was over. Standing alone in the meeting room, a sense of fulfillment and accomplishment rushed through me.

Where it started

Let’s rewind time to 2016. Dynamics AX7.0 was just released. It was a massive effort to bring the on-premises AX2012 R3 to a cloud hosted environment. But that is a story for another day. ?I’m having my very first week of DRI duty. It is a week of 24-7 support for all FnO customers. If something is wrong for any SCM customer, I need to solve it. Day or night.?It is easily the worst week of my life. Perhaps I’m just privileged. During the week I had several calls during the night. Stumbling out of bed, connected my laptop, and started to troubleshoot whatever the problem was. It was often a fruitless exercise. Sometimes it was a problem we already fixed in a later version – but the customer was stuck on a previous version for whatever reason. Sometimes it was a problem caused by a customization, or at least so it seemed.?In reality each customer had their own version of the system, with their own source code and implementation. Anything could be edited. As a supporter that was a nightmare – we didn’t even have access to the source code.?Despite lack of sleep, it was clear that improvements of the situation were needed.

Around the same time James Philips wrote an internal memo, introducing the “OneVersion” vision. “Every customer on the latest release, always” by April 2019. ??While I do love challenges, this one seemed outrageous. How is that even possible and still keep the rich ecosystem of solutions? We have a unicorn of a product. How can that be protected??

I can only imagine what the engineers thought when JFK gave his “Man on the moon before the end of the decade” speech. I think I went through similar motions for several months, two voices in my head having an argument. One saying: “This is so amazing” and one saying: “This will never happen”.

Luckily James and his leadership team recognized the challenges and freed up capacity to tackle this problem head on. I was included from the SCM side. We started by identifying the problems to solve.

My mind map

To get every customer on the latest solution, we must be able to upgrade them seamlessly and automatically. This meant no human intervention, which excluded merging of source code. To avoid merging, we had to prevent customers and partners from editing Microsoft source code, aka intrusive customization.??However, that is only half the problem. If each version is not backwards compatible, then customers cannot upgrade anyway, and we failed. For Microsoft, this would add significant constraints to what we could do going forward.

A landscape of the problem space unfolded and is captured in this mind map. It slowly expanded on the journey.

We started slowly to refactor code proactively, mostly around base enums where CAR (Customization Analysis Reports) told us there were many overlayerings. Meanwhile we prepared an official announcement of our plans to seal the application – first a soft seal and then a hard seal. Ultimately preventing customers, partners and ISVs from changing Microsoft X++ code.

The 2017 ISV Summit and Technical Conference

In January 2017 we shared our intentions externally for the first time. Most reacted with a can-do attitude understanding and appreciating the direction the product was moving. This became the main topic for ISV summit – dawn to dusk breakout sessions discussing implications and concerns. Parallel tracks and overflow rooms where required to accommodate the demand.

Lots of concerns were raised and we were busy taking notes. One statement that stuck with me was: “This is impossible, as we need custom inventory dimensions”. I must admit that at the time I shared the sentiment. Our na?ve answer to this was to add custom inventory dimensions on demand. Another concern was select statements in X++. They are compiled and not extensible – that would need a solution too.

The mountain ahead of us was taken shape, and the list of problems we had to solve kept growing.

At the following Technical Conference the official public announcement went out, and I started blogging about the situation, proving guidance and solutions to specific problems and describing new platform capabilities as they became available.?This made me the public face of this change, and I received mails from near and far from partners in the ecosystem. The change would make the system less flexible, and the passion about protecting the product and maintaining a flourishing ecosystem is one I share. In many aspects this product is a unicorn: Broad powerful functionality and extreme flexibility. These are rare in cloud solutions that are typically narrow and one-size-fits all. The feedback from partners were quite polar. Some being overly supportive, looking forward to the greener days with continuous updates. Some questioning the direction, fearing a world without a flexible ERP flagship product. ?In my team we valued each of these opportunities for engagements, and everyone got a reply. They enabled us to make decisions that would preserve the unicorn in the cloud.

Extensibility requests

The X++ code base needed significant rework, and we needed lock-step execution with partners to succeed.?To facilitate the communication Vanya Kashperuk set up a VSO instance to receive Extensibility requests from invited partners. This allowed us to learn where the Microsoft code was too rigid and didn’t allow partner solutions to integrate.

We started producing lists of what we refactored in each half yearly release. They were long.?At some point I sent them through a word cloud generator. Here are some of the results:

Chain of command

As an engineer caring deeply about code hygiene this project was dream coming true. Having a team chartered to refactor code is amazing. Typically, such activities need to be slipstreamed with business deliverables. Much to my despair I didn't like the refactoring we were doing.

At this point most of our extension points were provided via events. Solving an extension request involved introducing a delegate and raising the event. Introducing extra code, that wasn't tested. We spent hours debating and code reviewing this glue-code. It cluttered the code base, and for each we debated the values of only targeting the precise ask, or if being generic (at the risk of opening too much.) Knowing the pile of work ahead of us, we were going too slow and created a monster in the process.

One Thursday evening I used my lifeline and gave Peter Villadsen a call. We talked through the situation, and an idea started to form. Later same day Peter sent out the following mail. This is how Chain of Command started:

What started like an idea quickly became tangible. Peter and his team performed magic, and within a month a prototype was available. I’m fully confident that when I at some point look back at my career, then this invention will shine the brightest.?Considering that 60 years after object-oriented programming was invented, a new addition was introduced. Sure, it has some similarities with aspect-oriented programming – but also a lot of differences.?

In the following months this feature grew on all of us. It gracefully solved a series of problems we didn’t even consider originally, including how extensions can handle transactions, pass additional parameters downstream, and it merged nicely into the ExtensionOf classes that already existed and provided the ability for a stateful extension. All while not changing the API expose surface that we had to maintain for backwards compatibility. We knew we were on to something right. A term was coined for extending this way: "Wrapping". It belongs in the vocabulary right next to "overriding".

In SCM we had added our last event. Now extension requests were implemented using extract method refactoring – which resulted in Clean code. Many extension requests could now be rejected, as the capability came for free as protected and public methods supported chain of command automatically. Velocity picked up.

Inventory Dimensions

We were thrown a lifeline by an MVP partner. They proposed an ingenious approach to solving the inventory dimension problem. My team accepted the idea on the spot, and we started the long journey of refactoring the application to support the idea: Create 10 generic dimensions and allow them to be purposed at deployment time. Sharp focus was on execution, as we wanted to present the solution at the next ISV Summit. Over 800 methods got refactored. And the result was as good as we could have hoped for.?Previously, introducing a new inventory dimension for a partner would take 3-6 months.??At the ISV summit another MVP confirm doing the same in less than a 1 week with the new model, and without changing a single line of Microsoft’s X++ code. We even provided a sample implementation, introducing a new product dimension: “Flavor”.

Documentation

From the extensibility requests it became clear that we needed a better way of documenting the various solutions to the many different problems. So far, the only external facing documentation was on blogs. Internally we had a OneNote that was a gold mine. It didn’t really scale, we found ourselves spending lots of time giving similar replies as guidance to extensibility requests.?We needed official documentation.?This started another critical deliverable. Create a landing page documenting the extensibility knowledge for partners to extend our code as well as write extensible code themselves. Once it went online, we realized the vacuum in the market this filled. The number of daily page views was staggering. New contents got added on a weekly basis and today over 50 topics are covered – enough material for a book.

The 2018 ISV Summit

The atmosphere was changed, I believe chain of command did the trick. The generic inventory dimensions helped too. Many other application areas had been significantly redesigned to support extensibility, including price discounts and warehouse mobile flows.??

I left the summit with a feeling that we could succeed with this plan, and perhaps even on time. At this point we just needed to focus and keep up with the steady inflow of extensibility requests.

A few months later we released version 8.0, kicking off a new era for X++ customizations.

SysDa

One problem that still needed solving was the X++ select statements. Peter Villadsen provided the key to the solution. X++ select statements are compiled into a document object model. Suppose we exposed that to X++ - then we would have a rich API for creating select statement, and it would have full feature and performance parity. The data access framework, aka SysDA, was born, and a pile of parked extensibility requests could be solved.

X++ the most extensible language on the planet

A lot can be said about creating, owning, and maintaining a language. The cost is huge, for a small language like X++ investments are sparse. X++ will always trail behind mainstream languages like C# when it comes to tooling, performance, robustness, conciseness, and many other desirable aspects.?However, the freedom to innovate can be priceless, and in this short period of time X++ leaped forward. I will not hesitate to call X++ the most extensible language on the planet.

Getting to version 10.0

The ecosystem of ISV, partners and customer has always been a core strength of Dynamics for Finance and Operations. With the platform in place to support extensible solutions, we needed to enable partners and ISVs to migrate their solution.

Extensibility requests could now be logged via LCS. And we entered a race mode.?We wanted to enable as many solutions as possible to be overlayer free by April 2019 – the date from the memo.

We were tracking throughput, inflow, outflow and burndown. A race against time, each refactoring improving the code base and enabling a new scenario for customers.

To accelerate progress further, we invited a few ISVs with highly intrusive solutions to join us at the Microsoft campus in Lyngby and be part of our daily scrum teams for a period of about one month each. This allowed us to work closely together to find and implement the required refactoring to support their code bases.??Lots of learning and experiences were shared and new friendships were formed.

Don’t break anyone

While churning code, we were thinking about how to avoid breaking APIs and there-by solutions that had migrated.??Our first mantra was “Don’t break anything”.?The key was a compatibility checker. A small tool that would verify each pull request against a baseline, and if it detected breaking changes the merge would fail.

The compatibility checker tool was later released so partners and ISVs can provide the same guarantee to consumers depending on their assemblies.

Our analysis revealed that 90% of all methods were exposed, and thus couldn’t be changed under this model. This would severely hamper our ability to innovate. To counter act this, we introduced Internal and Protected internal. This enabled us to introduce new code while maintaining the freedom to refactor. At least until the first extensibility requests arrived. (Joris de Gruyter and team have recently been tightening up a few loose ends.)

Another analysis revealed that of all the exposed APIs less than 15% were referenced externally. In other words, we could change the majority without breaking anyone, if just we knew what was referenced externally.?A new idea was born.

We started including cross-reference information in the deployable packages. When a deployable package was uploaded to AppSource, a pipeline would aggregate the information, and feed it directly to CodeLens in Visual Studio, for each Microsoft engineer to see.

Here is an example. The method “futureValue()” has no known external (or internal) use, whereas “getCompany()” does.

This information enabled us to create a process for making changes to a subset of public and protected method. Each going through a governance approval process requiring proper justification and risk assessment.

Release Validation Program (RVP)

Being binary compatible is a good foundation. It ensures that binary references will resolve correctly on a live system when Microsoft assemblies are being updated as part of an upgrade cycle.

But why not aspire higher? Most changes are for a reason. Even fixing a product defect is about replacing one system behavior with another. With new releases being pushed out, businesses could have a logical dependency on the previous behavior.?To enable customers to detect such situations early we introduced the RVP program, where recorded tests will be executed on customer’s data and binaries, to detect impactful changes early on. ??I can recommend joining the program.

One version

This brings us to March 2019 and near the end of this story. Version 10.0 is ready for release. The release frequency will be increased from twice per year to 8 releases per year, we call them monthly updates.???

The Extensibility team is gathered in a restaurant in Bellevue. The food is excellent. The mood is high, and the company is even better. We are talking about the journey, for hours without end. We know that when we leave the restaurant, it will be the last time we are all together.

Today

As I’m writing this text late in 2021, we are having feature complete of version 10.0.25 this week.?We have released 25 versions publicly. 8 releases every year. All binary compatible. All allowing customers to upgrade seamlessly.?Each of our customers benefit from being on one of the latest releases always.??Sure, there have been a few misses over the years. Each a problem, but all something that we could solved timely, and each offering an opportunity for further learning and improvement. Now customers get the benefits of being on a proven and tested version, with all the latest product improvements.

Over 10,000 extensibility requests have been logged since we started. Thank you! We have answered each of them. The distribution across application areas is: SCM: 56%, GFM: 20%, SI, Retail and AXL <10 %. ??I’m humbled being part of the team that moved this mountain.

Supporting customers is much easier now. I’m indeed DRI today – no calls so far, and should my attention be needed, I know the customer will be running the same source code as all other customers. I’m prepared.

??

Michael Fruergaard Pontoppidan

December 2021

PS. Please share your story too.