Handling different versions of TwinCAT

This is a crosspost from my blog https://www.alltwincat.com/

Some time ago I wrote a post about developing code for different runtimes, and since then I've come to the realization that there is so much information missing in that blog post that I simply needed to write another one. What's even worse is that the information regarding development, deployment, and just general handling of several TwinCAT versions on the Beckhoff website is not that good. I've talked to numerous automation engineers regarding this topic, and everyone seems to have a different understanding about it. As I was working in a development project that had many PLCs (hundreds!) with different versions of TwinCAT, I finally had a good excuse to dive into the details of this. With this article, I'd like to share some of the facts and misconceptions.

I'm going to describe a (real world) use case. We were developing PLC software for a system running TwinCAT 3.1.4022.14, which at that moment in time was the latest version. We knew about the remote manager functionality (which had just been introduced in 4022), and we made sure to pin the project to 4022.14:

With every version of the TwinCAT XAE you get at least one compiler and a set of Beckhoff PLC libraries (such as the Tc2_Standard, Tc2_System, ...). If you install a newer version of the TwinCAT XAE on a developer machine, you get a newer compiler and newer libraries. Beckhoff describes the pinning as:

"To set the version with which a project is to be opened, you can use the option Pin Version on the General tab of the TwinCAT project node. If this option is activated, the TwinCAT 3 version set in the project will be used automatically when double-clicking on the project file.".

Sounded pretty much like it did exactly what we wanted - basically "freezing" the TwinCAT to a specific set of compiler and of Beckhoff libraries, or that's at least what we thought. Time passed, and since 4022.14 Beckhoff has released several new versions of TwinCAT. As we were working on many different projects simultaneously we needed to support different versions of TwinCAT. My main development machine for example looks like this when I select a TwinCAT version in the remote manager:

Opening the project on a machine with various versions of TwinCAT correctly showed that 4022.14 had been selected through the remote manager. We built the project (using the exact same versions of our internal libraries) and got this window that's well-known amongst TwinCAT developers:

This was bad. TwinCAT thought that we have changed the software, although we hadn't. Why was this? We had pinned the project, shouldn't that mean that we would get exactly the same binaries? We quickly concluded that it had to do with the Beckhoff libraries. Looking at the versions of the Beckhoff libraries being loaded when loading the project on a machine that only has 4022.14 installed we got (the blurred libraries are user/non Beckhoff libraries):

When we opened it on another machine that had some newer versions of TwinCAT it looked like this:

As can be seen, several of the libraries are a newer version (for example, Tc2_Utilities going from 3.3.26.0 to 3.3.38.0). When you are referencing a library in your project, you can either resolve it to a fixed version or to *, which just means the latest version that is available on the machine and not (as we had mistakenly assumed) the latest version of the library that is available for that particular version of TwinCAT XAE. Every single person I've ever worked with has made the same assumption, and when we noticed our mistake it was a moment of AHA! Pinning just sounded so... well... pinning! The reason we had not discovered this earlier was because we had used separate virtual machines for every project, and every virtual machine just had one version of TwinCAT installed. At this moment in time we started looking into working with a CI/CD pipeline utilizing a major build server for most of the TwinCAT projects, and that's the reason this thought of trying to use the remote manager instead was raised.

So, if the pinning didn't guarantee that a certain set of Beckhoff libraries is selected, what does the pinning actually do? It simply locks the available versions of the IEC 61131-3 compiler. If I open the mentioned project in a machine that has a vast amount of TwinCAT XAEs installed, and I go to the Properties of the PLC project and then to the Compile-tab it will look like this if the project is not pinned:

But if I instead pin the project to 4022.14 (or equivalently select 4022.14 in the remote manager before opening the project) I will get:

The change might be subtle, but if you look carefully you will notice that there are two compilers less to choose from, which are newer ones than 4022.14. What the pinning does is freezing the 61131-3 compiler version and allows you to select every compiler up to the version that you have pinned. That's all it does, and if you want to pin (or should I just call it freeze to differentiate it from the pinning?) the project you still need to manually select which version of each and every library that should be loaded which can be achieved by library placeholders. After carefully selecting the right version of each library we ended up with:

The keyword here is the "Resolved by placeholder redirection", that means that we explicitly referenced a specific version of a library.

So now we had two programs:

• One compiled with a machine that had only 4022.14 installed
• One compiled with a machine that had 4022.14 installed, and a few newer versions installed. Project was pinned to 4022.14 and all library placeholders were set to the versions in 4022.14

As we were still skeptical about this we wanted to do a binary diff of the actual compiled program (residing in C:\TwinCAT\3.1\Boot\Plc\) to be sure that the two programs were exactly identical, byte by byte. Despite our effort of pinning the project and locking every library to the same version that was shipped with 4022.14, the two compiled programs differed in size! The compiled .app-file was:

• 720 000 bytes on the 4022.14-only machine
• 722 324 bytes on the machine with many TwinCAT versions

Also the .crc-files didn't match on the two machines. When logging in to the PLC, we again got the question if we wanted to do online change, which meant the two programs were truly different. We got nervous and double-checked whether we actually compiled the same software and that we had cloned the correct version of the software from Git. Sure enough, it was exactly the same software in both machines. Now we had no other choice than to contact Beckhoff support. Beckhoff said that there was a bug in the remote manager functionality of 4022.14 (and maybe earlier?), and recommended us to upgrade. As I'm not a big fan of upgrading well-tested and working software (mostly for this reason), I wasn't too happy about it but an upgrade we eventually did. Doing the whole procedure all over again (but this time for 4022.29) still revealed a (smaller 720 984 vs. 721 156 bytes) difference in the compiled code, but now compiling the software and logging in to the PLC with it didn't bring up any "online change" window, which I guess means that the actual executed code is identical, so we considered this "good enough".

Some recommendations

At some point or another you will be getting close to the delivery of a system, and you will want to know exactly what you deliver. This can have many reasons other than the obvious one that you want to deliver what you tested. Maybe five years down the road there is some change to the system planned and software maintenance will have to be done. Or maybe the PLC might break? To guarantee that the software can be re-built there are some things that you should consider doing:

1. Pinning compiler

Pin the TwinCAT software (by the Pin Version option). This will guarantee that the correct compiler is used. If you do this and open the project on a machine that does not have the correct version installed, you will receive a warning which is beneficial.

2. Locking libraries

Lock the libraries using the placeholders to the version of the libraries that was shipped with the version of the TwinCAT XAE that you want to freeze it for. If you are freezing the libraries for the latest version you have installed on your machine, just select the latest version of the libraries. If it's for an older version, find out which version of the libraries are shipped with that version of TwinCAT by creating a virtual machine and just installing that version of the TwinCAT XAE. Also, don't forget to freeze your own libraries.

3. Lock visualizations and UML

If you are using the built-in Visualizations or the TF1910 TC3 UML, go to the Properties of your TwinCAT3 PLC project, click on Advanced and make sure that both the Autoupdate Visu Profile and the Autoupdate Uml Profile are not checked.

4. Supplements

Make a backup/copy of the correct version of all the supplements that you might use that had to be installed separate from the TwinCAT XAR. Examples of these are the TF6310 TCP/IP, TF6100 OPC-UA and TF6250 Modbus TCP supplements.

5. Operating system

Make a backup/copy of the complete operating system image that is provided from Beckhoff for the system that you want to maintain. Beckhoff provides these image files for some of their PLCs, although not all and you would need to go through the local Beckhoff support if it's not available on their FTP. This is less important as the software (generally speaking) can run on any other Beckhoff PLC as it's a PC-based solution.

Some thoughts

Nowadays I like to think of the Beckhoff libraries as just any other libraries you might have installed on your development computer. Installing more XAEs into your development environment is going to make the available compilers and versions of the libraries continuously grow. There is simply no direct link to the TwinCAT XAE version. Just as the TwinCAT XAE doesn't know which version of a library that you have developed belongs to a certain version of the XAE, it doesn't know it about the Beckhoff libraries either. There is no difference (in this aspect). Even if you contact the local Beckhoff support and ask if you can get a list showing which versions of the libraries are connected to which version of TwinCAT you won't get one. To get this information what I usually do is to create a new virtual machine, and then only install that particular version of the TwinCAT XAE. With only one version of the TwinCAT XAE installed, I can go to the library repository and see which versions of a library are installed for that particular XAE.

From what I've seen so far, in the world of automation even the stuff that is basic in the IT world as version control are not taken seriously. It simply seems that it's only necessary to make the machine work, and then possibly leave a laptop in a closed shelf next to the machine with a folder called "Machine Software Version 2020-05-22". If we as software developers don’t take this seriously, why then should we even care about what exact version of a library is used, or what exact version of a compiler was used to build the project? For me, having a software engineering background I think we can be just as good as in the IT world. It's not about being a control freak and knowing exactly each and every byte that goes into the PLC. It's about taking some responsibility for quality and making it easier for the next person that will have to maintain the software.

This raises a question of architecture for larger projects. If you have a project with a vast number of PLCs, possibly with different versions of TwinCAT - how do you want to maintain the build architecture? Should you have a build server that manages all of them through the remote manager and careful referencing of libraries, or should you have a separate virtual machine for every possible version of TwinCAT? This question obviously raises many other questions. Maybe you already have a test server that manages all software for all versions of TwinCAT and want to streamline the building into that pipe? Or maybe you want to do it the old fashioned way, possibly the way that the average automation engineer would solve it, by simply having a separate laptop/computer for each PLC and lock that computer into a valve. It's much fun to discuss these questions and looking into the best solution for a given project. I hope to come back to this topic in a future post.

I hope this gave you some new insight, and by sharing this knowledge I hope you won't have to step on the same mines as we did. How are you handling the development and build of various versions of TwinCAT? Have you developed any in-house tools to aid with this? Do you trust the remote manager, or do you always make a full copy of the complete development environment (in a virtual or physical machine) and store it somewhere? Please write a comment or two below!