Busting AD Myth Five in the new mini-series from NAG? - "It's hard to maintain code with AD embedded."
At NAG, we offer integrated Automatic Differentiation (AD) solutions. These solutions are designed to save time, energy, and money and accurately compute sensitivities in almost any application 10x - 6000x faster than alternative methods. However, when talking to some businesses, we do still hear the same ‘untruths’ or ‘myths’ related to AD. To put paid to these beliefs, we’re publishing a series of short articles that debunk these myths and explain the truth over the next few weeks!?
This week we look at….
Myth 5: “It’s hard to maintain code with AD embedded.”???
You’ve decided?you want?your C++ library or application to benefit from the advantages of?AD. So, let’s further assume you successfully integrated AD. At NAG, we understand things?change and new code is always being developed. This presents?you and your development team with another challenge: maintaining the AD in?your source code. Is maintaining AD code an expensive process that requires the whole development team to learn and understand AD in detail? Or is that a myth?? In this article, we look at the potential maintenance costs and pitfalls, depending on which approach to AD you take.
Writing derivative code by hand is not only difficult and error-prone, you also create two separate codebases: one is the primal code, i.e., the original model, and the other one is the derivative code, the adjoint. There are a few important rules?to remember when working in this way! First, whenever a change is made in the primal code, the derivative code needs to reflect that change. Every developer in your team needs to be aware of this crucial?fact and must be able to write the corresponding derivative code. If the primal and?derivative code run out of sync, the effects can be costly. Furthermore, it is important to be aware that a seemingly tiny change in the primal code may require global changes in the derivative code. Additionally, if, for example, you need Hessians as well as first-order?sensitivities, the required workload increases. In summary, maintaining a hand-written derivative code usually comes with huge costs and time delays.?
Assuming you’ve managed to successfully use a source transformation approach on your code, you end up having two separate codebases for the primal and the sensitivities (similar to writing it by hand). You are, though, in a much better position since you can rerun the source-to-source compiler whenever you make a change in your primal. Smartly designed, this can be part of your build system. However, experience shows that today’s source-to-source compilers are not as robust as you’d like them to be. This usually means that the primal code needs some massage to be digestible and the generated code also needs some manual postprocessing. These steps are not easily automatable. Furthermore, depending on the user-friendliness of the tool, cryptic compilation errors can be expected. In summary, this is a much better approach than writing AD code by hand, but it still?comes with many uncertainties and pitfalls.?
领英推荐
As described in?previous Myth Busting posts, integrating AD with operator overloading techniques using a robust tool like dco/c++ is the easiest approach. This holds true for the maintenance of the code. The main benefit is that you have a single code base. Every change in the primal code directly results in the corresponding change for the derivative code. When using templates in C++, all incarnations of your problem, primal, first-, and higher-order derivative codes coincide in the same source. Although compiler messages are likely to be more complex in case of an error, it is standard C++ and developers will be familiar with the types of errors reported. In summary, the approach with the lowest maintenance costs is operator overloading.?
Maintaining hand-written or source-to-source compiled derivative code is by no means impossible. However, if you rely on the additional performance you might achieve through these approaches, you should?try to make the generation and build process as robust as possible. Targeted testing and smart coupling with overloading tools can help with this but there will be considerably more effort involved.?
NAG’s AD toolset has been developed and enhanced over the last 12 years and it builds upon a further 10 years of AD R&D experience. We know that details matter. Myths are narratives which might sound like truths but by talking through these in some detail and sharing our experiences we hope to help businesses navigate these issues. Results matter, myths should not. ?
Take a look at our other posts on Myths in AD, including memory use and rewriting libraries!???
Myth 1: I’ll have to re-write my libraries.
Myth 2: I’ll run out of memory.
Myth 3: It won’t work on my code.?
Myth 4: AD will infect all my other libraries.?
Myth 6: AAD will destroy parallelism.