Top 5 Automated Testing concerns in MS Dynamics 365

In this article we will share and elaborate on the Top 5 Automated Testing concerns in Microsoft Dynamics 365. These are all raised by those who have approached us regarding testing for their Microsoft Dynamics application. Since 2013, we have been advising and running test automation projects in Microsoft Dynamics AX2009, AX2012 and now on Dynamics 365 FO. We have conducted hundreds of small and several enormous test automation projects and this is to summarize different questions, statements, and our position and advice on these.

Here are the 5 most common concerns when planning automated testing in Dynamics 365

Together with elaboration on these concerns, we have listed our advice on what to consider during test strategy definition process and what should be challenged to find the best way to proceed. This may be used by anyone interested in the area, but it is no secret anymore that robust manual testing (even if combined with RSAT) consumes too much time and resources to keep up with the rapid pace of updates and releases of the Microsoft One Version. All that considered, we have focused this article on the only viable option on the market, Executive Automats, which has been designed and developed from the very beginning to address these specific problems.

In summary, by the end of the document, we will have explained how Executive Automats may support your Dynamics 365 automated testing project to finish it quicker (in a few days or even hours), better (reaching up to 80% automated test coverage), and cheaper (to create real savings and ROI)

Concern #1. Is automated testing in Dynamics 365 really necessary…?

It is sometimes a consequence of other concerns, but phrased this way, is illustrated with the following statements:

• “It is not really that much to test recurrently; we can handle it with manual testing”
• “application updates from Microsoft are not really a big risk – we are not afraid”
• “we may involve business people for UAT and manual testing – enough for our needs”
• “testing is critical only during the upgrade or implementation, there will be no big need for testing after go-live”

Cloud affecting automated testing

Following “cloud revolution”, more and more applications were moved to cloud to improve its delivery, time to value and further maintenance. Software vendors were given a fantastic opportunity to provide value faster and better with more frequent minor and major updates and releases. Together with its unquestioned value it has a very specific impact on all applications that are tailored to a customer needs. Customizations and modifications are based on standard solutions and when standard is being improved or updated it may affect existing modifications. Critical business processes may fail in the application. Regression testing of the application after any upgrade becomes essential.

Aside of application updates and releases provided by software vendors, customized applications are evolving all the time. Not only because of internal business needs, but also, to follow new regulatory affecting business requirements. Testing (all types from unit tests, integration to a User Acceptance Tests and regression) is a natural part of the software development lifecycle and regression testing takes a huge role in this process. Automated regression testing gives a chance to adopt application changes (business value) faster with a lower number of bugs. What may be interesting, following different researches, shortening application lifecycle is usually accessed more important than savings that are brought by automated testing (which is usually a case as well).

Advice: questioning importance of automated testing is usually a risk evaluation statement and probably a consequence of cost (efforts) vs effects (ROI) analysis performed by the customer. This is often the position of those who are maintaining a non-cloud, mature application, that are not expecting significant changes and perceiving automated testing scope very exhaustive thus of a limited value.

We suggest re-analyzing a following aspects (all supported with Executive Automats):

• If you would invest once, 30% more of your regular manual UAT time, to shorten each and every subsequent test run by 50% with higher efficiency and coverage, would you still come to the same conclusion and forego test automation?
• Consider life after go-live. Microsoft One Version requires multiple entire application updates each year and most partners recommend 4 at a minimum. These updates cannot be delayed more than a few months. Global UAT tests must be performed al least every few months.
• Involving business people may be a solution, but it requires management, coordination and causes complaints, all while lowering test quality. Again, would you consider a huge limitation in manual UAT testing if you could automate it easily, one time at the beginning? If it took you less than 10 minutes to create a re-usable automated test case, would that make test automation more valuable than conducting manual UAT tests with each and every code change?
• It may truly be the case you don’t need “deep” coverage test plans in your application. However, it is likely there are a few critical processes that, if broken, may seriously affect your business? It is imperative that you invest a little time (few hours) to create test runs every night to mitigate this business risk. This is one of the main features of Executive Automats that allows you to quickly and easily schedule repeatable test suites.

More about Microsoft Dynamics 365 One Version https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/fin-ops/get-started/one-version

Summary : a root cause for limited or no automated testing scope is usually a consequence of resource and budget estimations (see point 2 below) necessary for the project setup and/or general risk assessment. In most cases, a more detailed analysis reveals that the budget is overestimated (due to inappropriate tools and methods) and risk is underestimated (ie. Life after go-live is often underestimated considering the time and effort needed to maintain a constantly updated application.)

Concern #2. Dynamics 365 automated testing project is too expensive

Traditional test automation, with its history starting in early 2000’s, still has a “bad reputation” because of its complexity (see point 3), limited value (point 4) and inefficiency in ERP platforms under continuous development. The need to involve technical people and dedicated resources made test automation expensive and useful only where absolutely necessary. Those who tried test automation in the past have suffered on automation project costs when it was tried on the implementation phases and were suffering even more during the maintenance phase.

Together with the cloud revolution, new testing and process automation tools appeared. Low or now code UI based test automation tools, have been introduced, but the main questions concerning test automation are still valid:

• “it is a huge investment to start”
• “it costs even more when it goes to maintenance”
• “I can’t automate much of manual test, so manual tests are still essential”

Low or no-code test automation tools challenges

On one side, low or no-code tools (mainly based on Selenium frameworks) have been introduced to make it possible to design automated test cases based on “trigger and action” automation approach (Robotic Process Automation, RPA, type of automation). It allows to automate things, but the way a test script is created makes it very time consuming and not really useful for test automation project needs. Testing projects requires hundreds or thousands of test cases (average for an ERP) and thus, RPA type of solutions are difficult to accept due to budget challenges to start and maintain. These tools, supported with developers, allow you to create robust test cases and do integration testing but are heavily limited in scale use, missing necessary DevOps (TFS/VSTS) integrations and the possibility to document test plan executions (ie. test reports in regulated environments).

Recognizing this limitation, in the beginning of 2019, Microsoft introduced its Regression Suite Automation Tool (RSAT). With a different approach for faster test cases creation (recording “happy path” processes and not focusing on single actions) it gives a way faster solution to automate scripts and, more importantly, it gives native Azure DevOps integration that is usually essential for project needs (Pipelines and Test Plans). Addressing the speed aspect of things, it is still woefully lacking when it comes to different needs of a testing team. Without specific functions to support no-code validations, negative tests, try-catches, optional steps, loops, conditional steps and many others, it is not providing real life support for the test team to automate much from a current manual testing scope and therefore, the expected test automation coverage is low and test automation project remains expensive.

Another common challenge for existing automated testing solutions is the maintenance and reusability aspect of things. Dynamics 365 testing projects are specific, the application is live, under continuous maintenance, and with Microsoft updates, business processes are constantly affected with changes. Following these changes, test plans are modified and thus, part of each test cases and scenarios needs to be updated. Neither a “trigger and action” approach (RPA type) tools, nor RSAT are prepared for flexible script modifications and testing teams should be prepared to recreate test cases from scratch EACH and EVERY time a script modification is needed. We have found that this aspect is rarely considered during normal analysis and is one of the main causes for exceeding the planned test automation budget.

Advice: For limited budgets, focus on the main testing problems first. If there are a few heavy and critical integrations to automate, consider an RPA type of tool, but be prepared for difficult maintenance and limitations in scaling. If you need to focus on automated testing in Dynamics 365 FO (without any integrations) consider RSAT, but do not plan much for testing coverage as you will find out it has severe limitations and difficulties regarding script maintenance. Once you are able to accurately calculate the return on this investment (ROI), it will be easier to extend your scope and scale of test automation.

We suggest (suggested approach with Executive Automats):

• Start with your main testing needs (critical processes and integrations), it may be easy to start (few hours invested) and long term value (“low hanging fruits”)
• Calculate the return on the investment to prove it creates real time and cost savings
• Extend the testing scope to increase test automation coverage, create higher savings, speed up your application life-cycle, and limit risks of operations.
• Use ONE solution that will support easy start-up and help when you will be ready to extend your testing needs.

Summary:

There are a few requirements to meet to make Microsoft Dynamics 365 automated testing project viable and affordable. For a limited testing scope, it is crucial to be able to start quickly in order to create ROI in short term. For extensive test plans, it is important to have an all-in-one tool that may support all testing needs and limit the extra effort needed to manage multiple tools. Must have include:

• easy to start, create and modify scripts (minimum entry costs)
• provide different no-code test features required for validations (limiting costs of operations)
• allow integration tests with different platforms for test project scale us (limiting cost when scaling up)
• provide effective test execution documentation to manage test results (limiting costs of reporting)
• provide Azure DevOps Integration (limiting costs of management)

Concern #3. Testing different integrations requires different tools…

There are a few common expectations from functional automated testing in Dynamics 365.

Besides the automation challenges listed in the previous points, current automation tools are also inadequate with regards to low reusability and flexibility of automation scripts. While no-code tools (non-enterprise solutions) provide too few specific testing features, large RPA type tools are complex solutions dedicated only to complex testing needs and are limited in test case volumes.

While RPA tools (“trigger and action”, flow type of design) are good to support integration tests and it makes a business case to spend long hours creating and stabilizing single critical interface testing, it is not really a solution when it comes to easier test cases that may be supported with click-by-click type solutions.

Being aware of that, the following concerns are being expressed:

• “we need to use different tools to automate different needs”
• “integration tests are not supported easily, heavy coding required”
• “validations and specific testing needs are not supported with most solutions”

Integration testing are sometimes the essence of the thing

As mentioned in point 2 above, the need to use different solutions is making the project unviable from the cost perspective (potential savings from manual tests), but it is also making things more difficult from the management perspective.

Testing teams are usually dedicated to perform a wide extent of tests. Any need to use different tools or methods is complicating their tasks. Different tools requires specialization, creates bottlenecks, and information flows. In Dynamics 365 projects, teams are interested in managing all ERP testing (including integrations) from a single place (80% Azure DevOps), and are expecting a tool to be integrated into this common workspace. Different test requirements concerning Dynamics 365 are covered with manual or automated test cases. Managing the entire project from a single place allows to keep test completeness, quality, and reports tracking. Limiting the number of supporting tools is essential to simplify the area and improve project effectiveness.

What is more, when working with Dynamics 365, you are probably utilizing part or more of Microsoft ecosystem and solutions. Why not to use tools that are already there? Power Automate is not dedicated for automated testing in Dynamics 365 but may become a perfect match with your testing tools where it comes to a specific interfaces testing. You may support desktop applications, trigger actions and invoke test cases within more complex cross-platform testing.

Advice:  Regardless you are planning to switch to an automated framework or yet planning a test project from scratch, it is important to decide on the planned integration testing scope. Select solutions and methods that will allow you to maximize coverage while minimizing tools. If there are very specific needs of very high significance, we suggest to solve them individually (including hard coding methods), for any others, we suggest to focus on a single solution that will cover maximum requirements.

We suggest (all supported with Executive Automats):

• Unify tools where it is not a must to differentiate
• Create a single workspace for all testing needs to improve management, quality and information flow
• Do not reinvent a wheel if there are solutions. Use Azure DevOps to support your application development process (Pipelines) and use Power Automate when automating heavy interfaces testing with D365

Summary: Different testing needs require different approaches. There are already dedicated tools for different applications and methods available to support integration testing, but to properly manage test plan completeness, quality, and reporting we suggest to minimize the number of tools. We recommend using a “satellite” supporting solution only for very specific and individual test cases of a very high significance.

Concern #4. Why can’t I use automated functional test cases for performance tests…?

Performance testing is another type of test automation and depends who we are talking with, this is of higher or lesser importance in MS Dynamics 365. Of course, together with Cloud and scalable infrastructure, performance issues have become less common. It is easier now to react to system overloads, but it is still extremally difficult to create necessary benchmarks for cloud infrastructure to support your business as usual.

When talking about UI based test automation projects and mentioning robots mimicking user actions (virtual users) at the application interface, it comes very quickly to the mind why not to use these robots to run in parallel. Running in parallel and mass executing the same functional test scripts may provoke a “close to real” controlled system overload for its measurement (ie. when preparing for system stress at “Black Friday” traffic).

It should be possible to reuse automated tests scripts for performance tests

What is disappointing, most existing test automation tools (used for regression testing) are not supporting this idea (to support performance testing). There are many reasons either technical or financial. Tools are not prepared to multiply test executions, schedule scenarios, or just handle specific platform limitations to accept dozens of the same virtual user’s runs. On the other side, to make a real life performance test, we need to simulate different users with different application permissions and thus operating with different virtual users’ accounts. Any tools licensed by robot instance quickly become unaffordable.

Together with the above, a performance test project is getting more and more complicated.

• Creating scripts to execute performance test scenario is time consuming
• Executing scripts in order and scheduling mass executions requires a lot of work or is just not really possible
• Use of dozens of virtual users’ accounts requires its creation (a lot of administrative work), assignment to different scripts (lot of manual work), and, in the end, it requires a tool that supports a number of virtual users at a reasonable price

Importantly, this is all to create a single performance test scenario.

Advice: Look for a tool that allows you to create scripts easily and run multiple sessions in parallel without multiplying virtual uses licensing costs. It would be great if you could reuse functional tests prepared in the past (reflecting real use cases) to schedule them into possible performance test scenarios.

We suggest selecting a tool that supports (all supported with Executive Automats):

• parallel runs of one virtual users (system robots) or allow multiple virtual users at a reasonable price
• use of existing regression test cases to schedule them into performance tests scenarios

Summary: It should be a nice side effect of having a UI based functional (regression) test automation framework in place to reuse it for performance testing purposes (to mimic 100’s of real users actions) . The supporting tool should allow you to run multiple, simultaneous session runs with the same virtual account or should allow you to use multiple virtual user accounts for a reasonable price. Having that it is easier and faster to plan different performance tests scenarios in a limited time. Be aware that many tools on the market charge per robot (Virtual User) which is often not openly presented and can quickly balloon project costs.

Concern #5. Dynamics 365 automated testing scope or strategy is not defined – I’m not sure when and how to start…

This point, listed here as the last one, may be just as important as the preceding points. It is common that companies are convinced and determined to proceed with test automation, but are not really convinced about when and how to start. On the other hand, testing strategy and your application maturity may determine whether test automation is already necessary or not.

There are 4 main questions considering “when to start”:

• “Should I start automation during the D365 implementation phase or wait closer to the go-live phase?”
• “I’m already after go-live, is it still reasonable to start test automations?”
• “Should we automate in AX2012 if planning upgrade to D365?”
• “When to start if planning a series of rollouts?”

On the other hand, there are questions concerning “how to start” automated testing in Dynamics 365:

• “What is the best project setup, who to involve?”
• “Can I fully outsource that and get Test Automation as a Service?”
• “Should we “start small” from critical processes only and extend in time or is it better to plan long term already from the beginning?”
• “If having manual test plans in place should I recreate 1:1 into automation or search for modification?”

When and how to start test automation in Dynamics 365?

 Answering the above is definitely not trivial and there is no single answer, but there are few factors worth considering to estimate possible best fits:

1. Can we dedicate a QA team to create automated test cases?
2. Should I dedicate developers to support test automation script creation?
3. Do we have updated process documentation in place to understand what is needed for functional tests?
4. Can I involve business people (Subject Matter Experts, Functional Analysts , Process or Module owners etc) to help us design better test cases?
5. Are there any existing manual test plans that may be used?
6. Can we prioritize where the application is most affected for errors through updates or which processes are critical for our business continuity?
7. Do we know how many business processes are supported with our application?
8. Can I estimate how much I want to cover with automated testing in Dynamics 365?
9. Is there any deadline we need to have it done?
10. Do you need any specific test execution reporting in place?

Advice: Our advice depends on the combination of the answers to the above, but there are few common rules:

Plan long term but start small.

• This is important to know what is the long term goal (applications tested, test coverage, timeline), but it is even more important to start where it is most critical.
• Our suggested approach is to start from a high level review of application failure risks, determine points of weakness or high significance processes and start from there. With an iteration approach, go back to the plan and extend test plans for other areas, processes and additional test cases to cover more specific risks.

Test real business processes.

• It is common to create tens of test cases that will not touch properly on real business needs. Without proper business process documentation or without business people involvement, testers are usually not able to define what is to be tested, nor where are the real risks of the application failure
• Our suggested approach is to involve those who know business processes and empower them with a solution that they can use to record these processes (for its further recreation into test cases). This approach provides two huge values:
• test cases are based on a real business processes and use cases and tests cases requirements and defined better,
• what has been recorded by a process owner may become a template business process documentation for further training and onboarding purposes.

Do not involve the technical team if not really a must

• They are always in shortage. Whatever done, this will not stay forever in automated testing. Test automation requires maintenance and today’s chance to have it coded, may not be the case in the future.
• UI based functional tests with Record & Play approach are a few times faster than coded test scripts. Using a tool that is easy to use by non-technical people will allow you to keep scripts maintained in the long run

Do not think of automated tests the same as manual tests.

• Of course it should be doable to re-create manual test cases into automated test cases if only test results validation may be automated. When creating manual test plans it is quite common that these are tailored to testing team capacity. In test automation frameworks, you may think “wide” on possible test cases as there will be no real limitation on how many cases may be executed in a specified period of time. If there is a need to test 10.000 cases in 1 day – it is doable. Think of the test plan as something that you may execute often.
• Focus on business processes functional testing, not single buttons or controls. Test automatically mirroring how a person would perform the whole process.
• ERP platforms are enormous and “old-style” test plans to test single buttons, one by one, creating tens of thousands of tests are just not effective. If you know how business is operating you may focus on the processes that are testing lot of buttons, controls, functions and integrations at once.
• UATs are usually the latest test phases, but also the most important. If you involve those performing UATs to re-create manual tests into functional automated test cases you will be able to avoid manual UATs in the future. Again, this is necessary with consistent periodic updates required in Microsoft’s One Version philosophy.

The faster you start, the better.

• The moment your business processes are deployed, you should start with automation. Business processes are changing and during an ERP implementation phase these are not existing nor stable for many months or to the very end of the project. Nevertheless, whenever you can see it is valuable to check process regression you should consider automated testing especially when creation and maintenance of the test script is very fast and easy.
• Those on AX2012 and planning D365 upgrade, should consider test automation as well. Involving process owners and investigating existing business processes for automation gives a great chance to find out processes and modules that are not really used and not needed for upgrade (a common error in upgrades is to “upgrade all” including what is not really used). If you have process based test automation in AX2012 in place, it will be much easier to replicate similar test plans into the upgraded D365 application.
• If you are about to go-live, with an end to end implementation of an upgrade project, you should not wait until the go-live phase. In cloud applications, there is definitely a testing life after go-live and you should absolutely expect test needs with every application upgrade (ie. Microsoft One Version and obligatory updates). Having complete test automation frameworks available already on go-live, gives complete process documentation and great value when testing regressions in the stabilization months (while maintaining and fixing the live application and the most demanding support months right after go-live)
• Because of the above point, it makes no sense to delay automation if you are already live in D365. Every application update or modification requires regression tests. UATs require a lot of management and its quality decreases over time. Automating tests in some areas will help when planning application rollouts to different locations.

A need for automated testing in Dynamics 365 returns

• Those who have decided not to run a test automation project end up doing it later anyway. In the case of facing a critical error (that are quite common due to continuous obligatory updates) it becomes immediate and serious in priority, but automation becomes considered even without those threats. With time, managers can see that the number of tests increases over time and it is difficult to keep up. Application development lifecycle struggle through manual test projects while test automation is fast and affordable.
• We can see, sooner or later, 80% of companies decide to utilize automation to at least some extent. It is easy to create savings in the long run but it is also possible in the short term.

If you need to start fast and small it takes only few hours to get there…

• Together with the cloud, test automation delivery takes just a few minutes (SaaS is available). If you chose the right solution, you will be able to create your first test suites and have them running by the end of day 1. If you decide to start small, you can create test cases to test your critical business processes and start its execution in only a few hours. A few hours to cover a continuous risk or the critical business process failure in the application.
• Together with the cloud it is not even needed to do it by yourself. Test Automation as a Service (TAaaS) is already here and you may simply involve those who create tests cases based on business processes to run these cases against your application whenever needed – on demand.

SUMMARY AND D365 TEST AUTOMATION TOOL CHECKLIST:

When analyzing “pros and cons” of the automated testing project in Dynamics 365, defining strategy and tools to use, there are several factors to consider and evaluate.

Most of concerns relates to efforts vs effects evaluation, testing risk assessment, tools usability or test project setup. We have learned in time, that the above concerns, not being properly addressed, are stopping companies to change their manual testing “status quo”. What is important here, at the end, all of these concerns are about a tool and methods considered in automated testing. Tool selection determines efforts, time to value, usability and possible strategy and framework to adopt.

Executive Automats solution and testing methodology have been created, by design, to address these concerns and simplify the decision “if” and “how” execute automated testing in Dynamics 365.

With its design, technology and usability, it has become a real solution and best fit for those looking for Dynamics 365 test automation tool :

It helps to create test automation value very fast to minimize entry costs

• It is 100% codeless that dramatically improves usability, time to value and adoption
• Allows to start in few hours and get first test suites running in Day 1.
• It requires only 2-10 minutes to create a single test case (business process test)
• Provides testing features required by a real test automation team needs
• It may be provided flexibly (when needed) as SaaS or TAaaS (Test Automation as a Service) from Microsoft Partners

It allows to scale up and support increasing testing needs:

• Supports integration testing (web-based and more)
• Provides all-in-one features to manage a whole test automation project from one place
• Is integrated with Microsoft Azure DevOps

It gives more than automated testing

• Allows simultaneous virtual user runs to create performance testing scenarios
• Provides robust test documentation
• May be used to create business process documentation

May be incorporated at any time of the application lifecycle

• AX2009/2022 to D365 upgrades
• Implementation
• After go-live maintenance and keeping up with Microsoft One Version approach

It is proven by the market:

• Already a 100+ customers using it worldwide
• Is certified by Microsoft
• Incorporated by number of local and global Microsoft Partners.

Thank you for reading.

For more details visit ExecutiveAutomats.com

Executive Automats Team