What are the best practices for system testing and verification?

Requirements are the base of testing, without them there is nothing to test against. The ultimate way I know of, (at least right now) is to have proper requirements review process. It is imperative that the testing competence (team in charge of testing) reviews and approves the requirements from a testability point of view. Not only making sure a requirement is testable by itself but that is also testable by the existing tools available, or the available time. Only then can the requirements be really approved. A good review process will help in at least 2 ways, making sure the requirements are indeed good but also improve the capability of the system competence to learn how to write better requirements through the review feedback.

from my experience, I've also seen complex systems where the requirements were well defined but in actuality not verifiable - in my case signals collection restricted the TB. so, when designing a complex system it the verification restrictions should be taken into place otherwise you can expect bugs, and even more, bugs that might be harder to fix. also, documentation culture should support that as well. architectural, micro-architectural and val plans should all be revised in each project so we can aways take a look at only 1 document at a time and fully understand all requirements and implementation.

As per my experience, beside functional testing of any application with proper test plan and test cases, we should focus on the the load & stress testing exclusively focusing to meet target TPS before production movement. And this to be performed on the exact environment on which application will be deployed. For the case of replacing an existing system, this testing to be done specially as just after deployment it will have to serve already existing users with their peak expectation for a upgraded system. Also for each system, the resource specs to be calculated precisely during the testing period so that resources can be enhanced specifically on the basis of request type system will have to handle in future instead of trial on live.

A loosely coupled and polymorphic System is very testable, with a scope of extending and mocking the logical components. Avoiding the use of primitives and using business-specific wrappers to encapsulate primitives, makes the system clean and test-friendly.

The process of building requirements should start with the stakeholders. That means starting with a high-level communications document (for example, a User Requirements Specification) that bridges the users, design, QA, implementation and validation/testing teams to make sure everyone is aligned on the basic intentions of the system before jumping into functional or detailed design considerations. How high level? At this stage, you should have a specification that covers all the user, regulatory and day-to-day QA needs while being broad enough to be satisfied by multiple possible implementations. These should then translate into test cases that are easily verifiable by each of those stakeholders at their interface.

totally agree, sometimes we tend to continue updating an existing code that is not always implemented in the best way possible and we don't stop to think about the validation strategy.

In my experience people tend to take a shotgun approach to testing. They write a lot of stuff and it seems good at first but over time it becomes a nightmare to maintain and changes to any functionality break a lot of tests. I think this point is good but doesn’t acknowledge the importance of doing the “right” testing and not going with a shotgun approach as mentioned. Make sure you choose the right things to test and then decide on the testing methodology

I agree, To choose the right testing and verification methods I will need: To understand System Characteristics and Requirements Identify and Classify Testing Types Needed like: Functional Testing, Non-functional Testing, Structural Testing (White Box), and est. Consider the System’s Environment and Constraints Select Appropriate Testing Techniques: Static Testing Techniques, Dynamic Testing Techniques Continuous Review and Adaptation

Choosing appropriate testing and verification methods is akin to tailoring a suit – it needs to fit perfectly. By customizing approaches based on system complexity and risks, teams can efficiently cover all aspects, from functionality to security. Balancing automation for repetitive tasks with the nuanced insights of manual testing ensures a holistic evaluation. Incorporating user-centric methods guarantees that the system not only functions flawlessly but is also intuitive and user-friendly. Additionally, continuous monitoring and feedback mechanisms solidify the process, allowing teams to adapt swiftly and maintain the system’s quality in the ever-changing landscape of software development.

I think one of the critical aspects to maintaining system integrity, especially for large scale distributed systems is to have as many automated tests as possible. This is something which is difficult to convince to both business as well as developers, but if we allocate time for automated tests right at the start of the project and ensure that this is one of the critical non-functional requirement to meet, then these can do wonders for the integrity of your systems in the long run. Especially, if you run these automated tests as part of your CI/CD pipelines and have gated deployment requirements, you will automatically raise the resilience of your system.

i think that we should keep our tests generic as possible and thus making it easier to adjust them to several use-cases and the changing requirements

Before drafting the test cases and executing them, the tester should atleast know the basic functionality of the product and its purpose of use for generating the better outcome.

i would recommend on implementing coverage as a good way of doing so one can use code coverage and also functional coverage..

Recording and analyzing test results and feedback serve as a crucial feedback loop, offering invaluable insights into a system's strengths and weaknesses. By maintaining meticulous records using standardized formats and tools, teams create a robust foundation for analysis. This analytical process, when objective and systematic, not only identifies defects but also unveils patterns, enabling teams to make informed decisions. Furthermore, structured evaluation allows for targeted improvements, ensuring that future iterations of the system benefit from a continuous learning process. In essence, this practice transforms raw data into actionable intelligence, empowering teams to refine the system iteratively

completely agreed, specially when creating a verification plan it is important to have as many stakeholders engaged as passible so that they can contribute to creating the best plan

To add on. The verification lead should also continuously engage with the team members for getting the best results and clarifying their queries ASAP.

this is extremely important, and to some extent it is done mostly by responsible enough developers when adding a new group member one should bare in mind this, since it is not always measurable/trackable

In software development: Clean-up and Review Test Data Clean-Up: Clean up the test environment and test information post-testing. Retrospective: After the completion of testing, hold a Retrospective to talk about what went well and what might be made strides for future ventures

Design of experiment (DOE). Why do you need to test? how are you going to test? What are you going to measure? what are the likely results? the limitations of testing techniques, sources of errors, mitigation of errors, accuracy and repeatability. Development of clear qualification test requirements, development of clear, qualification test procedures/plan. All of these aid and abet confidence in the operability of the product.

The Verification Engineering teams I work with focus on medical devices and perform a full suite of design verification efforts. Organizations should apply V&V through the entire dev process. Good practice is to organize around dedicated SW, systems, and verification teams. The approach should focus on evaluation and testing of requirements, performance, reliability, and worst-case conditions. Testing plans should focus on these areas. 1. Prototype feasibility and use demonstration 2. Integration 3. SW system 4. System (product) - HW, compliance, performance 5. Design Validation - acceptance, clinical, user needs met 6. Selecting technologies/good documentation 7. Perform dry runs and monitor results with issue tracking and remediation

Testing is most useful if it helps to find bugs/defects as early as possible. Especially in regulated industries system verification too often is something that happens in the very end of the project with the purpose just to make sure that every system requirement has been verified. However, testing should happen all the time. During development one could dry-run the system tests to capture issues early. Even better, one should perform unscripted exploratory testing. It can be done in connection with the dry-runs or execution of the actual test cases.

pre-testing or dry run testing can help determine if test methods are accomplishing their stated goals, and if they are truly verifying the underlying requirements.

Review derived requirements at every layer of system design: with users, designers, developers , implementers and testers. Ensure that derived requirements are an intelligent interpretation and add value to the original user requirement.