What are the best methods for detecting defects in system components?

Process of elimination. Do we get what we want from A to Z? No, okay, then there is a problem. Split it in half, and test again. Do we get what we want from A to M? Yes? Okay, the problem is NOT between A & M STOP LOOKING THERE. Take N to Z, split that in half? Do we get what we want from N to R? No? Split that in half, do we get what we want between Q&R? No? Okay, check Q, it works? Okay, Q is NOT the problem, Check R, does not work? Ah, R is the problem, it is Ronnie's fault. (Taken from actual class lesson from when I learned the Process of Elimination; being one of the more outspoken class members, I was often made the example). But the process is simple: test half of the system at a time: that is the quickest way to find the problem

All modern system should pass through DevOps CI pipeline having a static analysis tool. More standard way of doing Static analysis of code is by using Static analysis tool like "SonarQube". SonarQube is popular tool which can detect code quality, security issues and OWASP Top 10. It also detect major and critical coding bugs. It supports multiple programming languages. There is another aspect is security scanning tool like "checkmarx", "findbug", "PMD". Which can detect security issues in the code. There is 3rd aspect of open source 3pp dependency checking and find security and license issues. we have tools like "FOSSA" and "Black Duck". Of-course manual review is there for both Code and system.

Static analysis is always the first step in any system development. It's usually the area where most issues are resolved; before any physical assembly takes place.

Static code analysis doesn't have to start after you write the code. One of the best time to start using it is during code creation. In compiled languages, you should invoke the compiler often during the code authoring phase. When doing so, make sure to turn up the "loudness" of error checking. For example, when writing C/C++ code and using GCC as the compiler, add "-Wall -Wextra -Werror" at the minimum, or better still, add "-Wconversion -Walloca -Walloc-zero -Wnonnull -Wvla". If you're feeling really good about your code, turn on "-Wpedantic"

Initiating static analysis in software development proves beneficial by using tools to examine source code without execution. This early scrutiny uncovers coding standard violations, security vulnerabilities, and potential defects, enhancing code quality, reliability, and security. By proactively identifying issues, static analysis contributes to improved maintainability, reduces the risk of bugs, and fosters an efficient development process.

En mi opinión una de las formas mas efectivas y que "nunca pasará de moda" para la detección de fallos es mediante la prueba dinámica, ya que solo mediante la ejecución "real" de nuestro código podemos detectar deficiencias, fallos, distintos escenarios, etc.

Dynamic testing starts with Unit testing, where you write code based test-cases to test your own code. For example Junit, TestNG, Jest, Pytest and XUnit are few widely used Unit testing frameworks. After that we have integration test, where we have tools like Robot Framework, Selenium, SoapUI and Postman. Both UI and backend API can be tested. Similar way, Mobile Apps can be tested using different OS and OS versions, we have tools like Appium, BrowserStack. Then we have tools like Jmeter for Load and stree testing. Now a days, we also Test infra with intentional failures called "chaos Monkey", example what if service is down or DB corrupted or VM is offline, how system behaves in such situations.

A maioria das falhas de seguran?a está na lógica e n?o na sintaxe. Nesse sentido, acho muito válido durante o processo de desenvolvimento, incluir checkpoints para olhar junto com um experiente e malicioso engenheiro de software quest?es relacionadas a furos de lógica. (Ter um olhar "n?o viciado" é algo impressionante para detectar estranhices.) E sem dúvida, ter uma boa cobertura de testes para trazer mais seguran?as com as mudan?as ao longo do tempo também é muito bom.

Dynamic testing is the backbone of any system development. It is usually the area where hardware and software interface is observed; along with such things as embedded systems communications, timing, integration, and concept viability.

Dynamic testing is the most important type of testing especially in an fast growing environment. DAST are probably one of my favorite types of tools to use because it's essentially a robo hacker. DAST will allow you to have confidence that your software is running with very minimal bugs which leads to less pocket pinching vulnerabilities. DAST should not be the only tool used in system design but should also incorporate manual regression testing, SAST, and code coverage such as: unit, integration, and end-to-end testing. Overall the most important takeaway is that companies do overcomplicate system design but rather take advantage of what is needed currently

When it comes to Fault Injection you have to assume that anything that can go wrong, will go wrong. This means anticipating not just common errors, but also rare and unusual scenarios. Think about how the system should behave under extreme conditions, like network failures, corrupted data inputs, or hardware malfunctions. Know the limits of your system's capabilities and keep breaking them constantly to test in worst case scenarios.

Chaos engineering is a technique to intentionally inject faults into the system to test it's resilience. One can use libraries like chaos monkey, which injects faults into the running application and tests the behavior of the application.

To be honest, there are very limited teams and companies who does Fault injection. Although the term "Chaos Monkey" is popular. Basic aspect: Suppose system is calling a external API in an interval. There could be situation where API will not be available. To test such kind of scenario, i usually put wrong URL during deployment configuration and see how system behaves. Advance aspect: what if entire service/component is down on Cloud. It could be a container or entire EC2 is down, can our system still responsive to execute traffic. These all should be tested in distributed system where 3 node cluster should be tested with minimum running capability of 2 node, where 1 node is expected to be down.

A very important part of development. Faults in functionality can be discovered in this phase. Maybe an overlooked flaw in code or hardware-to-software interface affecting redundancy can be discovered. try-catch code blocks can be used to isolate system code functions. Hardware interface code failures in real-time can be observed.

I've utilized fault injection to detect defects in system components by intentionally introducing faults or errors at different stages, including design, compile, run time, and post-deployment. This method proved beneficial in evaluating system robustness, resilience, and fault tolerance, uncovering hidden defects and enhancing overall reliability. However, challenges included careful planning to avoid unintended consequences and effective coordination with team members to address unforeseen issues. Despite these challenges, the benefits of using fault injection outweighed the encountered problems, providing valuable insights into the software's performance under diverse conditions.

Whoa, whoa, whoa. Code coverage doesn’t detect anything. Such mapping only helps to inform your test plan, regression strategy, etc. In other words, “If we tested <x> only, how much of the design has been validated?” Imagine an equivalent analogy: you decide the best route home by making sure your maps are updated. No, you still have to navigate! That’s true with the testing, and this very category demonstrates a lack of understanding.

Performing code coverage analysis is one way to get confidence in overall robustness. The better the coverage, the more you can trust your code to do what you intended. But, as with any tool, it is critical not to put all your proverbial eggs in the code coverage basket. Just because a line of code was executed does not mean that all of the corner conditions were properly checked. Consider a C/C++ line like "my_array[index] = 42;" Just because code coverage says the line was executed at least once, does not mean that execution of that line will always succeed. For example, if index is a huge number, the code will crash.

Code coverage is like a Diwali cleaning checklist for your house. It shows which rooms are cleaned (tested) and which aren't, guiding your cleaning (testing) plan. But it doesn't clean (test) for you. If someone ticks off rooms without cleaning, the list shows a clean house, but dirt remains. Similarly, in software testing, adding 'shallow tests' just to increase code coverage makes it seem like the code is well-tested, but like unchecked dirt, hidden issues in the code can still exist. This checklist helps in planning but doesn't replace thorough cleaning (testing).

Code coverage is basically how much unit test cases you wrote to test your source code. It usually measures in percentage, if there are 10 lines of code and your unit test cases is covering 9 line where last line is left-out, then code coverage is not 100%. I strongly feel unit test cases are good and should be written. but too much of test-cases slow down team to delivery value in short time. So in an average in between 65% to 75% code coverage is good enough and should be accepted. Code coverage report sometimes helps you to detect code debts which requires refactoring of code. Usually Unit test cases tool and SonarQube generates a report for percentage of code coverage.

I completely agree with Steves's perspective on this issue. If prior steps (static analysis, dynamic testing, and fault injection) are correctly followed, this step should not be necessary.

Mutation testing is not very common, but it helps in detecting the quality of unit test case tool and unit test cases in a way how much they are capable to detect changes (mutation). Small alteration / changes in code can be swapping variables, changing operator, changing if condition true/false part and re-run test-cases to see if how it behaves. We have Mutation testing tools like "pitest" and "Mutpy" which are used by developers. These tools generates a scope of Mutation detection in %.

I view mutation testing as a post-development phase where different codes and hardware can be injected into a functional design. System modifications can be observed for their viability. In our field, something new is always going to be available and the ability to adjust to the changing times is paramount.

There is no doubt that the aformentioned points are absolutely crucial and almost required to detect defects in system components. But there is one other point to consider which is potentially the most important one because it helps accomplishing all the others in a better way: know and own your code. Knowing intimately the system component's code gives a clear understanding about their weak points and limitations. It helps you to better target your tests and completes the confidence that your tests already give you. It also helps to better refactor them or upgrade them. It is a team collaborative task (and I shall say duty) to intimately and continuously know and own their components code.

Hoje em dia é bem comum que os nossos sistemas?converse com outros sistemas como gateway de pagamento, ERPs/CRMs, notifica??o, biometria e etc... Além disso, a infraestrutura também pode ter as suas dependências, como por exemplo um servidor de VPN, firewall / WAF.? Esse tipo de cenário é bem ruim pq dificulta a pauta de resiliência de software, afinal de contas, de nada adianta por exemplo ter um sistema redundante com dependências apontando para o mesmo lugar. A abordagem que quero trazer, por mais óbvia?que seja, é a de ter um mapa com todas as dependências?do sistema, de modo que você tenha uma vis?o clara sobre quais ir isolando até chegar onde de fato está o problema. Basicamente isolar o problema em um contexto menor.

Detecting defects in system components involves methods such as regular code reviews, automated testing, static and dynamic code analysis, fault injection, regression testing, user acceptance testing, continuous integration/deployment pipelines, monitoring, logging, and penetration testing for security. Utilizing a combination of these methods ensures a comprehensive approach to defect detection throughout the development lifecycle, from coding to deployment and beyond.