101 Models of software

Here are 101 models of your code (please see attribution below). I have provided a condensed version of each model. Please see the original paper for very important additional details and insight.

1. Every line of code
2. Every branch on multi-branch lines
3. Every subpath through the program; e.g., every 10 line sequence in a 10,000 line program
4. Every path through the program from entry to exit
5. Every possible value in every logical operator
6. Trigger every assertion
7. Conditions when a loop is executed more than once
8. Every module, object, component, tool, subsystem – what about third party components
9. For heuristic decisions, which uses evolving rules or data, check rules periodically
10. Check boundaries for input, output and intermediate variables for ‘off-by-one’ errors
11. Every data item/variable/field
12. Constraints among variables; flight return date must be greater than flight start date
13. Every appearance of a variable; Enter x in each data screen and check value displayed in in all screens and reports
14. In object oriented code, send different types of (valid) data types and classes to objects.
15. Potential data conflicts; calendar application with identical appointments
16. Handling of every error state; put the program in error state and check for effects on stack, memory, keyboard input,…
17. Every complexity / maintainability metric against every module, object, subsystem, etc. – some may question these but some organizations find them an effective tool for highlighting code that needs further investigation and might need redesign
18. Conformity of every module, subsystem, etc. against every corporate coding standard; even if you don’t agree with these standards, companies use them to decide if a design is acceptable or not.
19. Table-driven code – external or internal table results in code branching to location specified; check every expression selects the correct table element; the program correctly jumps or calls through every table element; every address or pointer that is available to be loaded into these tables is valid; check the validity of every table that is loaded at a customer site.
20. Generate every type of interrupt in every way possible to trigger that interrupt.
21. Every interrupt at every task, module, object, or even every line.
22. Every anticipated or potential race.
23. Every time-slice setting, if you can control the grain of switching between tasks or processes.
24. Varied levels of background activity. In a multiprocessing system, tie up the processor with competing, irrelevant background tasks.
25. Each processor type and speed.
26. Every opportunity for file / record / field locking.
27. Every dependency on the locked (or unlocked) state of a file, record or field.
28. Every opportunity for contention for devices or resources.
29. Performance of every module / task / object – look for performance differences between test cycles
30. Free memory / available resources / available stack space at every line or on entry into and exit out of every module or object.
31. Execute every line (branch, etc.) under the debug version of the operating system. This shows illegal or problematic calls to the operating system.
32. Vary the location of every file – every component or file to a different directory, drive, or network location
33. Check the release disks for the presence of every file.
34. Every embedded string in the program. Use a utility to locate embedded strings. Then find a way to make the program display each string.

Operation of every feature/function/data handling operation under:

1. Every program preference setting
2. Every character set, code page setting, or country code setting.
3. The presence of every memory resident utility (inits, TSRs).
4. Each operating system version.
5. Each distinct level of multi-user operation.
6. Each network type and version.
7. Each level of available RAM.
8. Each type / setting of virtual memory management.
9. Compatibility with every previous version of the program.
10. Ability to read every type of data available in every readable input file format, including variations and subtypes
11. Write every type of data to every available output file format, including variations and subtypes
12. Every typeface supplied with the product..in all sizes and styles.
13. Every type of typeface compatible with the program.
14. Every piece of clip art in the product
15. Every sound / animation provided with the product….with all devices and drivers
16. Every supplied (or constructible) script which interacts with the product or external software or services
17. All commands available in a supplied communications protocol.
18. Recognized characteristics, e.g., speaker’s voice characteristics or more common fingerprint characteristics
19. Every type of keyboard and keyboard driver.
20. Every type of pointing device and driver at every resolution level and ballistic setting.
21. Every output feature with every sound card and associated drivers.
22. Every output feature with every type of printer and associated drivers at every resolution level.
23. Every output feature with every type of video card and associated drivers at every resolution level.
24. Every output feature with every type of terminal and associated protocols.
25. Every output feature with every type of video monitor and monitor-specific drivers at every resolution level.
26. Every color shade displayed or printed to every color output device (video card / monitor / printer / etc.)
27. and associated drivers at every resolution level. And check the conversion to grey scale or black and white.
28. Every color shade readable or scannable from each type of color input device at every resolution level.
29. Every possible feature interaction between video card type and resolution, pointing device type and resolution, printer type and resolution, and memory level.
30. Every type of CD-ROM drive, connected to every type of port (serial / parallel / SCSI) and associated drivers.
31. Every type of writable disk drive / port / associated driver. Don’t forget the fun you can have with removable drives or disks.
32. Compatibility with every type of disk compression software.Check error handling for every type of disk error, such as full disk.
33. Every voltage level from analog input devices.
34. Every voltage level to analog output devices.
35. Every type of modem and associated drivers.
36. Every FAX command (send and receive operations) for every type of FAX card under every protocol and driver.
37. Every type of connection of the computer to the telephone line (direct, via PBX, etc.; digital vs. analog connection and signaling); test every phone control command under every telephone control driver.
38. Tolerance of every type of telephone line noise and regional variation (including variations that are out of spec) in telephone signaling (intensity, frequency, timing, other characteristics of ring / busy / etc. tones).
39. Every variation in telephone dialing plans.
40. Every possible keyboard combination….The broader coverage measure is every possible keyboard combination at every error message and every data entry point.
41. Recovery from every potential type of equipment failure. Full coverage includes each type of equipment, each driver, and each error state
42. Function equivalence. For each mathematical function, check the output against a known good implementation of the function in a different program.
43. Accuracy of every graph, across the full range of graphable values. Include values that force shifts in the scale.
44. Accuracy of every report. Look at the correctness of every value, the formatting of every page, and the correctness of the selection of records used in each report.
45. Accuracy of every message.
46. Accuracy of every screen.
47. Accuracy of every word and illustration in the manual.
48. Accuracy of every fact or statement in every data file provided with the product.
49. Accuracy of every word and illustration in the on-line help.
50. Every jump, search term, or other means of navigation through the on-line help.
51. Check for every type of virus / worm that could ship with the program.
52. Every possible kind of security violation of the program, or of the system while using the program.
53. Check for copyright permissions for every statement, picture, sound clip, or other creation provided with the program.
54. Verification of the program against every program requirement and published specification.
55. Verification of the program against user scenarios. Use the program to do real tasks that are challenging and well-specified.
56. Verification against every regulation (IRS, SEC, FDA, etc.) that applies to the data or procedures of the program.

Usability tests of:

1. Every feature / function of the program.
2. Every part of the manual.
3. Every error message.
4. Every on-line help topic.
5. Every graph or report provided by the program.

Localizability / localization tests:

1. Every string. Check program’s ability to display and use this string if it is modified by changing the length, using high or low ASCII characters, different capitalization rules, etc.
2. Compatibility with text handling algorithms under other languages (sorting, spell checking, hyphenating, etc.)
3. Every date, number and measure in the program.
4. Hardware and drivers, operating system versions, and memory-resident programs that are popular in other countries.
5. Every input format, import format, output format, or export format that would be commonly used in programs that are popular in other countries.
6. Cross-cultural appraisal of the meaning and propriety of every string and graphic shipped with the program.

This list is from Cem Kaner‘s paper, ‘Software Negligence and Testing Coverage‘. Aside from the points made in the paper, this list is a great way to think about software.