Orthogonal array testing
Orthogonal array testing is a black box testing technique that is a systematic, statistical way of software testing.[1][2] It is used when the number of inputs to the system is relatively small, but too large to allow for exhaustive testing of every possible input to the systems.[1] It is particularly effective in finding errors associated with faulty logic within computer software systems.[1] Orthogonal arrays can be applied in user interface testing, system testing, regression testing, configuration testing and performance testing. The permutations of factor levels comprising a single treatment are so chosen that their responses are uncorrelated and therefore each treatment gives a unique piece of information. The net effects of organizing the experiment in such treatments is that the same piece of information is gathered in the minimum number of experiments.
Background
Orthogonal Vector
Orthogonal vectors exhibit orthogonality. Orthogonal vectors exhibit the following properties:
- Each of the vectors conveys information different from that of any other vector in the sequence, i.e., each vector conveys unique information therefore avoiding redundancy.
- On a linear addition, the signals may be separated easily.
- Each of the vectors is statistically independent of the others, i.e., the correlation between them is nil.
- When linearly added, the resultant is the arithmetic sum of the individual components.
Technique
Consider a system which has 3 parameters and each of them has 3 values. To test all the possible combinations of these parameters (i.e. exhaustive testing) we will need a set of 3^3 = 27 test cases. But instead of testing the system for each combination of parameters, we can use an orthogonal array to select only a subset of these combinations. Using orthogonal array testing, we can maximize the test coverage while minimizing the number of test cases to consider.[3] We here assume that the pair that maximizes interaction between the parameters will have more defects and that the technique works.[3]
| Test case ↓ | Parameter 1 | Parameter 2 | Parameter 3 |
|---|---|---|---|
| 1 | 1 | 1 | 3 |
| 2 | 1 | 2 | 2 |
| 3 | 1 | 3 | 1 |
| 4 | 2 | 1 | 2 |
| 5 | 2 | 2 | 1 |
| 6 | 2 | 3 | 3 |
| 7 | 3 | 1 | 1 |
| 8 | 3 | 2 | 3 |
| 9 | 3 | 3 | 2 |
Given that assumption, the table shows the set of nine combination of parameters which are sufficient to catch the fault, considering the interaction of the input parameters, which is very effective and economical. The array is orthogonal, because all possible pair-wise combinations between parameters occurs only once.[3]
The given L9 Orthogonal Array assess result of test cases as follows:[1]
Single Mode Faults - Single mode faults occur only due to one parameter. For example, in above Orthogonal array if test cases 7, 8 and 9 show error, we can expect that value 3 of parameter 1 is causing the error. Likewise we can detect as well as isolate the error.
Double Mode Fault - Double mode fault is caused by the two specific parameters values interacting together. Such an interaction is a harmful interaction between interacting parameters.
Multimode Faults - If more than two interacting components produce the consistent erroneous output, then it is a multimode fault. Orthogonal array detects the multimode faults.
Refer the following link for more details: http://www.51testing.com/ddimg/uploadsoft/20090113/OATSEN.pdf
Benefits
- Testing cycle time is reduced and analysis is simpler.
- Test cases are balanced, so it's straightforward to isolate defects and assess performance. This provides a significant cost savings over pair-wise testing.
References
- 1 2 3 4 Pressman, Roger S (2005). Software Engineering: A Practitioner's Approach (6th ed.). McGraw-Hill. ISBN 0-07-285318-2.
- ↑ Phadke, Madhav S. "Planning Efficient Software Tests". Phadke Associates, Inc.
Numerous articles on utilizing Orthogonal Arrays for Software and System Testing.
- 1 2 3 Dustin, Elfriede. "Orthogonally Speaking" (PDF). (subscription required)
External links
- Rao, Calyampudi Radhakrishna (2009). "Orthogonal arrays". Scholarpedia. 4 (7): 9076. doi:10.4249/scholarpedia.9076.
- Delius, Gustav W (May 2004). "Orthogonal Arrays (Taguchi Designs)". University of York.
- Kuhfeld, Warren F. "Orthogonal Arrays". SAS Institute Inc.
SAS provides a catalog of over 117,000 orthogonal arrays.
- Phadke, Madhav S. "Planning Efficient Software Tests". Phadke Associates, Inc.
Numerous articles on utilizing Orthogonal Arrays for Software and System Testing.
- "rdExpert Software for Orthogonal Array Testing". Phadke Associates, Inc.
Commercial toolset for Orthogonal Array Testing.