Configuration testing is a type of software testing that checks the performance of the application system under test against different combinations of hardware and software to arrive at the optimal configuration in which it can perform the best.
For example, if we are testing a Desktop User application, we test the number of combinations of memory sizes, OS (Operating System) versions, memory sizes, hard disk types, and CPUs. We target 4 OS: Windows, Mac, iOS, Android, and minimum and maximum memory sizes, lower and higher (latest) versions of the OS and different browsers versions from lower to higher (latest) versions.
As the scope of possible configurations is typically large, it is crucial to identify which OS-browsers platforms need to be supported.
Configuration testing is not only restricted to software but also applicable to hardware. In hardware configuration testing, we test different hardware devices like scanners, webcams, printers, etc., that support the application testing.
The pre-requisites of configuration testing are:
- A matrix consisting of different combinations of software and hardware configurations is to be created.
- As it is cumbersome and near impossible to test all configurations, they are prioritized.
- Finally, based on the prioritization, every configuration should be tested.
Objectives of Configuration Testing
- Primarily, to determine an optimal configuration of the application that is being tested
- To validate if the application satisfies the configurability requirements
- To identify defects that may not be found during testing by manually causing failures. For example, changing the regional settings of the system like language, time zone, date-time formats, etc.
- To analyze the general performance of the system, by adding or modifying hardware resources like Load Balancers, by increasing or reducing the memory size, by connecting various printer models, etc.
- To analyze system efficiency: how efficiently were the tests performed with the available resources to achieve the optimal system configuration
- To verify how effectively the system performs in a geographically distributed environment. For example: Despite having the server at a different location and clients at a different location, it is verified if the system works smoothly.
- To verify how easily the bugs are reproducible irrespective of the configuration changes
- To ensure how traceable the application items are by documenting and maintaining properly the versions which are easily identifiable.
- To verify how manageable the items of the application are throughout the SDLC (software development life cycle).
In configuration testing, there are two types:
- Software configuration testing
- Hardware configuration testing.
Software Configuration Testing
Software configuration testing usually begins when:
- configurability requirements to be tested are specified;
- test environment is ready and
- when the build has cleared the Unit and Integration tests.
Software configuration testing tests the application against multiple operating systems (OS) and software updates. This process is a very time consuming because each different software needs to be installed and uninstalled while testing.
To resolve this, virtual machines are used. Virtual machine is an environment installed on a software and it simulates the feel of a physical machine for the users and it also simulates real-time configurations.
So, the software is installed in the virtual machine and the testing is continued instead of installing and uninstalling the software in multiple physical machines. Multiple virtual machines could be used as well.
At Hurix, we perform Software Configuration Testing on a virtual machine. Typically, the functional test suite is run across multiple software configurations to verify if the application under test is working as expected. Multiple virtual machines are setup with different software configurations like Win 8, 10 and are tested simultaneously.
Manually failing the test cases by removing some of the configurability requirements and verifying for efficiency is yet another useful strategy that is employed. Our tests have identified several defects; for instance, an application that worked perfectly fine on Win 8, however, crashed on Win 10.
You might also like to read: All You Need To Know About Usability Testing
Hardware Configuration Testing
C.T is also called as Hardware Compatibility testing. During this testing, the tester will test if the software build supports different hardware technologies or not: for example, printer, scanners and different networks.
It is performed in labs with multiple physical machines which have different hardware attached to them. Whenever a build is released, the software has to be installed in all the physical machines and the test suite run on each of them to make sure that the application is running smoothly.
Manually running the tests involves a significant amount of effort and time.
Since there are many kinds of computer hardware and peripherals, the tester has to find out which hardware is used by the majority of the users and after prioritizing, run the test, as it is near impossible to test it on all the hardware available in the market.
At Hurix, when an application is under the test phase, we install it on multiple machines and a test suite is run on each machine. While doing Hardware configuration testing, we specify the configurations that we perform the test on, like say keyboard, mouse, hard disks, processors etc.
Also, system configuration that we test on like P4 CPU, 512 MB of RAM to 16 GB RAM (in Laptops), USB Ports, different net speeds like 2G, #G, 4G and WiFi and the responsiveness in resolutions like 1024 by 768 resolution monitor, printer etc.
At Hurix, we give a lot of importance to configuration testing as it is as important as white box and black box testing, and without it, the software might encounter compatibility issues with systems it was intended to be run on.
Conclusion
In conclusion, it is clear that configuration testing carries special significance as it helps to arrive at the optimal system performance. While it involves time and effort, virtual machines certainly make the task of configuration testing easier.
