Quality organizations are continuously thinking about new ways to improve testing efficiencies and lower costs. The three fundamental elements that constitute a testing function are:

• Requirements – based on which an IT system is developed or modified
• Testers – who build test plans and execute them
• Testing Infrastructure – test servers, mainframes, middleware components, test data, test labs

In the past 10 years, testing organizations have focused on enhancing every aspect of the testing lifecycle that includes improving both the requirement stability index (introducing requirements management tools and processes) and testing effectiveness (productivity improvements).

However, when it comes to Testing Infrastructure (TI), either the status quo has been challenged minimally or the infrastructure has not been aligned with the quality goals of the organization. Thus, quality is compromised owing to an inadequate TI. In many cases, TI is squeezed to such an extent that it leads to quality issues in production.

You may have noticed one or many of the following situations:

• Limited test environments that do not have connectivity with integration systems
• Unavailability of DBA to support test databases
• Test data not refreshed on a need basis or unavailable
• Unavailability of 24/7 support for test environments

The main reasons for the above situations are:

• Lack of necessary skills and incomplete understanding of the infrastructure components within the testing organization
• Need of extensive support from departments external to testing
• Expensive Testing Infrastructure

In this paper, an attempt has been made to delve deep to uncover TI challenges, suggest strategies and recommendations on ways to improve the efficacy and efficiency of TI.

Understanding Testing Infrastructure Needs and Current Habits

A testing organization generally performs the following basic functions while testing its infrastructure:

• TI blueprint maintenance
• Utilization management
• Downtime management
• Monitoring usage of testing tools and licenses, test data sub-setting and provisioning
• Monitoring of TI support

To gain a better understanding of how various testing organizations are managing their TI, a questionnaire-based survey was conducted among people working for 25+ testing organizations. The answers have been consolidated in the table below. This questionnaire can be used as a starting point to better understand the current state of an organization’s TI.

Number	Question	Yes/No %	Comments
1	Do you have a nominated role responsible for test environment management? (SPOC)	Yes – 75% No – 18% NA – 7%	Even though 75% said ‘yes’, most of the respondents highlighted the limited role played by testing organizations in this capacity.
2	Testers and test leads have a good understanding of the TI (hardware, configurations, connectivity, DB, etc)	Yes – 33% No – 67%	Majority of the respondents felt that testers have a limited understanding of their scope and have little clarity on hardware and/or configurations.
3	All test environments (server, DB, connectivity) have 24/7 support	Yes – 15% No – 85%	Despite all respondents working in an offshore model, 85% do not have 24/7 support.
4	Are you collecting environment downtime data for all test environments?	Yes – 90% No – 10%	More than 50% of the respondents highlighted that no action has been taken on the data collected.
5	If data is being collected, has action been taken to reduce the downtime?	Yes – 43% No – 57%	Data has been collected, but no action has been taken. It is easy to implement and hence a quick-win item.
6	Does your test environment have end-to-end testing capability to simulate 80% of the production scenarios?	Yes – 18% No – 82%	Majority of the respondents do not believe that this is necessary.
7	Does your testing organization conduct test environment audits to check for TI deviations after every three releases?	Yes – 4% No – 60% NA – 36%	Majority of the respondents were not aware of any audits.
8	Do you have test data refresh strategy for all test environments?	Yes – 21% No – 79%	Several respondents highlighted initiatives currently in progress for implementing a data refresh strategy.
9	Do you use virtualization or any other environment optimization techniques currently?	Yes – 12% No – 33% NA – 55%	Majority of the respondents were not sure about environment virtualization initiatives undertaken, if any.
10	Are any testing infrastructure improvement programs currently in progress?	Yes – 30% No – 60% NA – 10%	Even though 30% respondents said ‘yes’, less than 6% had clarity on the initiatives being undertaken.

Note: The data collection was questionnaire-based and no interviews were conducted. The above table is used to provide clarity on various TI topics. The accuracy of data has been verified only by the author.

Necessary Functions and Recommendations

1. Testing Infrastructure Blueprint Maintenance

A TI blueprint is a map of all test environments used by the testing organization. The blueprint provides a two-dimensional view of various testing hardware, application interfacing points, middleware components, databases, third-party application connectivity, etc.

The main advantage of creating and maintaining this blueprint is that it gives a clear understanding of:

• How many test environments you have per application
• The robustness of the TI of the entire organization

The blueprint can also be used to understand failure points, test data flow patterns for data refresh and data masking, TI support needs, capacity planning and isolating TI bottlenecks required for testing activities.

2. Utilization Management

The utilization management function is leveraged to maximize TI usage at minimal cost while ensuring improved application availability. If you were to ask your test managers how effectively you are managing TI utilization for your application, it is expected that 99 percent will be of the opinion that it needs significant improvements. This is due to day-to-day experience of environment availability for new projects, test environment downtime, visibility of usage of test environment, etc. The most important aspect of utilization management is to ensure that we strike the right balance between TI optimization and quality of the application performance in production. This correlation can be achieved through the formation of cross-functional teams, including the infrastructure support team and testing organization. The objective of this team is to study test coverage issues, production issues resulting from TI, data provisioning issues, support issues, and third-party application issues which result in quality issues in the production environment.

Another important issue with utilization management is that ROI from testing tools at an enterprise level is rarely performed. Many tools are bought but are utilized only sub-optimally.

3. Downtime Management

This is a simple metric that any organization can collect. Since a majority of the testing organizations work 24/7, downtime has a direct impact on testing schedule and testing productivity. While most organizations are trying to reduce unscheduled downtime, it is imperative to work on reducing planned downtime simultaneously. Downtime data provides a good view of infrastructure robustness and the skill of the support function.

4. TI Support, Data and Testing Tools

TI Support: Infrastructure support is the backbone for utilization and downtime management. In majority of the situations, testers are heavily dependent on developers and DBAs for TI support. Based on challenges faced in your testing organization, the testing community is encouraged to have a better understanding of the application testing infrastructure to perform first-level TI issue assessment. This will help reduce dependence on TI support from external groups.

Data: Moreover, your ability to increase test coverage is dependent on the quality and relevance of the data that needs to be tested. While there is no silver bullet to solve data provisioning issues, it is important to attempt a solution which can be a combination of standard tools and home-grown customized macros to support your application needs. Investments in test data management need to be linked to an increase in test coverage and improvement in testing productivity.

Testing Tools: It is observed that testing tools are almost always underutilized; both in terms of capacity and features. Lack of ownership and enterprise direction on testing tools lead to multiple tools, excessive rework, and skill issues, thus minimizing the ROI.

Recommendations:

• TI Blue Print: The TI blueprint must be audited and updated every three months. This calls for collaboration between the testing and IT infrastructure support teams.
• Utilization Management: Formation (News - Alert) of a cross functional team comprising of Infrastructure services and testing services to monitor, measure and improve TI utilization
• TI Metrics: TI governance needs to be set up to take complete ownership of this function and devise improvement programs which objectively measure the increase in efficiency for all subsequent releases. This governance must attempt to collect the following metrics:

Metric	Comment
% Savings in TI cost	Annual improvement goal
% Reduction in environment downtime	Annual improvement goal
% Reduction in TI-related production defects	Trend by release
% improvement in ROI from investments on tools	Annual improvement goal

• TI Support, Data and Tools: These three topics are complex and needs extensive planning to achieve efficiency improvements. Hence it is recommended that a Champion from the testing team should be identified and a clear implementation roadmap should be built specific to organization needs to achieve the necessary improvements.

Conclusion

Testing Infrastructure improvements can contribute significantly in improving testing efficiency and efficacy. TI’s immense diversity provides significant improvement opportunities for lowering the cost of quality. Hence, it is necessary for testing organizations to identify a single owner who can start strengthening the foundation for TI improvements. This foundation is required to enable the organization to take full advantage of the multiple technology solutions which have evolved over the past two years. These include cloud-based test infrastructure, TDM tools, mainframe hosting on Windows platform, application performance monitoring, among others. A lack of governance and foundation will prove to be a major impediment, posing a challenge to your test organization’s future competitiveness if not attended to at the very outset.

About the author
Vasudeva Naidu is a Delivery Manager with Financial Services Unit of Infosys (News - Alert) and is responsible for project execution and delivery. With over 16 years of experience, he has helped companies define and implement testing efficiency improvement programs and has successfully deployed TCOEs (Testing Center of Excellence). Vasudeva’s key professional achievements include growing and managing large testing teams with team sizes exceeding 400 people. He has implemented several testing thought leadership initiatives over the years.

Edited by Alisen Downey