Kalilur Rahman

5:45 PM, 17th Mar 2018

Intelligent Testing Skills Needed for the Next Generation - Upskill or Retire


Over the past 25 years, testing has gained prominence as a critical phase that determines the success of any project or program. Testing as a discipline/science is approximately a US$ 65–75 billion industry with an estimate of 9 to 15 % year-on-year growth (Ovum and other sources).

Given the stature of the testing industry, a crucial ingredient to make testing work is the people who deliver testing – testers. Like any major discipline, skills are of paramount importance in successful testing. Testing skills are crucial in aligning the test approach with the requirements of end users, making the outcome more relevant to the business context. Customer experience and stakeholder management are vital.

As the velocity of change in technology and the associated delivery methods have increased over the last 30 years, testing has kept pace. This combination of criticality (to the delivery and the business outcome) and the pace of change make testing itself the zenith of technology delivery.

As far as technical evolution and need for advanced testing skills needed for AI, Machine Learning, Big Data, Multi-Channel, Multi-Device Testing and DevOps driven innovation, I shall write another article. This article focuses more on general aspect of testing skills.

Testing involves deep thinking, hard work, and meticulous planning. Testing can be quicker with smart work in the form of continuous improvements and clever planning. A tester’s reward is the identification of genuine, showstopper-critical defects (early in the life cycle) that results in changes to the software, application, program, project requirements, design, or code. Testing is an assimilation of features delivered by the requirements, design, and development phases, and determines the product quality. In the context of quality management, testing is the cornerstone of quality and provides motivation for quality improvements upstream in the lifecycle. Rephrasing a JRD Tata quote2, a tester must forever strive for excellence, or even perfection, in any test, however, small and never be satisfied with the second best result or performance.

This article touches on the key skills a tester should have to succeed in the era of the next generation of testing. Some of the key testing skills (this is not a comprehensive or exclusive list) are:

  • Capability areas – skills including testing knowledge, technical knowledge, analytical and logical reasoning (ability to map Requirements for testing and ability to validate and verify results), end-to-end business, functional and domain knowledge (this is for specialists/specialised phases of testing and is not mandatory for all testers).
  • Soft skills including clear communication (written and verbal), skills in stakeholder management, negotiation, diplomacy, team bonding with global multi-site multi-vendor teams, and empathy with the clients (i.e. both testing and communicating in a manner sympathetic to the end user requirements and demands). Soft skills are very important for a tester to be successful.
  • Other skills including observation, transparency, focused drive towards success, curiosity, exploration, inquisitiveness, critical analysis, creativity, ability to toggle between purism and pragmatism, cognitive psychology with an ability to think in other people’s shoes, attention to detail, knowledge of applied statistics, ability to detect patterns, knowledge gathering skills, ability to withstand pressure, good memory for reconstruction.

Core Skills Needed by Testers

  • First things first – a tester should be familiar with testing concepts. At a bare minimum, an entry-level tester should have a theoretical understanding of the testing concepts, test phases, SDLC, requirements, as well as basic industry, functional, and product knowledge. Like any other mature discipline, testing has solid literature, tools, and standards available. A tester can learn in an iterative structured manner based on their level of knowledge and skills by making use of the numerous learning methods and tools available (online, classroom, simulations) in the market. However, theoretical technical testing alone will not be sufficient, as the testers need to develop an ability to strike a good balance between perfectionism and pragmatism that comes with experience.
  • A professional tester should have a basic understanding of the following testing terms and concepts, depending on geography and industry. Some of the nice-to-learn topics are:
  1. Testing terms and concepts (TMMi, BS 7925-1, NIST 500-234 or similar industry/global standard)
  2. Test types, test phases, test prioritization, risk-based testing, regression test selection, test automation, test traceability to requirements, shift-left testing, model-based standards, application based standards, application/product-specific functional standards, stakeholder management, requirements integrity levels, to name but a few
  3. Test policies (of testing organization, client firm)
  4. Test strategies (IEEE 1012), approaches and documentation (IEEE 829, ISTQB)
  5. Software engineering standards (ISO 12207, 15026, 15288, IEEE 1028, 1044, CMMi)
  6. IEEE Computer Society –SWEBOK focusing on software testing knowledge area and quality management, software (cognitive) ergonomics, and systems engineering discipline.

In addition to the basic knowledge, professional testers will be required to acquire knowledge in many areas within a program or project. These include:

  1. Vision, mission, goals of the program/project
  2. Customer experience and usability expectations
  3. Scope of the project/program/test phase
  4. Program or project test plan/test approaches
  5. Test phase plan (For system, domain/continuous integration/assembly/link, systems integration testing, user acceptance testing, operational acceptance testing, usability testing, etc.)
  6. Test process (generic or specific to a test phase)
  7. Defect management ( as per test policy or industry standard)
  8. Environment, data and support management, and test documentation
  9. Test tools and the skills to use them
  10. Test process and tester productivity improvement measures using tools, processes, and templates
  11. Lessons learned, dos and don’ts from past experiences in the firm, project, or earlier phases
  12. Challenge the constraints a team faces
  13. Navigate and understand the goals and mission of the project/program/test type
  14. Collaborate meticulously with the various teams/partners/vendors involved
  15. Manage risks and issues proactively with the project team(s)
  16. Provide excellent observation, recording, and documentation
  17. Be a cautious judge with critical thinking and the ability to be pragmatic when needed
  18. Be flexible enough to adapt to challenging situations without deviating from the original test strategy
  19. Handle ambiguity with a continuous learning mentality
  20. Take accountability for testing the relevant task in hand
  21. Treat clients with a ‘client first’ mentality
It is important for testers to understand the importance of their work – there are major examples of testing-related defects causing newspaper headlines, such as an aircraft being grounded due to battery issues, a financial institution facing IT integration issues, a floating point bug in a processor, a space shuttle failure, an airport baggage handling system going berserk, to name but a few.

From these categories, tests need to have three key inputs to succeed:

  1.  Project/program/solution context
  2. Technical/functional knowledge
  3. Personal experience and knowledge gathered These three key essentials give the testers a grip on:
  4. Stakeholder context
  5. Project understanding from a technical, functional, and business architecture perspective
  6. Testing methodology and good practices to follow
  7.  Soft skills needed to succeed
  8.  Pragmatism oriented Shortcuts and quick wins for effective testing

Once mastered, testers graduate to possess the skills (as a testing or subject matter expert) that allow them to:

▪  Challenge the constraints a team faces

▪  Navigate and understand the goals and mission of the project/program/test type

▪  Collaborate meticulously with the various teams/partners/vendors involved

▪  Manage risks and issues proactively with the project team(s)

▪  Provide excellent observation, recording, and documentation

▪  Be a cautious judge with critical thinking and the ability to be pragmatic when needed

▪  Be flexible enough to adapt to challenging situations without deviating from the original test strategy

▪  Handle ambiguity with a continuous learning mentality

▪  Take accountability for testing the relevant task in hand

▪  Treat clients with a ‘client first’ mentality

Requirement and Development Skills Mapped to the Tester’s Skills

Requirements determine the key testing scope of a project/program/ system/solution. Requirements are an integral pointer to the success or failure of a project or program. A quality business requirement will result in a high-quality development and a quality product/solution that meets the end user’s expectations. As per industry studies, a requirement- related bug found in production could cost between 10 and 100 times more than a defect found earlier in the life-cycle and as much as 10 times one found during the test phase. Similarly, a design or architectural miss can cost 25 to 100 times the initial cost4 if found in production, and approximately 10 to 15 times more if found during the build/test phases. Based on information on the Internet and various studies, the highest number of delivered defects is requirements-related5 and better value is delivered by improving the efficiency of defect removal during the requirements phase. This is largely due to a lack of a disciplined requirements management process. A disciplined, tools-managed process will help companies and programs prevent budget overruns by reducing the cost of poor quality. The tester needs to work with the key business analysts (BA), business transformation leads, and designers who have the business context and knowledge and help shape the final end user deliverables.

The quality of the requirements determines the quality of the end product and, hence, proper scrutiny of the requirements using static testing techniques such as reviews, walkthroughs, and inspection, visual validation (all manual), or using automated tools for requirements analysis for compliance is a must. Currently, standard tools are available in the market to convert requirements to test models and create test scenarios and conditions in an automated manner – covering boundary value conditions and negative scenarios. While it is good for testers to understand the fundamentals, it is equally important to check the output of the automated tools for consistency and accuracy. Experienced testers who understand the business function or industry very well can carry this out.

Requirements and Test Scope Management Skills

One of the key skills of the tester is to perform requirement verification checks. Some of the checks include singularity, ambiguity, generalization, solidity, comprehensiveness, style, extensibility, maintainability, reliability, integrity, robustness, scalability and testability. Terms and definitions differ between various companies and automated requirements verification tools. Testers need to be familiar with the terminologies to perform relevant requirement verification checks during the static testing phase or for test design inputs.

For test requirements, one of the key Six-Sigma concepts – SMART – can be used for any testing task or requirement. A SMART requirement is a good requirement. A test designer should have certain skills to define a test requirement in a SMART manner. Test requirements need to be very Specific about a particular task or an activity, results or output should be Measurable, and Attainable with Relevancy and a Time-bound response.

Additionally, a good test designer will be familiar with leading test de- sign techniques such as Equivalence partitioning, Logical Combination, Taguchi Method, Orthogonal Array, State Transition Model, Karnaugh Maps(K-Map), Risk-Based Testing, Exploratory, Fail-First to convert the requirements to a solid set of requirements, resulting in high quality test coverage with an optimum number of test cases. This will result in a high-quality product at optimal cost. Testing stops and it never finishes. as the number of tests, it is possible to execute is large. Hence, a tester must understand the client’s accepted level of risk under different client circumstances will drive different testing behaviours, and this is where having industry experience for similar clients can lead to behaviours resulting in higher value for the client.

Smarter test scope management can be achieved by using industry- leading standard test requirements and test management tools. Key requirements can be understood implicitly by testers by keeping constantly aware of industry trends and having the ability to incorporate new testing methods (innovative and intelligent testing). Building such a proactive team of testers will help build a very high performing test team.

Testing Specialization

Testing Generalist
  • To become a generalist, it would be good for a communications industry aligned tester to be familiar with eTOM (enhanced telecoms operations map) and key standards for the type of operator (e.g. 3G/4G for a wireless, PSTN/xDSL for a fixed line, OTT, DOCSIS standard for a cable MSO, etc). For Pharma - Focus on GxP Methodologies, How Research & Development, Manufacturing or Commercial Operations work etc.
Testing Specialist
  • To becoming a specialist, it would be good for a tester to focus on a particular type of testing and have in-depth knowledge of and capability in the key tools needed to perform testing. For example, to become a good performance tester, it is good for testers to have expertise in some/ all of the concepts.
  • Requirements gathering and analysis of non-functional performance measures
  • Performance modelling, and design and architecture
  • Workload, performance architecture and capacity models
  • Virtualization and extrapolation using performance technologies, such as low-latency tools, in-memory, solid-state devices, network tuning, data analytics, and caching using big-data
  • Performance test execution and statistics capture
  • Performance tuning measures, such as code and hardware
  • Performance testing using tools and automation
  • Skills to use probes/tools to measure and optimize performance

It is common to see deeply skilled specialists in one particular area or tool. Hence specialisation is a good thing to have if a tester chooses the path of the depth of expertise.

Options for Testing Testing Career Path

Test Career Trajectory

Testing generalist track

Tester → test module lead (design/execution) → test lead → work stream test manager → project test manager → program test manager →portfolio test director → testing practice lead

Test design/process specialist track

Tester → test design/process lead → test architect lead → work stream test manager → project test manager → program test manager →portfolio test director → testing practice lead

Test data/support generalization track

Test data team member → test data module lead → test lead → test support manager → project test support manager → project test manager → program test manager → portfolio test director → testing practice lead

Performance test specialization track

Performance test specialist → performance work modeling lead → performance test lead → performance test manager → performance test expert → program non-functional/performance test manager → portfolio test director → testing practice lead

Sample Functional Specialization: Telecoms billing specialization track

Telecoms mediation tester → telecoms bill run module lead → billing test module lead → billing test lead → billing testing work stream lead/ manager → program billing test manager → billing test portfolio director → testing practice lead

Next Generation Testing Industry – Skills Needed

Taking an example of the telecommunications industry (communications) has grown phenomenally over the past 20 years. The number of innovations and advancements made by the communications industry is second to none and is one of the engines for growth in the economy, the IT industry, and testing in general. With such phenomenal growth, numerous challenges come onboard.

From a testing skills standpoint, the challenges posed by this growth include the need to keep the various types of testing requirements for different communications industry projects and roll-outs. The telecommunications industry is vast and there are different functional domains. Testers tend to be specialists in one or more of these areas, but will seldom master everything because the technologies, frameworks, and standards are numerous.

Testing Career Progression Graph

Testing as a career has a lot of options from a growth perspective. Some choices for a tester to progress along the test management path as a generalist areas seen in the figure above.

In order to progress along a testing career track as a specialist, a tester can develop his career as a specialist in one of the following:

▪  Test data subject matter expert (SME), test architect, test metrics SME, test environment discovery, test environment delivery, defect management, war-room/problem manager, incident manager, test automation, test support tools, test requirements, report management, performance testing, security or penetration testing, Web/ usability testing, compatibility and accessibility testing, operation- al acceptance testing (OAT).

A tester can complete training and become a certified specialist in testing from standards organisations such as BCS, QAI, ISTQB, IIST, ASQ, ISEB, CSTM, CMST, PTCRB, and GCF. There is specialisation certification for tools such as HP Performance Centre, Selenium, Tricentis Tosca, Worksoft Certify, Panaya, SOAP, SilkTest, Rational Quality Manager – Performance Tester, and Rational Robot, to name but a few.

A tester should have a probing mindset and should be capable of dis- covering the features of a product, process, or service and understand what the customers want (from the requirements), how the product should work ( requirements and design), design the tests using various design techniques, select tests with a fail-first mentality with the aim of identifying defects in order to fix them, run the tests (using tools or manually), observe the test outcome, and validate the results against the expected behavior.


Testing as a career has become a mainstay and is the pinnacle of success for large-scale product development or program delivery. Testing is a key contributor to the development of the IT industry and the phenomenal growth in technology which has taken place over the past three decades. Testing as a horizontal (across the SDLC life-cycle) or a vertical (industry-specific) is a hotbed of innovation, creativity, and productivity improvement in terms of processes (with processes such as TMMi), products (automation tools, intelligent testing tools, productivity tools), and people (thought leaders and innovators).

Testers have the opportunity to become generalist test experts with good industry knowledge of the testing life cycle, or a highly skilled expert or specialist in a particular test type, test phase, test activity, tool, or industry, or an expert in test delivery and management before taking charge of a test centre or test practice.

Especially in communications industry related testing:

▪  There will be growth in some markets and maturity in others. Both will drive rapid technical and business changes, with the business and operating model proving to be very dynamic.

▪  In the next 5 to 10 years, the outlook is for continued change as the market for data further drives mobile towards being a commodity market and seamless, always on the move with the Internet of Things becoming a de-facto lifestyle.

▪  In content delivery, the changes in platform/delivery method mean increasingly interactive and mobile applications will continue to stretch testing methods both for media providers and telecoms companies.

It will be a nice challenge for the testing experts and testing services firms to keep up with the phenomenal growth in the industry and the variations and permutations for service options, commercial models, career growth, innovation, and transformation in various industries. Testing as a tipping point has been and will be a major contributor to the success of the IT industry. If there is one area where skills can be an evergreen option with continuous opportunities for learning and growth, it is testing. For one to be successful in the next generation of testing, it is important to keep learning and keep innovating. A career in testing comes with infinite options – akin to a famous quote by James Bach, which states: “Testing is potentially an infinite process”.

I hope you enjoyed reading the post and comments on this topic. Caring is sharing. Feel free to like, share or comment on what you think! Please tag me if you forward this for relevance.

Credits: Header and most of the images are designed using MS PowerPoint. All other linked quotes and images available freely on the Public Internet. Quotes are freely available on the Internet for reference. Respective trademarks owned by corresponding firms. Opinions highlighted are from a personal experience standpoint and in no way reflect the views of my current or past employers or clients.

NoteThis post is a repurposed and an article I published in "Testing Experience" Magazine sometime back. All comments and opinions expressed (other than facts referred from sources) are personal opinions of the author.

#TestingSkills #IntelligentTesting #TestManagement #QualityCenterOfExcellence #QCoE #TestingCenterOfExcellence #TCoE #TestingIsDeadLongLiveTesting #NextGenTesting #TestMaturity #QualityDelivery #TestingLifeCycle #QualityFocusedDelivery

Sources of Information

Push pixels...
Shovel coal into server...
Create mockups...
Defend the wall...
Draft storyboard...
Disrupt an industry...
Achieve profitability...
Become a unicorn...
Become Batman...