Software testing
 | This article's style of writing may not reflect the encyclopedic tone used on Wikipedia. (August 2026) |
| Part of a series on |
| Software development |
|---|
Software testing is the act of checking whether software meets its intended objectives and satisfies expectations.
Software testing can provide objective, independent information about the quality of software and the risk of its failure to a user or sponsor or any other stakeholder.[1]
Software testing can determine the correctness of software for specific scenarios but cannot determine correctness for all scenarios.[2][3] It cannot find all bugs.
Based on the criteria for measuring correctness from an oracle, software testing employs principles and mechanisms that might recognize a problem. Examples of oracles include specifications, contracts,[4] comparable products, past versions of the same product, inferences about intended or expected purpose, user or customer expectations, relevant standards, and applicable laws.
Software testing can be functional or non-functional in nature.
Software testing is often dynamic in nature: running the software to verify actual output matches expected. It can also be static in nature: reviewing code and its associated documentation.
Software testing is often used to answer the question: Does the software do what it is supposed to do and what it needs to do?
Information learned from software testing may be used to improve the process by which software is developed.[5]: 41–43
A commonly suggested approach to automated testing is the "test pyramid," wherein most of the tests are unit tests, followed by a smaller set of integration tests and finally a few end-to-end (e2e) tests.[6][7][8]
Economics
[edit]A study conducted by NIST in 2002 reported that software bugs cost the U.S. economy $59.5 billion annually. More than a third of this cost could be avoided if better software testing was performed.[9][dubious – discuss]
Outsourcing software testing because of costs is very common, with China, the Philippines, India and Pakistan being preferred destinations.[10]
History
[edit]Glenford J. Myers initially introduced the separation of debugging from testing in 1979.[11] Although his attention was on breakage testing ("A successful test case is one that detects an as-yet undiscovered error."[11]: 16 ), it illustrated the desire of the software engineering community to separate fundamental development activities, such as debugging, from that of verification. Software testing typically includes handling software bugs – a defect in the code that causes an undesirable result.[12]: 31 Bugs generally slow testing progress and involve programmer assistance to debug and fix.
Not all defects cause a failure. For example, a defect in dead code will not be considered a failure.
A defect that does not cause failure at one point in time may lead to failure later due to environmental changes. Examples of environment change include running on new computer hardware, changes in data, and interacting with different software.[13]
Goals
[edit]Software testing is typically goal driven.
Finding bugs
[edit]Software testing typically includes handling software bugs – a defect in the code that causes an undesirable result.[12]: 31 Bugs generally slow testing progress and involve programmer assistance to debug and fix.
Not all defects cause a failure. For example, a defect in dead code will not be considered a failure.
A defect that does not cause failure at one point in time may lead to failure later due to environmental changes. Examples of environment change include running on new computer hardware, changes in data, and interacting with different software.[13]
A single defect may result in multiple failure symptoms.
Ensuring requirements are satisfied
[edit]Software testing may involve a Requirements gap – omission from the design for a requirement.[5]: 426 Requirement gaps can often be non-functional requirements such as testability, scalability, maintainability, performance, and security.
Code coverage
[edit]A fundamental limitation of software testing is that testing under all combinations of inputs and preconditions (initial state) is not feasible, even with a simple product.[3]: 17–18 [14] Defects that manifest in unusual conditions are difficult to find in testing. Also, non-functional dimensions of quality (how it is supposed to be versus what it is supposed to do) – usability, scalability, performance, compatibility, and reliability – can be subjective; something that constitutes sufficient value to one person may not to another.
Although testing for every possible input is not feasible, testing can use combinatorics to maximize coverage while minimizing tests.[15]
Categories
[edit]
| This section contains an excessive amount of intricate detail. (June 2026) |
Testing can be categorized many ways.[16]
Automated testing
[edit]Test automation is the use of software (separate from the software being tested) for controlling the execution of tests and comparing actual outcome with predicted.[17] Test automation supports testing the system under test (SUT) without manual interaction which can lead to faster test execution and testing more often. Test automation is a key aspect of continuous testing and often for continuous integration and continuous delivery (CI/CD).[18]
Levels
[edit]Software testing can be categorized into levels based on how much of the software system is the focus of a test.[19][20][21][22]
Unit testing
[edit]Unit testing, also known as component or module testing, is a form of software testing by which isolated source code is tested to validate expected behavior.[23]
Integration testing
[edit]Integration testing is a form of software testing in which multiple software components, modules, or services are tested together to verify they work as expected when combined. The focus is on testing the interactions and data exchange between integrated parts, rather than testing components in isolation.
System testing
[edit]System testing, a.k.a. end-to-end (E2E) testing, is testing conducted on a complete software system.
Static, dynamic, and passive testing
[edit]There are many approaches to software testing. Reviews, walkthroughs, or inspections are referred to as static testing, whereas executing programmed code with a given set of test cases is referred to as dynamic testing.[24][25]
Static testing is often implicit, like proofreading, plus when programming tools/text editors check source code structure or compilers (precompilers) check syntax and data flow as static program analysis. Dynamic testing takes place when the program itself is run. Dynamic testing may begin before the program is 100% complete in order to test particular sections of code and may be applied to discrete functions or modules.[24][25] Typical techniques for these are either using stubs/drivers or execution from a debugger environment.[25]
Static testing involves verification, whereas dynamic testing also involves validation.[25]
Passive testing means verifying the system's behavior without any interaction with the software product. Contrary to active testing, testers do not provide any test data but look at system logs and traces. They mine for patterns and specific behavior in order to make some kind of decisions.[26] This is related to offline runtime verification and log analysis.
Exploratory
[edit]Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1984,[27] defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."[28]
Preset testing vs adaptive testing
[edit]The type of testing strategy to be performed depends on whether the tests to be applied to the IUT should be decided before the testing plan starts to be executed (preset testing[29]) or whether each input to be applied to the IUT can be dynamically dependent on the outputs obtained during the application of the previous tests (adaptive testing[30][31]).
Black/white box
[edit]Software testing can often be divided into white-box and black-box. These two approaches are used to describe the point of view that the tester takes when designing test cases. A hybrid approach called grey-box that includes aspects of both boxes may also be applied to software testing methodology.[32][33]
White-box testing
[edit]
White-box testing (also known as clear box testing, glass box testing, transparent box testing, and structural testing) verifies the internal structures or workings of a program, as opposed to the functionality exposed to the end-user. In white-box testing, an internal perspective of the system (the source code), as well as programming skills are used to design test cases. The tester chooses inputs to exercise paths through the code and determines the appropriate outputs.[32][33] This is analogous to testing nodes in a circuit, e.g., in-circuit testing (ICT).
While white-box testing can be applied at the unit, integration, and system levels of the software testing process, it is usually done at the unit level.[34] It can test paths within a unit, paths between units during integration, and between subsystems during a system–level test. Though this method of test design can uncover many errors or problems, it might not detect unimplemented parts of the specification or missing requirements.
Techniques used in white-box testing include:[33][35]
- API testing – testing of the application using public and private APIs (application programming interfaces)
- Code coverage – creating tests to satisfy some criteria of code coverage (for example, the test designer can create tests to cause all statements in the program to be executed at least once)
- Fault injection methods – intentionally introducing faults to gauge the efficacy of testing strategies
- Mutation testing methods
- Static testing methods
Code coverage tools can evaluate the completeness of a test suite that was created with any method, including black-box testing. This allows the software team to examine parts of a system that are rarely tested and ensures that the most important function points have been tested.[36] Code coverage as a software metric can be reported as a percentage for:[32][36][37]
- Function coverage, which reports on functions executed
- Statement coverage, which reports on the number of lines executed to complete the test
- Decision coverage, which reports on whether both the True and the False branch of a given test has been executed
100% statement coverage ensures that all code paths or branches (in terms of control flow) are executed at least once. This is helpful in ensuring correct functionality, but not sufficient since the same code may process different inputs correctly or incorrectly.[38]
Black-box testing
[edit]
Black-box testing (also known as functional testing) describes designing test cases without knowledge of the implementation, without reading the source code. The testers are only aware of what the software is supposed to do, not how it does it.[39] Black-box testing methods include: equivalence partitioning, boundary value analysis, all-pairs testing, state transition tables, decision table testing, fuzz testing, model-based testing, use case testing, exploratory testing, and specification-based testing.[32][33][37]
Specification-based testing aims to test the functionality of software according to the applicable requirements.[40] This level of testing usually requires thorough test cases to be provided to the tester, who then can simply verify that for a given input, the output value (or behavior), either "is" or "is not" the same as the expected value specified in the test case. Test cases are built around specifications and requirements, i.e., what the application is supposed to do. It uses external descriptions of the software, including specifications, requirements, and designs, to derive test cases. These tests can be functional or non-functional, though usually functional. Specification-based testing may be necessary to assure correct functionality, but it is insufficient to guard against complex or high-risk situations.[41]
Black box testing can be used to any level of testing although usually not at the unit level.[34]
Component interface testing
Component interface testing is a variation of black-box testing, with the focus on the data values beyond just the related actions of a subsystem component.[42] The practice of component interface testing can be used to check the handling of data passed between various units, or subsystem components, beyond full integration testing between those units.[43][44] The data being passed can be considered as "message packets" and the range or data types can be checked for data generated from one unit and tested for validity before being passed into another unit. One option for interface testing is to keep a separate log file of data items being passed, often with a timestamp logged to allow analysis of thousands of cases of data passed between units for days or weeks. Tests can include checking the handling of some extreme data values while other interface variables are passed as normal values.[43] Unusual data values in an interface can help explain unexpected performance in the next unit.
Visual testing
[edit]The aim of visual testing is to provide developers with the ability to examine what was happening at the point of software failure by presenting the data in such a way that the developer can easily find the information he or she requires, and the information is expressed clearly.[45][46]
At the core of visual testing is the idea that showing someone a problem (or a test failure), rather than just describing it, greatly increases clarity and understanding. Visual testing, therefore, requires the recording of the entire test process – capturing everything that occurs on the test system in video format. Output videos are supplemented by real-time tester input via picture-in-a-picture webcam and audio commentary from microphones.
Visual testing provides a number of advantages. The quality of communication is increased drastically because testers can show the problem (and the events leading up to it) to the developer as opposed to just describing it, and the need to replicate test failures will cease to exist in many cases. The developer will have all the evidence he or she requires of a test failure and can instead focus on the cause of the fault and how it should be fixed.
Ad hoc testing and exploratory testing are important methodologies for checking software integrity because they require less preparation time to implement, while the important bugs can be found quickly.[47] In ad hoc testing, where testing takes place in an improvised impromptu way, the ability of the tester(s) to base testing off documented methods and then improvise variations of those tests can result in a more rigorous examination of defect fixes.[47] However, unless strict documentation of the procedures is maintained, one of the limits of ad hoc testing is lack of repeatability.[47]
Grey-box testing
[edit]Grey-box testing (American spelling: gray-box testing) involves using knowledge of internal data structures and algorithms for purposes of designing tests while executing those tests at the user, or black-box level. The tester will often have access to both "the source code and the executable binary."[48] Grey-box testing may also include reverse engineering (using dynamic code analysis) to determine, for instance, boundary values or error messages.[48] Manipulating input data and formatting output do not qualify as grey-box, as the input and output are clearly outside of the "black box" that we are calling the system under test. This distinction is particularly important when conducting integration testing between two modules of code written by two different developers, where only the interfaces are exposed for the test.
By knowing the underlying concepts of how the software works, the tester makes better-informed testing choices while testing the software from outside. Typically, a grey-box tester will be permitted to set up an isolated testing environment with activities, such as seeding a database. The tester can observe the state of the product being tested after performing certain actions such as executing SQL statements against the database and then executing queries to ensure that the expected changes have been reflected. Grey-box testing implements intelligent test scenarios based on limited information. This will particularly apply to data type handling, exception handling, and so on.[49]
With the concept of grey-box testing, this "arbitrary distinction" between black- and white-box testing has faded somewhat.[34]
Installation testing
[edit]Installation testing is a type of software testing that verifies that users can successfully install and set up software in its intended environments (e.g., operating systems, computer hardware). Most software systems have installation procedures that are needed before they can be used for their main purpose. Installation testing focuses on these procedures and whether they're sufficient for achieving an installed, usable software system.[50]: 139 Procedures of this kind may involve full or partial upgrades, and install/uninstall processes.
- A user must select a variety of options.
- Dependent files and libraries must be allocated, loaded or located.
- Valid hardware configurations must be present.
- Software systems may need connectivity to connect to other software systems.[50]: 145
- Valid, accurate, and sufficient documentation (e.g., installation guide, user manual, quick reference, README file, etc.) must be present and accessible.[51]
Compatibility testing
[edit]A common cause of software failure (real or perceived) is a lack of its compatibility with other application software, operating systems (or operating system versions, old or new), or target environments that differ greatly from the original (such as a terminal or GUI application intended to be run on the desktop now being required to become a Web application, which must render in a Web browser). For example, in the case of a lack of backward compatibility, this can occur because the programmers develop and test software only on the latest version of the target environment, which not all users may be running. This results in the unintended consequence that the latest work may not function on earlier versions of the target environment, or on older hardware that earlier versions of the target environment were capable of using. Sometimes such issues can be fixed by proactively abstracting operating system functionality into a separate program module or library.
Smoke and sanity testing
[edit]Sanity testing determines whether it is reasonable to proceed with further testing.
Smoke testing consists of minimal attempts to operate the software, designed to determine whether there are any basic problems that will prevent it from working at all. Such tests can be used as build verification test.
Regression testing
[edit]Regression testing focuses on finding defects after a major code change has occurred. Specifically, it seeks to uncover software regressions, as degraded or lost features, including old bugs that have come back. Such regressions occur whenever software functionality that was previously working correctly, stops working as intended. Typically, regressions occur as an unintended consequence of program changes, when the newly developed part of the software collides with the previously existing code. Regression testing is typically the largest test effort in commercial software development,[52] due to checking numerous details in prior software features, and even new software can be developed while using some old test cases to test parts of the new design to ensure prior functionality is still supported.
Common methods of regression testing include re-running previous sets of test cases and checking whether previously fixed faults have re-emerged. The depth of testing depends on the phase in the release process and the risk of the added features. They can either be complete, for changes added late in the release or deemed to be risky, or be very shallow, consisting of positive tests on each feature, if the changes are early in the release or deemed to be of low risk.
Acceptance testing
[edit]Acceptance testing is system-level testing to ensure the software meets customer expectations.[53][54][55][56]
Acceptance testing may be performed at the end or in the middle of a project, including after each completed user story in an agile project.[57]
Tests are frequently grouped into these levels by where they are performed in the software development process, or by the level of specificity of the test.[56]
- User acceptance testing (UAT)
- Operational acceptance testing (OAT)
- Contractual and regulatory acceptance testing
- Alpha and beta testing
Sometimes, UAT is performed by the customer, in their environment and on their own hardware.
OAT is used to conduct operational readiness (prerelease) of a product, service or system as part of a quality management system. OAT is a common type of non-functional software testing, used mainly in software development and software maintenance projects. This type of testing focuses on the operational readiness of the system to be supported, or to become part of the production environment. Hence, it is also known as operational readiness testing (ORT) or operations readiness and assurance (OR&A) testing. Functional testing within OAT is limited to those tests that are required to verify the non-functional aspects of the system.
In addition, the software testing should ensure that the portability of the system, as well as working as expected, does not also damage or partially corrupt its operating environment or cause other processes within that environment to become inoperative.[58]
Contractual acceptance testing is performed based on the contract's acceptance criteria defined during the agreement of the contract, while regulatory acceptance testing is performed based on the relevant regulations to the software product. Both of these two tests can be performed by users or independent testers. Regulation acceptance testing sometimes involves the regulatory agencies auditing the test results.[56]
Alpha testing
[edit]Alpha testing is simulated or actual operational testing by potential users/customers or an independent test team at the developers' site. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing before the software goes to beta testing.[59]
Beta testing
[edit]Beta testing comes after alpha testing and can be considered a form of external user acceptance testing. Versions of the software, known as beta versions, are released to a limited audience outside of the programming team known as beta testers. The software is released to groups of people so that further testing can ensure the product has few faults or bugs. Beta versions can be made available to the open public to increase the feedback field to a maximal number of future users and to deliver value earlier, for an extended or even indefinite period of time (perpetual beta).[60]
Functional vs non-functional testing
[edit]Functional testing refers to activities that verify a specific action or function of the code. These are usually found in the code requirements documentation, although some development methodologies work from use cases or user stories. Functional tests tend to answer the question of "can the user do this" or "does this particular feature work."
Non-functional testing refers to aspects of the software that may not be related to a specific function or user action, such as scalability or other performance, behavior under certain constraints, or security. Testing will determine the breaking point, the point at which extremes of scalability or performance leads to unstable execution. Non-functional requirements tend to be those that reflect the quality of the product, particularly in the context of the suitability perspective of its users.
Continuous testing
[edit]Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.[61][62] Continuous testing includes the validation of both functional requirements and non-functional requirements; the scope of testing extends from validating bottom-up requirements or user stories to assessing the system requirements associated with overarching business goals.[63][64]