This essay is targeted at the managers of the digital development (IT) section of businesses that want to make or save money. It will come as a surprise to some people, that there is more than one “school” of testing. I have been exposed to two extremes, and through my 12 years of experience, and with personal integrity, have accepted one of them as supporting and representing the high standard of professionalism that I endeavour to offer my employer. This is probably not the one you are acquainted with. Allow me, please, to enlighten you.
Testing is a specialised cognitive skill. As with any such skills, from people management to oil painting, it cannot be reduced to a number of steps (and because of that it can’t be done by a machine – more about that later). A test case is not a test. At best, it is a test guide or idea. There is no way a “test case” could ever be written to represent all the analysis that a tester does, intrinsically, heuristically, and because it is impossible to do 100% testing there is no way that producing “comprehensive test cases” ever comes close to representing the test coverage or the actual analysis done. If a confirmation can be done by a machine, then it is a “check”, not a test. Testing involves mindful discerning analysis of the observations made during deliberate experimentation.
As a simplistic example of this, imagine a “test case” that confirms that Box C shows the factor of boxes A and B. If this “test case” was automated, we’d have to set the values of boxes A (6) and B (7) so we know that Box C will get the same results each time. A coded check would confirm that 42 shows in Box C: pass. A person checking would intrinsically notice if the result displays aligned properly, in the correct format, font and size and colour, in base 10, if it took a long time to display, or if Outlook crashed when the result displayed. A person testing could change Box B to 8 and see Box C does not change from 42 – turns out that was hard-coded so the auto-check would pass… There are infinite tests that could be done: even having different programs open at the same time on your computer can affect a function. A business is not going to ask a tester to test every possible combination of programs, browsers, operating systems, settings…
What would a “pass” on this test reveal? If it is done by a different tester in a slightly different way eg inputting Box B before content in Box A, or tabbing instead of mouse clicks is that a different test? It could get a different result. Also, if it fails the first 2 times, works on the third, is broken by a regression issue, fixed and is currently working at release, that usually shows as Count of Test Case = 1, Pass = 1, 100% Pass – and tells us nothing about the potential risks of the function.
At best, a passing test is a rumour of success… A failing test is an allegation of failure. – M Bolton
We should also keep in mind that “test cases” do not appear out of thin air. Since they are usually expected to be produced in advance of the tester getting access to the product, they are generally based around the testers’ other available oracles, for example the BRD. If these “test cases” are to accurately reflect the main functional requirements of the product, it means the design and requirements need to be produced to a high standard as well. It is not reasonable to expect better detail from test cases than the rest of the project documentation. Waterfall methodology is fundamentally flawed. Can any business really believe that they are going to design an entire 6 month project 100% correctly from the start? If the business is interested in making money and being at the cutting edge of digital development and consumer interests, they should be working iteratively from Minimum Viable Product and enhancing from there, and that does not support the argument for the creation of a complete suite of static “test cases” before testing even starts.
The expectation of and reliance on “test cases” and automation in many digital workplaces indicates a fundamental lack of understanding of what testing actually is. This is commonly enabled and enforced by members of the test community itself, many of whom have never questioned the inconsistencies inherent in the establishment process (which shows an indicative lack of discernment for people who should be professional sceptics). I have heard various dogma about why people think we need test cases. I say “dogma” because, if they were arguments, they would be dropped when the people who believed them were given a compelling rational counter-argument that clearly disproved the belief…
- “We have written test cases so new testers know how to test a system.”
- First, test cases are not designed to teach anything. They don’t explain what user need the program is supposed to address or how the user is anticipated to use it, or what the out of the various functions mean to the user, or why the designer decided to choose one function over a different one, or the business priorities with respect to the program.
- Second, you would think that, if you’ve hired an experienced professional tester, they’d already know how to test. All they need to do is learn what the program is for and how to use it, experiment, and question to start getting a feel for the product and the project.
- Third, if you want to teach someone to test (ie use critical thinking, structured analysis, intuitive leaps and intelligent informed reasoning supporting their reports), you have to encourage them to think far beyond learning to follow a set of mindless, repetitive steps that can in fact get in the way of analytical thinking.
- “… so we can get test coverage and test progression metrics.”
- It is not true that all tests are equal. A fail in one critical path is worth far more than 10 typos in page content, yet each test case is counted equally when “pass rates” are calculated.
- Does the metric really reflect the value you want to measure? 80% of the test cases may be run, but that’s because they were easy – the remaining 20% may take 80% of the time, so this doesn’t show how close we are to completion.
- There are many more such inconsistencies with this idea. I recommend reading this article here before considering reducing tests and test activities to numbers.
- “… so we can prove coverage and compliance.”
- As I pointed out above – we will never have complete coverage, and any set of documentation that says otherwise is false.
- This also assumes that every tester follows every written step, every time they do the tests, which I don’t think is a universal truth. It seems to me far more useful to inform the test professional of the details if you feel you are somehow obliged to run specific tests on certain occasions, and allow them to determine the constraints of that requirement.
- “…to have a set of “standardised” regression tests.”
- No tester will ever follow a set of instructions the same way twice, and a different tester will behave significantly different with the same instructions, so the idea of standardisation is completely illogical, unless the checks are run by a machine in an otherwise closed environment.
- Even the best testers will begin to miss things when asked to mindlessly follow the same set of checks and tests every time the program is updated. Intelligent people get bored with repetition, and rush things, to get it over with. It is not testing if your mind is not engaged.
- A (usually) truncated set of tests such as this will obfuscate the areas of change and real risk. Instead of a wild shot-gun blast of tests in the general direction of the potentially affected areas, it is far more efficient and effective to work with the developers to understand what code was changed, and where that code is used, and then target the testing to the areas of most risk. Regression defects may appear random on the surface, but they always have a cause. Working with the devs also helps them understand how they have to think about more than just the function under work, so they write better code.
- “…so someone knows where to pick up if your tester gets hit by a bus.”
- This is the best reason I have seen, but still, if your tester has been giving clear and informative progress and issue reports along the way, it should be obvious what they have done, and any good tester should be able to see what yet needs to be done. Progress reports that say “Of 33 test cases, I have run 17, and 5 of these have failed” do not actually tell a replacement tester what testing has been done.
I was appalled to see, in the results of a job search, the term “Automated tester” being commonly used. Gone is the facade of testers being “test automation engineers” – they now say explicitly that testing can be done by a robot. We have cheapened our profession so much, allowing people to believe that what we do can be reduced to a series of steps, and our valuable analysis is just a number, that they now rationally but erroneously intend to replace us with machines. I’m not even particularly concerned for my job: this is disastrous for our digital future, as our increased use of, reliance on, and integration with digital solutions that have no better critical functional analysis than an automated check.
The only real thing that testers produce are reports on the state of the product under test, for the business to decide if they are happy to release the product or not. This is what I do, and I try to do it to the best of my ability. If the business requires their testers to conform to bad testing practices, they will get poorly tested products, misleading metrics and a false sense of security, and it will cost more in time and issues. If your tester is informed, professional, and has integrity, but cannot change the way things are done, they will leave you. If you want to just wave a “tester” at the product to get the rubber stamp and don’t want real information to burst your bubble, why pay professional prices? An office temp could do it. If you really want a tester, who gives you good information, so you can have justified confidence in your product, let them do their job properly.
The established process wastes significant time and money on writing and/or running symbolic ritualistic checks, gives people meaningless or misleading numbers, and then allows people to naively believe that these prove something by which they could make reliable project decisions. This process I reject, as insufficient for good testing: wasteful, unprofessional, limiting, unethical, ignorant, dogmatic rubbish. And yet, this latter kind of testing is what the majority of potential employers look for, knowing no other way. If your organisation can afford to waste huge amounts of money on bad practices, bureaucratic rituals, shoddy products and customer dissatisfaction (eg, in my experience this means government, banks and insurance companies) and that is how your company operates, you don’t need or want a tester like me.
This essay is targeted at the managers of the digital development (IT) section of businesses that want to make or save money. A professional tester should be sceptical, especially of their own perceptions, asking questions, learning new test techniques and skills, efficiently targeting their testing to the areas of most risk, owning and learning from their mistakes, working to inform the stakeholders about risks, uncertainty, what kind of bugs they are getting, what other issues the project faces, the state of the product, and how the tester know their testing is sufficient for the needs of the business. If this is what your business wants, then you actually do want a tester. I am in awe of the people who represent this kind of testing: I am inspired, challenged, and constantly reminded that there are brilliant thinkers in the world, who work with vast integrity, and I want to be counted amongst them.