Curiosity Killed the Computer

OK – the title might be overstating it a bit, but I’m a test analyst, and we get tired of hearing about how “automated testing” replaces the need for manual testers.

I recently saw this TED talk about machine deep learning.

I was somewhat surprised about how much the process for teaching machines was so akin to one of the fastest ways to teach expertise to humans – the exposure to 200-300 excellent examples in a short time, with a small amount of expert guidance, and seen in this presentation by Kathy Sierra.

I was thinking, it might not be so long after all, before machines are doing all our observational jobs, if they can learn pattern recognition like that (and they are more accurate and much faster and never forget…)

And then I saw this wonderful talk about aliens (or maybe not aliens). Even though number crunching is one of the primary uses for computers they could still miss evidence that humans could pick up, and more significant than that, they don’t see what we don’t program them to see. That is, we can see things that are not expected, but because we don’t expect an anomaly, we don’t make our computers see it. The pattern recognition a computer uses might throw a wobbly when confronted with a 3-wheeled car, or a wrecked car, but we would still see it is a car. Computers look for recognisable or definable characteristics. Humans characteristically have 2 arms, but we recognise that there are exceptions… We might be able to teach a computer the pattern recognition to sex chicks, but it won’t cope with a hermaphroditic or mutated example (unless we thought of that possibility ourselves, first).

Humans have observational skills that cannot yet even be explained, like the amateur astronomer that Bill Bryson talks about in “A Short History of Nearly Everything”. Reverend Robert Evans can see supernovas…

To understand what a feat this is, imagine a standard dining room table covered in a black tablecloth and someone throwing a handful of salt across it. The scattered grains can be thought of as a galaxy. Now imagine fifteen hundred more tables like the first one—enough to fill a Wal-Mart parking lot, say, or to make a single line two miles long—each with a random array of salt across it. Now add one grain of salt to any table and let Bob Evans walk among them. At a glance he will spot it. That grain of salt is the supernova

We may not all be exceptional, like this, but these examples show there is an argument yet to justify the inclusion of human observational skills on top of machine pattern recognition and deep learning even when the machines get good at it and are in common use. We learn new things when someone sees something they don’t expect, and says “WTF?” and follows the lead. Test wisdom says “the difference between what we know and what we need to know is why we test in the first place”. Machines might be able to take care of the things we know, but it will need a true AI before the need for complementary human analysis can be questioned.

Do You Really Want a Tester?

This essay is targeted at the managers of the digital development (IT) section of businesses that want to make or save money. It will come as a surprise to some people, that there is more than one “school” of testing. I have been exposed to two extremes, and through my 12 years of experience, and with personal integrity, have accepted one of them as supporting and representing the high standard of professionalism that I endeavour to offer my employer. This is probably not the one you are acquainted with. Allow me, please, to enlighten you.

Testing is a specialised cognitive skill. As with any such skills, from people management to oil painting, it cannot be reduced to a number of steps (and because of that it can’t be done by a machine – more about that later). A test case is not a test. At best, it is a test guide or idea. There is no way a “test case” could ever be written to represent all the analysis that a tester does, intrinsically, heuristically, and because it is impossible to do 100% testing there is no way that producing “comprehensive test cases” ever comes close to representing the test coverage or the actual analysis done. If a confirmation can be done by a machine, then it is a “check”, not a test. Testing involves mindful discerning analysis of the observations made during deliberate experimentation.

As a simplistic example of this, imagine a “test case” that confirms that Box C shows the factor of boxes A and B. If this “test case” was automated, we’d have to set the values of boxes A (6) and B (7) so we know that Box C will get the same results each time. A coded check would confirm that 42 shows in Box C: pass. A person checking would intrinsically notice if the result displays aligned properly, in the correct format, font and size and colour, in base 10, if it took a long time to display, or if Outlook crashed when the result displayed. A person testing could change Box B to 8 and see Box C does not change from 42 – turns out that was hard-coded so the auto-check would pass… There are infinite tests that could be done: even having different programs open at the same time on your computer can affect a function. A business is not going to ask a tester to test every possible combination of programs, browsers, operating systems, settings…

What would a “pass” on this test reveal? If it is done by a different tester in a slightly different way eg inputting Box B before content in Box A, or tabbing instead of mouse clicks is that a different test? It could get a different result. Also, if it fails the first 2 times, works on the third, is broken by a regression issue, fixed and is currently working at release, that usually shows as Count of Test Case = 1, Pass = 1, 100% Pass – and tells us nothing about the potential risks of the function.

At best, a passing test is a rumour of success… A failing test is an allegation of failure. – M Bolton

We should also keep in mind that “test cases” do not appear out of thin air. Since they are usually expected to be produced in advance of the tester getting access to the product, they are generally based around the testers’ other available oracles, for example the BRD. If these “test cases” are to accurately reflect the main functional requirements of the product, it means the design and requirements need to be produced to a high standard as well. It is not reasonable to expect better detail from test cases than the rest of the project documentation. Waterfall methodology is fundamentally flawed. Can any business really believe that they are going to design an entire 6 month project 100% correctly from the start? If the business is interested in making money and being at the cutting edge of digital development and consumer interests, they should be working iteratively from Minimum Viable Product and enhancing from there, and that does not support the argument for the creation of a complete suite of static “test cases” before testing even starts.

The expectation of and reliance on “test cases” and automation in many digital workplaces indicates a fundamental lack of understanding of what testing actually is. This is commonly enabled and enforced by members of the test community itself, many of whom have never questioned the inconsistencies inherent in the establishment process (which shows an indicative lack of discernment for people who should be professional sceptics). I have heard various dogma about why people think we need test cases. I say “dogma” because, if they were arguments, they would be dropped when the people who believed them were given a compelling rational counter-argument that clearly disproved the belief…

  • “We have written test cases so new testers know how to test a system.”
    • First, test cases are not designed to teach anything. They don’t explain what user need the program is supposed to address or how the user is anticipated to use it, or what the out of the various functions mean to the user, or why the designer decided to choose one function over a different one, or the business priorities with respect to the program.
    • Second, you would think that, if you’ve hired an experienced professional tester, they’d already know how to test. All they need to do is learn what the program is for and how to use it, experiment, and question to start getting a feel for the product and the project.
    • Third, if you want to teach someone to test (ie use critical thinking, structured analysis, intuitive leaps and intelligent informed reasoning supporting their reports), you have to encourage them to think far beyond learning to follow a set of mindless, repetitive steps that can in fact get in the way of analytical thinking.
  • “… so we can get test coverage and test progression metrics.”
    • It is not true that all tests are equal. A fail in one critical path is worth far more than 10 typos in page content, yet each test case is counted equally when “pass rates” are calculated.
    • Does the metric really reflect the value you want to measure? 80% of the test cases may be run, but that’s because they were easy – the remaining 20% may take 80% of the time, so this doesn’t show how close we are to completion.
    • There are many more such inconsistencies with this idea. I recommend reading this article here before considering reducing tests and test activities to numbers.
  • “… so we can prove coverage and compliance.”
    • As I pointed out above – we will never have complete coverage, and any set of documentation that says otherwise is false.
    • This also assumes that every tester follows every written step, every time they do the tests, which I don’t think is a universal truth. It seems to me far more useful to inform the test professional of the details if you feel you are somehow obliged to run specific tests on certain occasions, and allow them to determine the constraints of that requirement.
  • “…to have a set of “standardised” regression tests.”
    • No tester will ever follow a set of instructions the same way twice, and a different tester will behave significantly different with the same instructions, so the idea of standardisation is completely illogical, unless the checks are run by a machine in an otherwise closed environment.
    • Even the best testers will begin to miss things when asked to mindlessly follow the same set of checks and tests every time the program is updated. Intelligent people get bored with repetition, and rush things, to get it over with. It is not testing if your mind is not engaged.
    • A (usually) truncated set of tests such as this will obfuscate the areas of change and real risk. Instead of a wild shot-gun blast of tests in the general direction of the potentially affected areas, it is far more efficient and effective to work with the developers to understand what code was changed, and where that code is used, and then target the testing to the areas of most risk. Regression defects may appear random on the surface, but they always have a cause. Working with the devs also helps them understand how they have to think about more than just the function under work, so they write better code.
  • “…so someone knows where to pick up if your tester gets hit by a bus.”
    • This is the best reason I have seen, but still, if your tester has been giving clear and informative progress and issue reports along the way, it should be obvious what they have done, and any good tester should be able to see what yet needs to be done. Progress reports that say “Of 33 test cases, I have run 17, and 5 of these have failed” do not actually tell a replacement tester what testing has been done.

I was appalled to see, in the results of a job search, the term “Automated tester” being commonly used. Gone is the facade of testers being “test automation engineers” – they now say explicitly that testing can be done by a robot. We have cheapened our profession so much, allowing people to believe that what we do can be reduced to a series of steps, and our valuable analysis is just a number, that they now rationally but erroneously intend to replace us with machines. I’m not even particularly concerned for my job: this is disastrous for our digital future, as our increased use of, reliance on, and integration with digital solutions that have no better critical functional analysis than an automated check.

The only real thing that testers produce are reports on the state of the product under test, for the business to decide if they are happy to release the product or not. This is what I do, and I try to do it to the best of my ability. If the business requires their testers to conform to bad testing practices, they will get poorly tested products, misleading metrics and a false sense of security, and it will cost more in time and issues. If your tester is informed, professional, and has integrity, but cannot change the way things are done, they will leave you. If you want to just wave a “tester” at the product to get the rubber stamp and don’t want real information to burst your bubble, why pay professional prices? An office temp could do it. If you really want a tester, who gives you good information, so you can have justified confidence in your product, let them do their job properly.

The established process wastes significant time and money on writing and/or running symbolic ritualistic checks, gives people meaningless or misleading numbers,  and then allows people to naively believe that these prove something by which they could make reliable project decisions. This process I reject, as insufficient for good testing: wasteful, unprofessional, limiting, unethical, ignorant, dogmatic rubbish. And yet, this latter kind of testing is what the majority of potential employers look for, knowing no other way. If your organisation can afford to waste huge amounts of money on bad practices, bureaucratic rituals, shoddy products and customer dissatisfaction (eg, in my experience this means government, banks and insurance companies) and that is how your company operates, you don’t need or want a tester like me.

This essay is targeted at the managers of the digital development (IT) section of businesses that want to make or save money. A professional tester should be sceptical, especially of their own perceptions, asking questions, learning new test techniques and skills, efficiently targeting their testing to the areas of most risk, owning and learning from their mistakes, working to inform the stakeholders about risks, uncertainty, what kind of bugs they are getting, what other issues the project faces, the state of the product, and how the tester know their testing is sufficient for the needs of the business. If this is what your business wants, then you actually do want a tester. I am in awe of the people who represent this kind of testing: I am inspired, challenged, and constantly reminded that there are brilliant thinkers in the world, who work with vast integrity, and I want to be counted amongst them.

Anarchy for Beginners

Anarchy: absence of government and absolute freedom of the individual, regarded as a political ideal. (Google definition)

I do not use the word anarchy to describe chaos. That’s a different thing.

I am an anarchist. In other words, I am a romantic, a political idealist, and I believe in liberty, as defined in J.S Mill’s On Liberty, when the only time a person has the right to infringe another’s liberty is in their self defence, or the defence of their community, and only then if the interference is not worse than the offence they are trying to stop.

In my idealistic dreams, we have free speech, and we all respect that, because we know that, even if someone says something that is wrong, misleading or offensive, we need to hear it so we can keep alive the proofs, and not rely on unsupported dogma and fashionable ideas to stop the misinformation and rhetoric. When I was about 20 I met someone who claimed he believed the world was flat. All the trite “proofs” I had that it was, in fact, spherical, he countered happily – no doubt he had heard them before. Because I didn’t know how to prove the Earth was spherical, all I had was the insufficient dogma of my education.

This was a defining moment for me. If we care about the truth of something, we have to know that truth for ourselves, and if it is something that someone else would argue against, we had better know their arguments and the answer for them in advance, or how could we discredit their arguments to our own satisfaction? How could we not allow ourselves to be rationally swayed by their more compelling arguments? There’s a big difference between wanting to believe something is true, and knowing it is true. As Michael Freedman used to say: “Believe nothing, question everything. Discover the truth for yourself.” I studied ethics to know what really made an action right or wrong, and I’m employed as a professional sceptic: a test analyst. (If you don’t believe that’s what a tester is, watch their faces when a dev says “it should work like this…”)

Of course, in reality, this is not how free speech works. Humans tend to be:

  • gullible or lazy – we’ll let others tell us how the world works. This is the legacy of a spoon-fed “education”
  • biased, prejudiced, and disinclined to question things we want to believe
  • emotional and easily stirred to irrational responses
  • attuned to sensationalism, self-interest and scaremongering

So anything said could be believed without question or not properly debated in an equal and open forum.

Anarchy is also about freedom of action, with the qualifier above that people can rightly stop you if your actions interfere with their actions enough. No-one has any right to say what you can and can’t do with your adult self past that. I’ll dress how I like, have sex and form relationships with whomever I want (assuming they want to, as well). I will choose what I read or watch. I’ll imbibe what I want and take my own risks, and from there, I will accept the consequences of these actions. These consequences might include the risk of illness or injury, isolating myself from other people, failure and ruin, but if that’s the risk I’m willing to take, it’s my choice to make. There are complications, of course, like if you have dependants, and when exactly someone is considered adult enough to take those risks, but I dare say that by now I might be considered enough in command of my own faculties to make those decisions.

The consequences of enforcing safety through laws and standards is a vicious cycle of a blithely careless population who do not take responsibility for their own injuries, and a more and more restrictive set of laws to protect them from their carelessness, since no-one else wants to be responsible for them, either. These laws are called “patronising”, but I generally call them “matronising”, as they are far more like how mothers treat their children, keeping them anxious, fearful and dependant for as long as possible. Another term I use is “smother love”. You do not show you care by disenfranchising another of their personal responsibility. You instead make them dependent, needy, and define, reinforce and validate their weaknesses.

Once again, in real life this gets complicated. We live in groups with various rules and laws against our personal autonomy, and some ability to hurt us in the enforcement, and about the most choice we have is which groups we allow to tell us what to do, and whether we obey them or not. I choose to subject myself to the laws of NZ, for instance, and because I don’t want to spend time in prison I don’t get caught breaking them. I have a t-shirt that says “No-one rules, if no-one obeys” and that is always an option. Conscientious objectors have existed throughout human history, and paid the price for their disobedience, but they have also been the ones that have brought about change in the rules.

To me, the strongest weapon of the anarchist is not freedom of speech or action, but the freedom to NOT do what someone expects or orders. I recommend reading And Then There Were None by Eric Frank Russell. It changed my life. The anarchist community of the story had an unbeatable weapon: each had a plaque with the words “I won’t” pressed into it. We, in our Western communities at least, are very good at imposing our will on others. We will order, ask or expect people to do things with no right to do so, no contract or agreement, and no thought that they might say “no”, and we commonly take offence, or are at least surprised if they actually refuse. We also commonly don’t even acknowledge when they do comply with our illegitimate requests. We task each other to establish dominance, we take each other for granted all the time, we assume that they will want what we want and do what we want. Our society accepts this pervasive coerciveness as normal, without comment. We expect polite compliance to our impolite impositions. There are several breeds of people who can’t say “no”, all with their own pathology, and these come about because we want to please and be appreciated by others, and to avoid awkwardness and confrontation.

Learning to say “I won’t” was one of the best things I have done for myself. For instance, unless I choose to for my own reasons, I won’t:

  • Take responsibility for another person’s mistakes
  • Tolerate being treated rudely, unfairly, with disregard or contempt
  • Allow myself to be coerced into a decision or action just because someone has acted in a nice or helpful manner to me in the past. This is a biggie: our ideas of reciprocity don’t apparently allow us to choose how we reciprocate, if it’s not negotiated in advance. Nowadays, unless I know someone very well, and can trust they won’t expect me to do something I don’t want to do, I will ask someone if their contribution is a gift, in which case I will accept it as such, with nothing owed, or I will make explicit my intention to pay them and negotiate the terms to nullify the debt
  • Allow myself to be puppetted by or bullied through my emotional tendencies, for example, desire to be accepted, polite or kind, or by my principles, such as honesty, generosity, friendship or integrity. If someone is using these things to get me to do something that I don’t want to do I will drop it, and pick it up again later, when it’s not longer poisonous. We only keep these things for our self respect, and if they cost our self respect to keep, they are no longer useful. A simple example is when the sleazy Uncle wants to give you a friendly shoulder massage. Are you polite, or do you say no?

Anarchists can have leaders, but they don’t have rulers. We can accept that someone is an authority on some topic, but deny that it gives them authority over us. We choose to cooperate because it is in our own best interests, as social creatures, but we don’t have to. Anarchists should not give orders where they have no right to do so, or dictate how a person should live. They should say “please” when they make a request, and “thank you” after, to show it is a request, and they should accept “no” to a request with no displeasure. They should not coerce, impose or assume compliance. I like being an anarchist – it makes me both free, and a better person to other people.