One tests to get a product without any vulnerabilities. And of course, the result is what matters! But how important is the process?
For me, the process is what happens: something begins, then stuff happens and at some point, it ends. That’s a process. I think most of the time, when people are talking about “the process” they’re really talking about a process model.
Over the years people have described dozens of process models, but there have been as many processes as there have been software development projects. The most important thing to remember is that we shouldn’t be mistaking the process model for the process.
Why is that?
Because whenever anybody talks about how things are going, or how we do things around here, it’s always under-described. The process model is always a kind of idealization or story of what the actual process is.
Sometimes people can make a mistake by failing to recognize that actual processes tend to be messy. They’re complicated, human, full of errors, nervousness, and other social forces. Then people claim that these social forces are wrecking the process model.
But let’s remember is that software development is fundamentally a human activity. It’s subject to all kinds of human influences and social forces. Some of those are unhelpful, for sure. But successful projects are successful because of social forces too.
For me, it’s important not to treat the process model as a religious ritual to be followed rigidly. Think of a process model as a heuristic, a fallible way of solving a problem, that we apply to help guide us and steer us. That means that we must retain the skill, and judgment, and wisdom to stop applying the process model when it isn’t helping.
How do you spice up your own testing process?
For me, the most fun is working with another tester, who shares the same enthusiasm for learning about the product, but who is temperamentally different from the way I am. For instance, I love testing with my principal colleague James Bach. When we’re working together, often we start with me observing and taking notes and commenting while he’s doing the driving.
Then if we notice a problem, I might investigate it while he goes off and does some research on some aspect of what we’re seeing. I remark on some data or behavior that we might examine; then James gets an idea to write a bit of code to examine it more deeply or extensively. We help each other along, but we also critique each other’s observations and perceptions.
When the effort is shared it’s much more engaging, because each person sees things the other might miss, and each person can cover the gaps in each other’s skills and preferences. It’s a lot like playing music. I play Irish traditional music for fun. I play on my own, for my own amusement, but it’s much more enjoyable to play with other people. The sound is more full, but the cool part is that we provide energy and inspiration for each other.
How do you think testers should look at their products?
One big part of the job of the tester is to look at the things that appear to be simple and see the complexity underneath. Another thing that testers must do to do is to see the things that seem horrendously complex, and recognize underlying patterns of simplicity.
Alternating between perspectives, between focused and defocused views, is a skill that can be learned, trained, and developed. Here’s another alternation: taking a “what happens” view, a behavioral view of the product, and then taking a “what happens to stuff” view — a data-oriented view. Another alternation: moving between a functional perspective — “what does the product do? What can it do?” — and an operational perspective — “how do people actually use the product in their settings?”
There’s also an alternation to be done between the builder’s, insider’s view of the product, and the outsider’s view. That doesn’t just mean testing like a user, although that’s a good idea. It means testing from the perspective of anyone who’s not building the product, but who might be affected by it: technical support people, operations people, customer service people, documenters, the users’ managers, the users’ customers…
In the tester’s mind and activities, we have to be able to visit all of these different world views and see how they relate to the product and to each other.
If testing skills can be learned, trained, and developed, does that mean everyone can become a tester?
Everybody is a tester or can become one to some degree. I mean, we’re born testers. Little kids test all kinds of stuff – their toys, their food, their own abilities… and their parents’ patience! And the parents test their kids’ patience too! Testing is a natural human impulse. To some degree, that impulse gets suppressed in school and in everyday social life. No matter how curious I might be, I don’t test the breaking strength of wine glasses at dinner! (Not intentionally, anyway!)
Being a professional tester is a specialty that requires time, determination, and practice to develop. Not everyone is suited to that. Not every person has the patience to look at software deeply and critically in a search for problems.
Most people want to believe and hope that the product is okay, and that it will be good enough. The tester’s job is to doubt that; it’s our job to be professional doubters. We are professionally uncertain when everyone around us is sure of something. That makes testing a socially challenging job.
People don’t usually consider software testing to be a creative or artsy profession! But isn’t software testing an art?
Well, I’d say so. Software testing is a deeply creative process, but only if you’re doing it right!
There’s a chapter in Computer Programming Fundamentals that’s one of the first real essays on software testing. In that chapter, Jerry Weinberg points out that the real challenge for testing software is not to prove repeatability or reliability, but to test for adaptability. He says that in order to do that, and to find problems, we need to develop a suspicious nature and a lively imagination. That’s pretty artsy, don’t you agree?
You said that sometimes you must break rules to test things, to work around constraints, and to discover problems that matter. What is your rule for “breaking the rules”?
If I had a rule for breaking the rules, I’d break it! Testing is not about necessarily breaking rules, but about breaking conventions and preconceptions. Sometimes that means dropping your ideas of what’s supposed to be.
For instance, here’s a convention: sometimes people talk about “positive testing” and “negative testing”․ In Rapid Software Testing, we’d say that a “positive test” is one in which we honor every explicitly declared assumption or requirement for the product; then we try to perform actions, walking through a kind of idealized, “normal” flow through a task or process. Maybe we’ll discover a problem as we doing that, or maybe we won’t, but here’s the thing: people act “normally” or “typically” for very long in the real world.
Even if they don’t stray from the “normal” path deliberately, they can get distracted, or surprised, or confused, or impatient. Intentionally or not, they will eventually do something that is outside of the path that the builders imagined or intended.
I can’t read anybody’s mind, and I don’t know about anyone’s intention, I cannot predict the future. But what I can do is I can say “All right, what if somebody did something that is not in the mental model of how things are supposed to go?”, and then I can deliberately choose to violate at least one explicitly declared requirement or condition. You could call that a negative test, I suppose; but we prefer to call that a test. Testing well requires us to challenge the product, and our assumptions about how it’s going to be used.