Software Testing Basics, Part 1
Software Testing Principles
June 8, 2024
The problem is not that testing is the bottleneck. The problem is that you don’t know what’s in the bottle. That’s a problem that testing addresses.
Contents
Who am I?
I'm a recent grad of a software bootcamp. Although I picked up programming in high school, I didn't write a line of code from 2005 to 2023.
So what makes my blog worth reading? There's plenty of sources out there with more claim to authority on the topic of software testing. Why should anyone read this one?
Well, first of all, this is for me. I've been in position my times to do provide education and training, and I enjoy it. Beyond that, I've never learned a topic half as well as I do when I have to teach it to someone. This blog gives me a good way to practice what I learn and hopefully help others along the ride with me.
For the reader, this can be another perspective on the topic. Plenty of resources out there are an authority on software testing, but how many of them can remember what it's like to not know what's going on? As I learn the topic, we can bumble through our mistakes together. If anyone disagrees with my takes, whether you're a peer or an experienced software tester, I hope you will let me know in the comments below.
So with all that out of the way, let's talk about testing.
Software Testing Principles
- Do it early
- Show the presence of defects
- Exhaustive testing is not possible
- Defect clustering
- Pesticide Paradox
- Testing is Context Dependent
- Absence of Errors Fallacy
Do It Early
Sit down right now and type until you have all the code for an operating system. Maybe don't shoot so high; let's start with a API for an e-commerce app. Make it simple, just add items to the cart and take payment at the end. What's the likelihood that when you run the code for the first time, for any of these projects, that it will be smooth and do what you wanted it to? Not high at all.
From simple typos, to "I forgot to implement that function," or "I didn't think it worked like THAT!" Every time you change the code, you're adding new failure points and possibly giving older code the opportunity newly fail. The fewer changes you make between tests, the easier it is to find any bugs that you've just introduced.
When we scale this up to business-level, catching bugs early is critical to the process. If we have to finish a feature, or a sprint, or a service, before testing it, then we end up writing a lot of code that will be deleted or rewritten when we do run into bugs. Testing each function as it gets written and added to the current build ensures that the function does what it's intended to and minimizes the code you have to check if the test failed. Then, testing each feature as it's incorporated to the current app ensures that the parts communicate well and accomplish the business purpose before pushing to prod where they might create costly errors. Saving rework saves time and many, and any bug we can chase down before it shows up in prod is a savings on the bottom line.
Even in my limited experience, I've had occasion to apply this principle. My team and I want to build a chat app (Chat-12) and I designed the database schema. We have Users, who have Chats, which have Messages, and we want users to be able to move their chats around in the window. So I planned on giving each Chat an (x, y) location, and each Message a title. After implementing the backend API, I did a code review with an eye toward the frontend requirements, and realized that I had partially misconcieved the schema. My idea was that the displayed Chat name would be the name of it's first Message, but who wants to type a title for every chat message? Further, if there are multiple Users who "have" the same Chat, one user moving it's location would affect another user, which is not what was intended.
It's good that we found these issues now, before writing the frontend interface, but it could have been even better had I identified the issues before writing the CRUD interactions at all. Now we have another 5-10 hours of refactoring and testing ahead.
Show the Presence of Defects
A good pair of pruning shears can prune plants without a hitch. If you test them on all sorts of bushes, shrubs, and small branches, they'll likely pass every test. That does not mean good pruning shears are fit for use to mow your grass. That is because testing shows the present of defects, not the absence of errors.
This is why defined requirements are critical to software development. We need to identify the specific use cases and methods that should be implemented in or to accomplish the business purpose. Without that, the customer might be looking to cut the grass, and we hand him pruning shears while saying, "This tool is fully tested and 100% A-OK!"
Depending on the project, fully-specced requirements might not be possible. For example, if you're running an Agile shop, you'll want to implement smaller features and compose the app over iterations. That only means that each sprint ought to have well thought-out and comprehensive requirements so that each feature can be deployed "defect-free." Just remember that once the tests are passed and the feature is in production, there may still be lurking defects.
In my previous example, I'll admit my data schema didn't start with well-defined requirements. I knew what data we would need on the frontend, and I knew it should be split into User, Chat, and Message entities. But, clearly, I didn't have a clear enough definition of shape of the data that the frontend would require of the backend. I wrote the API along with passing tests, but it still wasn't fit for use. I was cutting grass with pruning shears.
Exhaustive Testing is Not Possible
Imagine you have have a website that tracks the user's daily water intake. It takes an integer from 1 - 100 for cups of water consumed. You probably could write a test case for every permutation of a single entry, you just need 100 for each of the acceptable integers (you should test the unnacceptable ones, but we'll come back to that).
Now what if the user wants to track intake as 1-5000ml? You could test all of those, but really it's a waste of time. And what if they then ask to enter decimal values? What if you have to accept decimal values to the hardware's limit of accuracy? We haven't even really tested all cases. In the first example, you should test that 0 and 101 are not allowed. But what about 50.5, 1002, and -4? You can't test every value. That is what is meant by exhaustive testing is not possible.
When I wrote my first suite of tests, I made sure to test each end of every logic branch in my functions. As I learned more, I'm finding additional methodical ways to define test cases, but at the beginning I was using Equivalence Partitioning and Boundary Value Analysis. By identifying where the logic of the function changes, we can test each side of the logic branch, and not worry so much about the (possibly) infinite other cases.
Defect Clustering
In a large and complicated come base, some of that code just retrieves and displays the "Welcome" blurb, or with the help of a built-in library, shows the current date and time. These aspects of the program have low coupling with other parts of the software, and simple logic paths that are easily accounted for. They are less likely to suffer from defects, while the complex and interdependent modules are like to experience several defects.
Therefore, we can save ourselves effort by focusing our testing energy on the areas likely to break. We might check that the clock works when it is first implemented, but after that, there is no reason to check it on future tests. By contrast, if our app uses several functions to calculate a shopping cart, authorize, and checkout, then we should focus our tests on this area. Not only is the subject (taking payment) critical to the business, but defects are likely to arise due to the complexity.
Pesticide Paradox
Once the feature is tested and launched, you'll want to keep running your tests in case later changes make the current features fail. On the other hand, the fact that these tests exist means that you may expect more defects in this area.
Why would passing tests indicate a danger of defects? Similar to the way insects and bacteria adapt to chemicals meant to destroy them, the fact that the tests are in place can cause the development process to find new ways to create a defect.
Imagine I have a function that returns "Good morning" when the clock is between 6AM and 12PM and a test that checks the result at 11:59AM and 12:01PM. The developer working on the feature didn't implement the function the first time, and your test found the defect. On the next sprint, the function has to be rewritten to include new functionality, and the dev is wary of making sure it returns "Good Morning" properly. Only now it always returns "Good Morning", and the negative case isn't tested. We have a pesticide-resistant defect here!
We have to continually look for new ways to test our software. Running the existing tests is a maintenance action, but every new feature should have time spent reconsidering the existing test suite.
Testing is Context Dependent
Obviously, you wouldn't use HTTP to test a database interaction layer. And you shouldn't need to interact with a database to test your routes. In the same way, you wouldn't copy-paste tests from the Youtube code and use it in your console RPG. Your test suite needs to be written for the app your are creating.
My first Birddex tests required mocking responses from the code inside the CRUD functions. My latest test suite replaces the live database connection with an in-memory database so that the function can perform real interactions while in a controlled environment. Even if I resolve that discrepancy, the apps have different tables with different data shapes, and even the Users tables don't contain the same data.
By necessity, tests are highly coupled with the tested app. Even when we do black-box testing, we need to make sure that the environment the app runs on is reflective of the production environment.
Absence of Errors Fallacy
Earlier we talked about how testing shows errors; it doesn't confirm the absence of errors. In philosophy, we say "you can't prove a negative." There is no test that can demonstrate that all failure states have been accounted for. As we discussed regarding exhaustive testing, it's not feasible to test every success case, either.
Therefore, it's a fallacy to accept a suite of passing tests and take it to be perfectly functional software. All software requires maintenance and the possibility of failure cannot be conclusively eliminated. Instead, we need to make effective use of the time we have for testing, and keep in mind that there will likely be fixes needed in the future.
Conclusion
Thank you for reading, and I hope you got something out of this. I've certainly learned the value of understanding and practicing the Software testing Principles. We need to do testing early and quickly to keep up with development. We need to make effective choices of test cases to execute in order to find the most defects with the least time and effort. Using the principles can maximize the impact of our testing suite.
Comments
Post a Comment