Testing
Setting the Stage
Testing can either be automated or manual.
In DevOps, the goal is that you have as much automated testing as possible. Therefore, the primary focus of this package is automated testing.
Nonetheless, some manual testing will inevitably be necessary. Therefore, manual testing strategies are provided at the end.
Definition
Automated testing begins with a defined test plan, comprised of required features and previously discovered bugs. The test plan is the contract of what the software is expected to deliver.
A consistent set of inputs is used for each test. This allows multiple programmers to execute the tests in the exact same way.
The tests are run automatically - not by hand (manually) by each engineer.
If a bug is discovered, a new test is written to catch the bug so that once it is fixed, you will always be checking to confirm it does not return in the future.
Types of Tests
Benefits of Automated Testing
| Benefit | Details | |
|---|---|---|
| Save time |  |
• Running automated tests is MUCH faster than manual testing. • Bugs that are found later in development are more difficult and take longer to fix. |
| Save money |  |
• Saving engineering time saves you money! |
| Quality |  |
• End up with fewer bugs in production code (studies have shown between 40%-90% less bugs[source]!) • New tests are written for each new bug that is found. This ensures that future features/bugfixes don’t break existing functionality. • Existing tests are repeatably executed with every release. • Encourages the design of more modular / testable code. |
| Maintainability |  |
Increase the testability, readability, and adaptability of the code: • Testability: create thorough and reproducible tests • Readability: tests become a record of bugs and features. Makes it easier for other engineers to understand the code • Adaptability: promotes the creation of modular code |
Addressing the Arguments Against Automated Testing
| Argument | Counterargument | |
|---|---|---|
| Creating unit tests is too time consuming |  |
The result is higher quality software and less bugs in the field. The time you spend writing automated tests would end up being spent (and then some!) during manual testing anyway |
| You can’t write the test until you know the design |  |
If you don’t know enough about the requirements to write a test, then you don’t have enough information to write quality code. We must avoid the tendency to start coding before a proper plan is in place |
| Unit testing is hard! |  |
It can be difficult at first, but like most things it gets easier the more you do it • It is definitely difficult if you wait to add unit tests at the end (don’t do that!) • This could also be an indication that your design is not easily testable (too much functionality in a single function / function block / task, etc) • Not everything is able to be unit tested. But in those cases, you will rely on manual testing |
| I don’t know how! |  |
Hence why this DevOps package exists! |