Test Management – Part 2: Methods and Tools

Testing yes, but how?

We test all day long! From the morning to the evening, our eyes are checking, so that you could mean, everyone could test. However, the methods and tools that lead to constructive results of tests are varied. There are so many points (who, when, where, how much, etc.) to consider when trying to set up a test case. How to proceed? What should you orient yourself towards? Regarding the requirements? The system? Your own experiences and expectations? Moreover, where should you store all the information and work with it? With the keyword "tools", some providers have established themselves, which I will discuss later.

Some methods

First, I make the claim that any action that expects a result can be called a test. Science classifies them into several categories. Based on the test pyramid from the first part of the Test Management blogpost series, the three most important processes can be classified as a kind of a box: black, white, and grey. A further 3 overarching methods are then described. A proper test coverage involves both positive (application works as expected) and negative (application can gracefully handle invalid input or unexpected user behavior) testing.

Black Box – Test (system oriented)

This functional and/or non-functional test technique is based on an analysis of the specification of a component or system [1]. The tester evaluates the functionality of the software under test without looking at the internal code structure. This can be applied from the beginning of the software development. That is why is it used in agile framework.

Figuratively speaking the test object is studied from the outside to make sure the behavior of the software is as expected. The test cases are based on the formal software requirements and specifications.

Furthermore, non-functional testing means to prove how well the system performs, if the involved interfaces communicate trouble-free or the look & feel of a report. For that, the tester passes input data to make sure whether the actual input matches the expected output. In contract to this, the expected report with expected values are functional aims.

White Box – Test (structure oriented)

This test technique is focused on the internal structure of a component or system [2]. The examination perspective of the test object is from the inside to make sure the internal sequences of the software run as expected.

This testing is done usually at the unit and integration level. It should ensure statement, branch and path coverage. A diagram that shows how modules call upon each other or how the business processes interact, can represent the tested structure. To design the test cases programming skills are beneficial. This method does not ensure, that all requirements where implemented, rather the lines of code are logically working.

An example could be the definition of a key indicator in the report. The required definition is: total of assets is the sum of the values from ‘Properties’ and ‘Equities’. The SQL-code can be “SELECT SUM(PROPERTIES + EQUITIES)

AS total_assets FROM v_investment”. The code seems ok and it will provide a result. A white box test will show, that these lines of code will give “0” as result if there is a null value in the database. But this was not the definition of the expected value.

Grey Box – Test (system oriented)

This test technique a mix of the both previous techniques, because the tester has limited knowledge about the process, what makes the method inefficient. A test is designed based on the knowledge of algorithm, architectures, internal states, or other high-level descriptions of the program behavior. To conduct this methodology, it is advisable to use automated software testing tools, because the test cases for grey box testing may include, GUI related, security related, database related, browser related, operational system related, and further topics.

As example for this test type take the SQL-code from the example below and face it with the definition/requirement. The Product Owner expects that the delivered values will be added together, so that the correct SQL-code should be: “SELECT SUM(NVL(PROPERTIES,0) + NVL(EQUITIES,0)) AS total_assets FROM v_investment”.

Negative - Test
Negative testing is carried out to find the faults that can be a reason of significant failures. It should stress the system beyond the limits for what it was built for. The tester should use it an incorrect manner for example to write letters in a date field. It is a kind of negative doing to show the positive result that the software does not work as designed. This technique is possible for all kind of box test.

Smoke-Test
This method does not underwrite that the software is 100% bug free, but it is used to ensure that little changes in code structure have not broken the basic functionality of the system. It belongs to a defined restricted check list of functional and non-functional test cases that is repeated after every deployment for example when a report is deployed from the integration to the live system to ensure all major parts are available. See some topics: report link is available, the value of total assets is the expected value, you are not able to type into a choice field.

Regression - Test
This test suite ensures whether the changes, enhancements or bug fixes in the previous version have a negative inflow on the software in the newly deployed version. It can be performed by retesting the whole software application or only the risky use cases, for example recheck within a firewall. Regression tests should be carried out regularly, at least after the deployment of a new version of the software.

Acceptance - Test
Last, but not least, a live deployment should be finished with a User Acceptance Test (UAT). Testers outside of the business department can test the software only by interpreting the requirements. To ensure, the product works like expected, domain knowledge is needed. End user should test it against the specification guide and enlarge the test plan with typical business cases like the report should show the total of the assets only if the delivered values are positive.

Some tools

The tools are just as varied as the methods. A distinction must be made between testing tools and test management tools. In this post the focus is on the testing tools. A ranking over all testing tools is difficult to do, because there are specialized tools for every test technique.

Testing tools are grouped according to their field of application: browser, acceptance, performance, mobile web apps, etc. Automation accelerates development which is essential to survive in the competitive market of the digital age.

A popular automated testing tool is by provided Selenium. Primarily this open source tool is used for automating web applications for testing purposes. There are three kinds of suites, depending on the purpose: WebDriver (browser-based regression automation suites and tests), IDE (it is a plug-in which creates scripts to aid in automation-aided exploratory testing) and Grid (distributing and running tests on several machines) [3]. This fully-featured tool for all automation requirements does not need programming knowledge, but it supports different programming languages like Java, Python and C#. Because of automation, testers don’t get bored doing the same tests and obviously there is less monetary investment for human resources. This tool can be run on different systems and browsers simultaneous.

An aspiring testing tool is Co.meta by Amvara [4]. It is based on Selenium WebDriver and uses also Python as programming language in the background. Usually it operates on a Linux virtual box, so that fast implementation and using is ensured. Testers can describe the test cases easily and define test plans which can be copied for further environments. The reporting site shows three statistics about test result, number of test cases and duration, so that no additional reporting tool is needed.

A suggestion for automated acceptance tests is the tool “cucumber”. It links specification and test documentation, so tests can be run and completed faster [5]. If Selenium supports more the Test-Driven Development (TDD), Cucumber employs the Behavior Driven Development (BDD). In BDD, whatever you write must go into Given-When-Then steps, so that the business department can be involved deeper and write their test cases by themselves. The scenario for the total sum example can be: GIVEN I get the right value – WHEN the code replaces the value ‘null’ into ‘zero’ – THEN the value for the total of assets is not equal to zero.

All tools are suggested for a performing agile software development and can be combined. The waterfall project constellation uses less automation regarding the testing pyramid as shown in the previous post.

Test management tools consolidate all information (no matter how tested) and support the subsequent reporting, for example with statistics and diagrams and will be carried out in the next blog post.

Summary and conclusions

All mentioned testing methods can be performed manually, half automated or automated. The black and the white method constitute together the minimum testing package to ensure a high quality of the software product. Using a single method is not recommended because it doesn’t provide the different views on the object/product/application. There are testing methods and techniques, but basically, they are simply named according to the focus of testing and are thus classified as an independent method. It would take 100 lines to list each of the test techniques. Find a consolidation of this information in the scheme below:

Part 1 of the Test management blog triology describes the doing of the tests. But we all know that we must be documented and reported the results. This topic will be handled in part 3 including suggestions for relevant tools.

[1] ISTQB Glossary ( https://glossary.istqb.org/en/search/black%20box%20 )
[2] ISTQB Glossary ( https://glossary.istqb.org/en/search/white%20box%20 )
[3] https://selenium.dev/
[4] https://cometa.amvara.consulting/
[5] https://cucumber.io/