Compliance Testing (Conformance Testing) in Software Testing

 

Conformance Testing

Conformance Testing is a software testing technique used to certify that the software system complies with the standards and regulations as defined by IEEE, W3C or ETSI. The purpose of conformance testing is to determine how a system under test confirms to meet the individual requirements of a particular standard. Conformance Testing is also called Compliance Testing.

It may deal with some technical aspect but intentionally it includes:

• Performance
• Functions
• Robustness
• Interoperability
• Behavior of system

In this tutorial, you will learn-

• What is Conformance Testing?
• Types of Conformance Testing
• Why do we need Conformance Testing?
• What do we need to test?
• When and how to perform Conformance Testing?
• Conformance Testing Process/Program:
• Advantages of Conformance Testing:
• Disadvantages of Conformance Testing:
• Conformance Testing for Mobile System:

Types of Conformance Testing

Conformance Testing can be logical or physical, and it comprises following types of testing;

• Compliance Testing
• Load Testing
• Stress Testing
• Volume Testing

Conformance Testing

Why do we need Conformance Testing?

• To check for the system’s requirements fulfillment
• To check whether the system documentation is complete with needful
• To check the development, design and evaluation as per specifications

What do we need to test?

• The standards through which the implementation takes place
• The call of the system that is to be developed
• Scope of specifications
• Specification objectives

Conformance Testing is initiated by the management with total assurance about the team and their understanding of standards, specifications, and procedures.

To build an efficient application, standards and specifications should be clearly mentioned to avoid ambiguities. If not then conformance testing is itself useful to take the necessary steps to make it relevant and reliable.

When and how to perform Conformance Testing?

• When we need to check the system’s overall consistency, completeness and correctness with respect to requirement specifications at that point conformance testing are performed
• There are several methodologies for performing it not much different from our Development Lifecycle Stages such as;
• Analyzing requirements specifications
• Preparing test plan
• Preparing test cases and its purpose
• Create documentation for prepared test designs
• We need to focus on some additional aspects of specifications such as subsets, which are generally of three types such as ‘Profile’, ‘Level’ and ‘Modules’.
• Profile is a subset of specifications which includes system’s functionality to satisfy requirements and dedicated to a particular group of users
• Levels are nested subset of specifications where Level1 indicates basic specifications and Level2 indicates Level1 + additional/new functionalities
• Modules are collection of significantly correlated standards and specifications

Conformance Testing Process

Conformance Testing Process or Compliance Testing Process is necessary to ensure efficiency, performance and to avoid future risks for a software application. An effective conformance testing includes following step by step process :

1. Analysing Standards and Specifications
2. Selecting Test tools and Test suite
3. Designing Testing Procedures
4. Checking necessary validations
5. Adapting relevant testing policies and certifications

Let us have a quick look at diagrammatic representation of Conformance/Compliance Testing Process for better understanding :

Compliance Testing Process

With the help of Use Case, this process will be easier to understand;

It is mentioned above that Conformance Testing occasionally called Compliance Testing but still there are some basic facts which make Conformance Testing different from Compliance Testing.

We will review the compare and contrast between both;

Conformance Testing	Compliance Testing
Conformance Testing is formal and precise way of testing standards	Compliance Testing is informal and less precise way of testing standards
Certification of conformance is applicable only to the operating system that possesses official Certification Authority	An operating system that provides a single API of POSIX (Portable Operating System Interface) are said to be Compliant
Conformance Testing is used for testing the system that provides full support to given standards	Compliance Testing is used for testing the system that provides support for some of the given standards

Advantages of Conformance Testing:

1. Assures the proper implementation of specifications
2. Assures portability and interoperability
3. Provides appropriate utilization of standards
4. Makes possible to assure that interfaces and functions are working as expected
5. Helps to identify the areas that are to be conformed with those which are not to be conformed such as syntax and semantics

Disadvantages of Conformance Testing:

Like other types of testing, Conformance Testing also has some tasks that are quite difficult to perform and may become a challenge. Some of them are listed below;

1. To perform testing for conformance efficiently, it needs to identify the class of the system need to be tested along with the suitable methodology
2. Categorizing specifications into Profiles, Levels, and Modules
3. Predefining values
4. Designing extensions, options and approaches to run the testing process successfully.

Conformance Testing: Critical Views:

Conformance Testing has some areas that need extra attention and knowledge for successful implementation;

1. Testing methodologies that are best suited
2. Selection of test tool that will automate the testing process
3. Clear understanding of Conformance Testing that shows only non-conformance instead of conformance
4. Designing and Implementing Conformance Testing Program
5. Defining how formally it should be performed
6. Policies and regulation

Conformance Testing for Mobile System:

Just like other types of Software Testing, Conformance Testing can also be done for Mobile systems. Once the system is assembled, it is checked for the use with prior to its deployment through Conformance test.

According to the type of networks such as GSM or CDMA, the tests are performed to achieve conformance and interoperability.
These tests may include;

• Protocol Testing
• Safety/Security Testing
• SIM card Testing
• Radio Frequency(RF) Testing
• Audio Tests
• Specific Absorption Tests

Example Use Case Diagram

Following Use Case Diagram shows how the conformance testing is performed for deployed mobile systems;

Furthermore, this can elaborated in the following manner for SIM Card Testing in two stages;

1. Password Validation

Password Validation

2. Making Call

Making Call

There can be two cases for a system such as
1. Call Succeed
2. Call Failed
Similarly, for Protocol Testing consider following Use Case;

Protocol Testing

Following Test Case Template will help you to understand how we can write test cases for Protocol Testing

Test ID	Test Scenario	Test Steps	Test Data	Expected Result	Actual Result	Status	Remarks
PT001	Protocol has no deadlock	Start with the communication process	_	Should establish communication successfully	Communication established successfully	Pass	_
PT002	“Ping” Command operates correctly for Timeout	Enter Ping command as ‘Ping -w’ in command prompt	Ping 1000	Should show message “Transmit Successful.”	The message shows “Transmit Successful.”	Pass	_
PT003	Count the no. of ‘echo’ requests	Enter Ping command as ‘Ping -n’ in command prompt	Ping 4	Should Start pinging 0.0.0.4 with 32 bytes data and Should show the message “Transmit Successful.”	Starts pinging 0.0.0.4 with 32 bytes data but shows the message “Transmit Failed.”	Fail	Semantic/Functional error
PT004	Any state can be reached from any state	Exit from a state and return to previous or next	_	Should successfully change the state	Successfully changes the state	Pass	_

These terms show that Conformance Testing is also useful to test the implementation of technical standards.
To bring uniformity into the test process, the test cases are being defined. Depending upon the type of networks, hundreds of test cases may be defined and executed to ensure the working of the mobile system as per requirements specification.

Summary:

• Conformance Testing is also called as Type Testing which is a formal way of testing
• It is used to determine whether the system follows specification of requirements
• It includes Stress Testing, Load Testing, Volume Testing and Compliance Testing
• It is also considered as Compliance Testing but there basic differences between both
• Conformance Testing Program, Testing Methodologies and Test Tools are some most critical areas of Conformance Testing
• Assurance for standard utilization and interoperability is achieved through Conformance Testing in Software Engineering.
• Conformance Testing is performed for Mobile Systems too.