Service-oriented architectures (SOA) are an enabler for business-aligned IT systems: functions, which are covered so far by individual systems, become enterprise-wide and platform-spreading, being accessible in a standardized form. But SOA systems increase requirements and complexity of testing.

General quality goals such as performance, functionality and robustness are also applicable in testing SOA systems, however, with slightly different focus. SOA testing starts earlier in the development life cycle, sometimes very close to architecture and software development efforts and must address an incremental strategy for checking modules and entire systems. Additionally SOA systems require testing at different levels: service, bus and infrastructure level including validation of individual elements and their dynamic interaction. Especially the verification of a system, which has often no GUI, faces the testers with new requirements on planning and setting up a test suite. Another difference relies on “testing for reuse”, which addresses planning and running QA for future projects.

This paper describes an integrated approach and the experiences by testing several IT-systems over a period of 2.5 years at AOL Germany. In 2004 AOL decided to transform their traditional subscription-based ISP to a service-oriented architecture. These systems are related to various infrastructure services (e.g. registration services, billing services, account services) provided, e.g. in form of remote WebServices that have later been used by several countries for specific purposes like a self-care platform for prospects and customers or customer care tools.

The established testing methodology combines several quality techniques, which address special aspects like non-functional tests as well as the definition of criteria for conducting UAT on a non-test environment.

A central aspect of QA, of course, was “functional testing” to verify all of a product’s required features and functionality and to identify gaps between expected / specified system behaviour and actual system behaviour. Additionally we conducted “fault tolerance testing”, which tests infringement and manipulation of the interface, e.g. by sending malformed / bar / incorrect requests to the server.

A special test aspect is “performance & load testing”, which verifies that the system capacity in respect to load, performance, and scalability meets the requirements. This should also include requirements of planned future projects and applications using the same services and thus competing for the same resources. “Stress testing” is a form of testing which is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results.

Another important aspect is “security testing” including any test procedure or approach to unveil security holes in a system or environment.

Additionally we conducted further assessments, which contribute similarly to the overall assessment of the test items. The most relevant are the “quality of documentation”: each API is accompaniedby an API documentation; it is an integral part of the API. The quality of API documentation determines learnability and user acceptance and – in the end – determines productivity and effort for applicationsbased on the API.

Furthermore, “quality of API design” is a central aspect. Evaluating API designs includes many aspects: the choice and strength of design metaphors, the adequateness between problem complexityand solution complexity, the use of data types, the error processing model used (e.g., exceptions vs. error codes), the consistence with other services, the compliance with general Java standards (e.g., Java naming conventions for classes, methods, variables, and packages). “Standards support” is to be evaluated as well. Common standards, de-facto standards, as well as company standards are to be considered.

A special aspect of SOA is their “quality of service” (QoS). Mission-critical systems in enterprises address advanced requirements such as security, reliability, availability or transactions. For example, we conducted special "concurrency tests" to verify multi-user capabilities of the services.

This entire service testing effort took place as user acceptance testing (UAT) in a greater context of a project or programme. Goal was to ensure that all services meet all functional and non-functional requirements from a service user's point of view.

Of course, testing is mainly conducted in test environments – as a general concept of QA. But still, testing in production environment is additionally needed if there are requirements that particularly take effect in production environment (e.g., load, scalability, availability, security). Some issues, in addition, are applicable to any environment (e.g., installation issues, environment issues, network issues, configuration issues).

The test clients ran on the Java 2 platform (J2SE 1.5.x) on Windows XP on Intel Pentium CPU. All automated tests were performed using the following interfaces: WebServices (SOAP over HTTP).The test client(s) are based on Apache Axis as a SOAP client. Additionally we developed an own test framework extending JUnit for creating SOA-focussed test suites.