Introduction

Let’s face it; testing is tedious. But ask passengers of a Boeing 737 class airline if software testing is tedious (Boeing 737 MAX groundings). Or ask a banker who lost millions due to a security breach of an untested API. Faulty software can cause enormous costs to organizations, individuals,  and even lead to loss of life. APIs are no different; faulty APIs can be disastrous to your organization and organizations using your API. Finding and mitigating API faults through careful API testing is crucial to developing robust APIs.

In this article, we discuss the theory and techniques behind API testing, and we discuss how RapidAPI for Teams can help organizations test internal and external APIs. This article is also careful to distinguish between testing APIs an organization consumes compared to an API an organization produces. The difference, though often overlooked, has serious implications for when you test an API and how you mitigate errors.

• Testing APIs your organization consumes is different than testing APIs your organization creates. 

You should test APIs your organization consumes early in the development lifecycle as errors will affect how your organization develops client software for an API. Testing APIs your organization produces are usually tested later in the development lifecycle.

Finally, although there are many different ways to test, here, we only consider black-box testing. Moreover, we assume the functional testing typically conducted by a tester rather than a developer. A tester tests via interaction with the system and compares the expected results with actual results. The tester often knows little to nothing about the system’s internal functionality; he or she only understands what it is supposed to do in response to some input or action is taken via a user interface.  We limit our discussion to this type of functional testing, although we replace the user interface with API endpoints.

• Wikipedia article on testing (Software testing).
• White-box vs. Blackbox testing (Differences between Black Box and White Box Testing).

A public API exposed to end-users is a product, and you should test it thoroughly. APIs should not be relegated to informal testing by developers because it lacks a user interface. Testers should test APIs thoroughly by testers with formal testing plans. RapidAPI for Teams helps facilitate this testing.

Functional Testing

Functional testing is a type of testing where testers treat a system’s internals as a black-box. Ideally, the person testing has little to no knowledge of the application’s internal workings. Often people that perform the functional tests do not know how to program. Functional testing has three primary goals, ensure the product has no bugs, ensure the product meets requirements, and ensure the product does not break after being modified (regression testing).

• Explanation of functional testing on Wikipedia (Functional testing).

Functional testing should follow a testing plan and should test the application from the viewpoint of the customer. Wikipedia defines the steps taken when performing functional testing as follows.

1. The identification of functions that the software is expected to perform
2. The creation of input data based on the function’s specifications
3. The determination of output based on the function’s specifications
4. The execution of the test case
5. The comparison of actual and expected outputs
6. To check whether the application works as per the customer’s need.

As the six steps from Wikipedia illustrate, functional testing is a hands-on type of testing performed by a person interacting with a software system. Of course, this “interaction” begs the question; if an API has no user interface to test, how does one perform functional testing (again, assuming a tester with little programming knowledge)? The answer to this question is that the API is the external interface a tester “interacts” with and not a UI; moreover, using RapidAPI for Teams can help facilitate functional testing by testers and others with little programming knowledge. We return to this observation in a moment, but let’s first review REST APIs and RapidAPI for Teams.

Application Programming Interface (API)

An API is a set of functions and procedures that allow external applications to communicate and use features of another system without understanding that system’s internal functionality. Representational State Transfer (REST) is an architectural style for creating an API using HTTP. Typically, REST APIs accept and return data as strings formatted using Javascript Object Notation (JSON). REST and JSON combine offers an intuitive platform for sharing resources over the Internet using the standard HTTP protocol. REST APIs using JSON has become the de facto architecture for writing APIs. Most software these days use APIs to separate internal back-end functionality from the UI. This separation allows many different UIs to utilize the same back-end functionality.

• An overview describing and defining APIs on Wikipedia (Application programming interface).
• A comprehensive definition of REST APIs on RapidAPI (What is an API?).
• An overview of JSON on Wikipedia (JSON).
• Definition of JSON on RapidAPI glossary (What is JSON?).

Many organizations offer APIs as public APIs that other organizations can use. These APIs might be free or require a subscription. Using external APIs allow organizations to rapidly add functionality to a system without requiring the organization to code the functionality.

RapidAPI for Teams

As more organizations moved to modify internal APIs away from legacy standards such as Service Oriented Architecture (SOA) solutions to RESTful APIs, they also started to realize the business opportunities by making their APIs available to other organizations. And thus, the API economy was born. As the API economy grew, so did the complexities of managing APIs. Organizations had difficulties managing internal APIs, finding external APIs meeting the organization’s needs, and marketing external APIs the organization created. Organizations started to consider API management and governance, and how to participate in the new API economy more effectively.

Several organizations recognized the business opportunity in helping other organizations participate in the API economy. One such organization was RapidAPI, which realized that providing a single clearinghouse for finding, subscribing, testing, and accessing external APIs was a lucrative business model. RapidAPI also realized that helping organizations manage their APIs, both internal and external, presented a business opportunity.

RapidAPI for Teams offers a uniformed dashboard for an organization to define teams and manage APIs. It also provides an intuitive interface for publishing, subscribing, and managing APIs. One feature RapidAPI offers is an API dashboard; from this dashboard, developers can view an API’s schema, sample JSON responses, and a way to test the API directly from his or her dashboard.

• RapidAPI for teams frequently asked questions (RapidAPI for Teams [The Complete Tutorial + FAQs]).
• RapidAPI’s governance and management offerings (Create an Enterprise API Marketplace).

• Getting started using RapidAPI for Teams (Get Started with RapidAPI for Teams).
• API collaboration using RapidAPI for Teams (API Collaboration).

In this article, we limit our consideration to testing using RapidAPI for Teams. Refer to one of the referenced links for more information on RapidAPI for Teams to understand more on its functionality.

API Testing

API testing is often seen as a subset of integration testing, where you test the API in coordination with an application’s more extensive functionality. However, treating API testing as integration testing is outdated, as APIs no longer are merely internal strategies of decoupling an application. Today APIs are often considered the final product and not some component of a final product. Therefore, consider API testing as bonafide functional testing.

API testing ensures that expected messages received as an API’s response matches the actual response from the API. The reasons for API testing are the same as the reasons for testing any software. You need to ensure the API functions correctly (i.e., ensure there are no bugs), that the API meets the original functional requirements used to create the API, and to prevent errors from being introduced into the API as it is modified and iteratively improved.

Differences from Traditional Testing

Using an application’s User Interface (UI) is how testers usually test software when treating the application’s internals as a black-box. Although there are other types of testing, a tester testing an application by interacting with the UI is the ultimate determination of if an application works or does not. But a REST API has no such user interface available to a tester for testing. Instead, an API provides one or more endpoints that provide data or services to other applications. Because of this difference, we must also consider different techniques testers might use to test an API functionally.

Functional Testing an API

So how does a tester perform functional testing?  Admittedly, there are many testing tools on the market, such as SoapUI or Postman. Still, if we assume a technically naive tester (a realistic assumption), RapidAPI for Teams provides a convenient, non-technical interface for testing an API’s endpoints.  Consider the following screenshot of RapidAPI’s dashboard. Via the web interface, a tester can enter inputs, click a button, and view the results all within the dashboard.

Now, a tester will require a rudimentary HTTP response code and JSON format understanding, but other than these two prerequisites, little technical knowledge is required. There is no need for Curl, Postman, or any other tool to test an API, all that is needed is understanding how to use the RapidAPI dashboard.

Note that if a tester does have the technical knowledge, the dashboard offers code snippets in many different languages that a tester might use to build a testing script. For example, a tester could write a bash script that uses Curl to call an API.

RapidAPI for Teams

RapidAPI for Teams facilitates functional testing by providing a uniform interface to all developers/testers on a project. An administrator can add as many developers and testers as there are team members.

By adding users to RapidAPI for teams, all developers and testers are ensured the same interface to the API. Moreover, they all have the same API dashboard available for testing APIs. There is no need to instruct team members on where to locate an API or the endpoints the API provides; RapidAPI for teams provides this information. There is also no need to instruct members in using a tool such as Postman, a command-line tool such as Curl, or other tools. Instead, functional testing can be performed directly in the RapidAPI dashboard.

API Producer Compared to API Consumer

One difference between API testing compared to other types of software testing is the distinction between APIs your organization creates compared to APIs your organization consumes. Testing an API you consume is very different than testing an API you produce. If creating an API, then you can fix any bugs testing might uncover. When consuming an API, you cannot fix the API’s bugs; instead, you must modify your client software to accommodate the API bug. Also, testing should occur much sooner in a project’s lifecycle when consuming an API.

Producer

When writing an API, then you test after completing the API, which is usually relatively late in the software lifecycle. Of course, in today’s agile development world, you develop and test iteratively throughout the development lifecycle. However, you must wait until you have built something before you can start testing it. Moreover, the goal of testing is to find and fix bugs and to ensure software matches the software requirements. As an API producer, it is essential to produce a high-quality API, and the goal of testing is to ensure such quality.

Consumer

As a consumer of an external API, you have little to no control over errors exposed by testing that API. You can notify the API owner; however, ultimately, they decide if they wish to fix the error. The goal of testing is not to produce a high-quality API. The purpose of testing is to ensure the API functions as advertised before you start writing client code to the API.

You mitigate errors in an API you consume by modifying your application’s interface with that API and not the API itself. Changes to accommodate any API error should be isolated to the API consumer layer of your application, lest the API’s producer fixes the mistake, and renders your entire application inoperable.

When you test is also different between APIs you consume compared to an API you produce. When consuming an API, you must ensure the API works as described before you develop a client interface to that API. Developing a client interface involves generating data objects from an APIs schema, writing code to handle API exceptions, and creating code to send requests and receive responses from the API. Although useful APIs have well-written documentation and a valid JSON schema, you should still test the API. For example, as you will see later in this article, what if the response payload does not match the JSON schema?  If you had skipped testing, and immediately generated data objects from the JSON schema, then you would have numerous errors when trying to consume the API. Isolating these errors is easier if found before writing client software.

If writing client software to an API, then testing that API should occur much earlier in the project lifecycle than you would typically test your software. Imagine building a client application only to find out the API does not match your expectations. If the API proves buggy, you can find a more robust alternative. If only a few bugs, you can avoid bugs in your client software by accommodating the API’s bugs.

Functional Tests

Let’s now consider what to test when performing functional testing of an API. An API provides one or more endpoints that provide resources or services to an external client. Although it has no UI, it is testable via its endpoints. You test APIs via their endpoints. The following are what you should test when testing API endpoints.

• Test the API response payload,
• test the HTTP response codes when successful,
• test the HTTP response codes when unsuccessful.
• test the API resources state, and
• test the HTTP headers.

If you perform these functional tests, you can be assured API functions correctly according to specification.

Testing Response

APIs respond to requests. When sending a request to the API, you should test that the API returns the expected response. If the API has a JSON schema, then the response JSON should be validated against the schema. Otherwise, you can test the API more generally by ensuring the response is valid JSON, data returned in the response match what is specified by the documentation, and other “sanity checks” to ensure an API returns what you expect.

If you are creating an API, then testing the response against the expected response ensures you produce a robust API. If you are consuming an API, then ensuring the actual response matches the expected response allows you to write robust client software. If writing client software, always test an API before you begin writing the client software. It will save your organization from discovering the API’s bugs after the client code has already been written.

The easiest way to robustly test an API’s response is by validating it against a JSON schema.

JSON Schemas

Although optional, a JSON schema should be required. A JSON schema makes testing and consuming an API much easier by serving as a contract specifying the API’s response format. Clients can develop software from the JSON schema directly rather than trial and error by examining the actual JSON payload.

RapidAPI Example

Consider the Twinword Sentiment Analysis API on RapidAPI. It is an API for analyzing the sentiment of the text. Suppose we are writing client software that consumes this API. Before we begin developing a client to this API, we should test the API thoroughly.

The Sentiment Analysis API offers a JSON schema that we can use to validate the responses. So our first test will be to validate the response against the schema. You can conduct part of this test in the RapidAPI dashboard; however, to validate the response against the schema, we must turn to another tool that validates JSON using provided JSON schemas. Here we use the online JSON Schema Validator tool (JSON Schema Validator).

Before we can use the schema, we must modify it, as RapidAPI does not offer a “raw JSON” option to facilitate using the schema. Although in a real project, we would save the schema to an external file, here we copy and paste the schema directly into the JSON Schema Validator.

The RapidAPI dashboard presents the schema as a tree-view with annotations. When we cut and paste the content, the annotations, commas, and other formatting is lost. We can use the JSON Schema Validator to help us fix the schema. Note, a “raw JSON” option is a planned future feature for the RapidAPI dashboard and is currently under development.

After fixing the schema, we use the RapidAPI dashboard to get a response.

After copying the response into the JSON Schema Validator, we find an error. The schema specifies that the result_code is of type color, but in the response payload it is 200. As a result, the response does not validate against the schema.

Note that if we modified the response to return a color code as the result_code value, the response validates against the schema.

But modifying the response is not a valid way to fix the bug; instead, we must change the schema.

Now, this is where, as an API consumer, how you mitigate errors is different than that of an API producer; we cannot fix the bug. In this example, I notified Twinword, and they assured me they are fixing the bug. However, in the interim, you can use the modified schema to generate your data objects and develop your API client code. And if you isolate the client code from your broader application, you can easily change your API client once the bug is fixed.  But as this example illustrates, always test an API’s responses against the JSON schema if the schema is provided.

Testing HTTP Response Codes

HTTP response codes should match the response. Far too many APIs in production returns a Status 200 with an error message as the response payload. The following table summarizes the most common HTTP response codes.

When testing an API, test that the HTTP response code matches the response body. If creating an API, then upon finding an incorrect response code, fix the error. If consuming an API, then modify your client code accordingly to accommodate the mismatch. Accommodating this mismatch is essential, as many external APIs simply return a 200 status code with an error message in the response body.

• For a more comprehensive listing of HTTP status codes, refer to the following resource (HTTP Status Codes).
• The following is a good discussion of REST API status codes and error handling (REST API Error Codes 101).

Response Codes Importance

Returning the proper response code ensures your API is robust and usable by potential consumers to your API. Consider that most organizations using your API will use a framework when writing client code to your API. That framework often includes listeners that handle responses returning status codes indicating an error. These listeners simplify development, as errors are treated uniformly. For example, consider the following two examples, the first using node.js and the second using Spring’s HttpClientErrorException.

Node.js Example

Consider the following code snippet. It defines a callback when an error is received. However, what constitutes an error depends upon the HTTP status code. If an API returned a 200 Success status code, the callback would never be called. Instead, the developer would be required to parse the response for an error message. As an experienced developer can attest, error handling by parsing a response body is error-prone and brittle.

var request = https.get("https://teamtreehouse.com/" + username +".json", function (response) {
    console.log(response.statusCode);
});

request.on("error", function (error) {
    console.error(error.status);
});

Spring RestTemplate Example

Spring RestTemplate error handling is another example where incorrect HTTP status codes matter. When a RestTemplate receives a response, if it receives a status code indicating an error, then it throws an HttpClientErrorException (for 4xx status codes) or an HttpServerErrorException (for 5xx status codes). You can implement a ResponseErrorHandler to handle the response payload in this class.

@Component
public class MyRestTemplateResponseErrorHandler implements ResponseErrorHandler {
 
  @Override
  public boolean hasError(ClientHttpResponse httpResponse) throws IOException {
  }
 
  @Override
  public void handleError(ClientHttpResponse httpResponse) throws IOException {
  }
}

Then in your service class that calls the endpoint, you register the error handler.

@Service
public class MyConsumerService {
 
    private RestTemplate restTemplate;
 
    @Autowired
    public MyConsumerService(MyRestTemplateBuilder restTemplateBuilder) {
        RestTemplate restTemplate = restTemplateBuilder
          .errorHandler(new MyRestTemplateResponseErrorHandler())
          .build();
    }
 
    public Customer fetchCustomerById(String barId) {
        return restTemplate.getForObject("/customers/123", Customer.class);
    }
 
}

In the above snippet, if when fetching a Customer, the endpoint returns an HTTP status code indicating an error, MyRestTemplateResponseErrorHandler handles the error. There is no code parsing a response body looking for error messages.

Testing Response Codes Example

The node.js and Spring examples illustrate the importance that an API returns an appropriate response code when an error occurs.  However, all to frequently APIs in production return a 200 Ok response code with an error message in the response body. Consider the following example when testing the Meme Generator’s Generate meme endpoint in RapidAPI’s dashboard for the Meme Generator API.

I entered dont-exist as the meme parameter to ensure the API cannot find a meme with that name. After clicking Test Endpoint, I received an HTTP response code of  200 Success and an error message telling me the meme was not found. This discrepancy is a bug, as the proper response code is 404 Not Found not 200 Success. Neither the node.js code nor the Spring example would have handled the error. Instead, you would be required to modify the sample code to parse the response and, upon an error message, raise an exception.

If you were the author of the Meme Generator API, you could correct this bug. However, as your organization is subscribing to the API, you do not have the luxury of fixing the API. Instead, you must code the client to handle the error message, as the following pseudo-code illustrates.

if response.statusCode() == 200 {
  if message.instring("error message") {
    String errMessage = parseEror(message);
    throw new MyError(errMessage);
  }
}

Handling errors like this is error-prone, brittle, and just plain ugly. If you are writing an API, be sure to test the API and fix instances where it returns incorrect response codes. If consuming the API, then either find another more robust API that offers the same functionality or modify your client application error handling accordingly.

• When consuming an API, ensure all endpoints return the correct response codes, both when successful and in error.

RapidAPI for Teams makes testing an HTTP status code easy, as the example above illustrated. Without having to write client code, you can test the responses directly in your web browser using the API dashboard provided by RapidAPI. Never assume an API will return the correct response code; use the RapidAPI dashboard to test every endpoint with both positive and negative tests.

• Many APIs in production, even APIs created by rather large companies such as ForgeRock or Facebook, return a status code of 200 Ok even when an error occurs.

Testing API State

Testing the API state should consist of multiple steps, as the steps rely upon each other. Suppose you had an API that allowed creating, viewing, modifying, and deleting a widget.

Make State Tests Enterprise System Agnostic

Assume the back-end system is a black-box you cannot test.  Test a resource’s state solely through the API. Do not, for example, check a back-end datastore to ensure the resource was created. By assuming we can only test via the interface, the advice given here for testing the state of an API’s resources applies to both consumed APIs and APIs you produce.

Create

Test creating a new resource if an API allows creating resources.  After creating the resource, fetch the resource. If the POST request followed by the GET request succeeds, and the JSON widget resource matches the expected JSON, then the test passes. Otherwise, the test fails.

• Test resource creation by creating a resource and then fetching via the API.

If you create using POST, then trying to create an existing resource should fail. If you create using PUT, then the existing resource should be overwritten by the new resource, as PUT is idempotent while POST is not. See the following discussion for more detail (rest-put-vs-post). But the correct HTTP method to create a new resource is POST.

Update

After creating a new resource, test modifying the resource. After modifying the resource, fetch the resource and ensure the state was changed. Testing varies slightly depending upon if the update uses the PUT or the PATCH HTTP request methods.

• The difference between PUT and PATCH (What’s the Difference between PUT vs. PATCH?)

Put

API developers should use the  PUT HTTP command to update a resource by replacing the resource altogether. Although this distinction is not usually problematic, suppose you had a resource with an optional property, as the following code snippet illustrates.

POST http://nowhere.com/customers

{
  "id":"123",
  "name":"John",
  "age":"32"
}

PUT http://nowhere.com/customers

{
  "id":"123",
  "name":"Jerry"
}

GET http://nowhere.com/customers/{123}

{
  "id":"123",
  "name":"Jerry"
  "age":null
}

You first create the customer using POST and his age is 32.  You then replace the customer, with a new name and no age specified using the PUT command.  You then fetch the resource using the GET command, and you should receive the customer with his name changed and no age specified.  If you do not, then the test failed, and you should fix your API or modify your client application if using another organization’s API.

• Test that PUT updates an object by completely replacing the object.

If the PUT request followed by a GET request succeeds, and the JSON widget resource matches the expected JSON, the test passes. Otherwise, the test fails.

Patch

API developers should use the PATCH HTTP command to update a resource when partially updating a resource. When using PATCH, you are not replacing a resource. For example, consider the customer resource again, only now update the resource using PATCH.

POST http://nowhere.com/customers 

{ 
  "id":"123", 
  "name":"John", 
  "age":"32" 
} 

PATCH http://nowhere.com/customers 

{ 
  "id":"123", 
  "name":"Jerry" 
} 

GET http://nowhere.com/customers/{123} 

{ 
  "id":"123", 
  "name":"Jerry" 
  "age":"32" 
}

When updating the customer using PATCH, only the name was changed, as we omitted age. Because PATCH only modifies the updated properties and does not replace the entire resource, the customer’s age of 32 remained.

• Test that PATCH only updates a resource’s modified properties.

If the PATCH request, followed by a GET request succeeds, and the fetched resource reflects the changes, then the test succeeds.

Delete

After creating and/or updating a resource, delete the resource. Then attempt to fetch the resource. The request should fail. Unlike the other tests, this test is testing that you receive an appropriate HTTP error code and an error message formatted as JSON.

• Test that DELETE deletes a resource by attempting to GET the resource after deletion.

DELETE http://www.nowhere.com/customers/{123}

GET http://www.nowhere/com/customers/{123}

HTTP/1.1 404 NOT FOUND
Date: Wed, 21 July 2020 07:28:00 GMT
{
  "error":"Customer 123 does not exist."
}

Resource State Testing Examples

Assume we have developed an API to our enterprise widget application. Client applications can create, update, fetch, and delete a widget. Assume, as just mentioned, we cannot test the resource state by examining our underlying widget application; we can only test using the exposed API methods.

Assume a simple widget API as follows.

Also, assume the following simple JSON structure for a widget consisting of an id and color.

{
  "id":"123",
  "color":"red"
}

To adequately test the API, and remain enterprise system agnostic, we much test using multiple steps: the step that changes the resource state followed by a step fetching the resource.

Test HTTP Headers

HTTP response headers are part of a response that contains meta-data associated with the response. Headers contain information regarding a request’s authorization, caching, cookies, and other meta-data such as accept. RapidAPI’s dashboard allows testing a response header by displaying the header in the Results tab.

Example – Accept Header

The accept header specifies the response body format. Test to ensure the response’s format matches the response body format. If the response contains JSON, then the accept header should be application/json. If the response contains XML, then the accept header should be application/xml.

Although it might seem that all APIs would return an accept type header, but in reality, they do not. Although including an accept header is not critical, as its omission will not make client software using your API more challenging to use, but including the accept header makes your API more professional.

Test the headers returned by an API and ensure they match what you expect.

Summary

Testing an API should be as rigorous as testing any end-user software, especially if a public-facing API intended for public use. In today’s new API economy, API’s are as important as any desktop software. An API should be tested thoroughly. Ensure the JSON response payload validates against the JSON schema. Ensure the HTTP status codes correctly identify the response or failure. And for goodness sake, ensure no endpoint returns a 200 Ok response code with an error message as the JSON response payload. If an API changes a resource’s state, test each state change. Moreover, test that state change using only the API; treat the back-end (for instance, the database) as a black-box. Finally, do not leave testing to individual developers; have a testing plan and use testers.

RapidAPI for teams facilitates more robust API testing.