How to Use Anomaly Score API to Detect Anomalies in Your Data

The Anomaly Score API is a powerful tool that allows you to detect anomalies in your data quickly and accurately. In this tutorial, we will guide you through the process of initializing a model and using it to identify anomalies in a toy dataset.

Step 1: Initialization

To initialize the API, you need to determine the most descriptive set of data you own and understand which kind of anomaly you want to identify. As an example, if you add specific fields like dates or UIDs, the anomaly algorithm will add them as one dimension of interest, which might result in a sparse dataset. In a real-life scenario, you will extract that information from a database (i.e., transactions, network access, etc.) and convert the tuples into a concatenation of a simple string where fields are separated by commas and lines with line break characters.

For this tutorial, we will use a random generated CSV data with the function extracted(), which creates a sample CSV with integers, doubles, and strings from a set. We chose to use CSV as a transportation layer instead of JSON to reduce the parsing time to zero and increase throughput. If in your organization, you have multiple datasets with different columns, you can use the premium feature to initialize different anomaly engines bounded with that set of information.

    private static String extracted() {
        StringBuffer b = new StringBuffer();
        String [] v = {"SELL","BUY"};
        String [] v2 = {"upper", "upper-middle", "middle", "working","lowe"};
        for (int i=0;i<9;++i){
            for (int j=0;j<3;++j){
                b.append((int)(Math.random()*10));
                b.append(",");
            }
            b.append(v2[(int)(Math.random()*10%4)].equals("middle"));
            b.append(",");
            for (int j=0;j<5;++j){
                b.append((int)(Math.random()*10)+"."+(int)(Math.random()*10));
                b.append(",");
            }
            b.append(v2[(int)(Math.random()*10%5)]);
            b.append(",");
            b.append(v[(int)(Math.random()*10%2)]);
            b.append("\n");
        }
        return b.toString();
    }

The call to the service init will initialize the model starting from this dataset ( WARNING autogenerated code by Rapid API add extra quotation marks and unnecessary escapes)

        HttpRequest request = HttpRequest.newBuilder()
                 .uri(URI.create("https://anomaly-score.p.rapidapi.com/init"))
                 .header("X-RapidAPI-Key", "XXXXIp8pmnBI$NFZSh6c4YRMTB-umiDl")
                 .header("X-RapidAPI-Host", "anomaly-score.p.rapidapi.com")
                 .header("Content-Type", "text/plain")
                 .method("POST", HttpRequest.BodyPublishers.ofString(extracted()))
                 .build();
         HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
         System.out.println(response.body());

The response is a JSON contain some statistics information about the operation

  {result='OK', entries='10', elapsed_train_ms='29'}

Step 2 Evaluating

Once the model is train, in this case we are using the default one, but for premium account is possible to use multiple model accessible selecting them by URL parameter {class}. We can retrieve the evaluated anomaly score for a batch of entries passed as well as a CVS in the body of POST call.

    	 request = HttpRequest.newBuilder()
                 .uri(URI.create("https://anomaly-score.p.rapidapi.com/eval"))
                 .header("X-RapidAPI-Key", "XXXXIp8pmnBI$NFZSh6c4YRMTB-umiDl")
                 .header("X-RapidAPI-Host", "anomaly-score.p.rapidapi.com")
                 .header("Content-Type", "text/plain")
                 .method("POST", HttpRequest.BodyPublishers.ofString(extracted()))
                 .build();
         response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
         System.out.println(response.body());

The response will be an ordered list with the calculated scores for each line.

0.5408411942272195,0.5334701190612287,0.5805091711501308,0.5346272588802626,0.5254399635148846,0.5330849626111178,0.5742540767316547,0.5381137597050663,0.5175306833386774,0.6114952349362224

The same concept shown above can be collapsed to the /stream API which accept the CSV and the same content is used both for training and for retrieve the anomaly scores.

Follow the sample code in Java

package com.loreii.rtf;

import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Tests {

    public static void main(String[] args) throws IOException, InterruptedException {
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://anomaly-score.p.rapidapi.com/init"))
                .header("X-RapidAPI-Key", "XXXXIp8pmnBI$NFZSh6c4YRMTB-umiDl")
                .header("X-RapidAPI-Host", "anomaly-score.p.rapidapi.com")
                .header("Content-Type", "text/plain")
                .method("POST", HttpRequest.BodyPublishers.ofString(extracted()))
                .build();
        HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println(response.body());

        request = HttpRequest.newBuilder()
                .uri(URI.create("https://anomaly-score.p.rapidapi.com/eval"))
                .header("X-RapidAPI-Key", "XXXXIp8pmnBI$NFZSh6c4YRMTB-umiDl")
                .header("X-RapidAPI-Host", "anomaly-score.p.rapidapi.com")
                .header("Content-Type", "text/plain")
                .method("POST", HttpRequest.BodyPublishers.ofString(extracted()))
                .build();
        response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println(response.body());

    }

    private static String extracted() {
        StringBuffer b = new StringBuffer();
        String [] v = {"SELL","BUY"};
        String [] v2 = {"upper", "upper-middle", "middle", "working","lowe"};
        for (int i=0;i<9;++i){
            for (int j=0;j<3;++j){
                b.append((int)(Math.random()*10));
                b.append(",");
            }
            b.append(v2[(int)(Math.random()*10%4)].equals("middle"));
            b.append(",");
            for (int j=0;j<5;++j){
                b.append((int)(Math.random()*10)+"."+(int)(Math.random()*10));
                b.append(",");
            }
            b.append(v2[(int)(Math.random()*10%5)]);
            b.append(",");
            b.append(v[(int)(Math.random()*10%2)]);
            b.append("\n");
        }
        return b.toString();
    }

}