Data Transformers in scikit-learn. Part 2

 Today we continue with DataTransofmers but with potentially more useful example. What we will do today is enrich a dataframe with data collected online. Particularly, we'll enrich a dataset containing company names with market trasnactions data from Media-Market API:

Media-Obsessed Market API Documentation (remote-works-default) | RapidAPI

To use the API, you need a rapidAPI account. The API is free for reasonable use (it has a limitation on queries per hour)

Firstly, let's get our original dataframe (you can easily create your own, as you only need a list of tickers):

df_companies = pd.read_excel(os.path.join(path1,"companies to analyse.xlsx"))

The data then looks like this:

What you need to pay attention to: the data needs to have a column with tickers and the tickers are lowercase. If they are not, just transform using df_companies['Ticker'].str.lower().

Then let's get our json data. RapidApi will give you a snipper, which you can just copy to your code. It should look something like:

import requests

url = "https://media-obsessed-market.p.rapidapi.com/transaction/"

headers = {

    'x-rapidapi-host': "media-obsessed-market.p.rapidapi.com",

    'x-rapidapi-key': "...."

    }

response = requests.request("GET", url, headers=headers)

print(response.text)

Then you can also see how the data will look in a dataframe. Luckily, RapidAPi provides a compatible with pandas JSON, so all you need to do is:

data = json.loads(response.text)

df = pd.json_normalize(data['results'])

The dataframe then looks like this:

You should notice that you can immediately merge the first dataframe with the second on Ticker and company columns.

Now that we have all components together, all that is left is to assembler our JsonTransformer, in which we encapsulate all logic:

import json

import pandas as pd

class JsonTransformer(BaseEstimator, TransformerMixin):

    #the constructor

    '''setting the add_bedrooms_per_room to True helps us check if the hyperparameter is useful'''

    def __init__(self, drop_zero = True):

        self.drop_zero = drop_zero

    #estimator method

    def fit(self, X, y = None):

        return self

    #transformation

    def transform(self, X, y = None):

        url = "https://media-obsessed-market.p.rapidapi.com/transaction/"

        headers = {

            'x-rapidapi-host': "media-obsessed-market.p.rapidapi.com",

            'x-rapidapi-key': "679767132dmsh0b5fd5bb20571d1p137b11jsn2af344efe4f9"

            }

        response = requests.request("GET", url, headers=headers)

        data = json.loads(response.text)

        df = pd.json_normalize(data['results'])

        X = pd.merge(left=X, right=df, left_on="Ticker", right_on="company", how="left")

        if self.drop_zero:

            X=X.dropna()

        # if we are OK with a DataFrame format:

        return X

        #Otherwise

        #return X.values

Now test how it all works:

json_adder = JsonTransformer()

df_with_json = json_adder.transform(df_companies)     

And the result then looks like this (note, that we've intentionally left Pandas representation for human readability ):

Simple, isn't it?

And if you want to make your Json enricher more universal, you can also pass the column containing lowercase tickers into the constructor. That will allow you to enrich any dataset with transactions.