Data Transformers in scikit-learn

 Data preparation in an ML or data science task is a critical part of successful analysis. Done wrong, this step can sabotage the whole process. However, looking around the web you will see that most of ML posts just process data in a dataframe, often even without creating a separate function for it. This is sometimes good for explaining what happens but in production it is often suboptimal, unreliable, and prone to errors.

To improve the process, most ML frameworks offer special data pipelining routines. Today we focus on data transformers of scikit-learn. As mentioned in the official documentation, the benefits of using data transformers are:

- performance (use grid search over parameters of all estimators in the pipeline),

- safety and encapsulation (avoid leaking statistics from training set in cross-validation), and 

- convenience (all data transformations can be packaged using transformers into one pipeline).

1. Problem

We will look at Nasdaq listed companies and prepare data for predicting daily spread based on the attributes of a company.

2. Get the data

First get the data. We can get Nasdaq data from Nasdaq screener tool:

 https://www.nasdaq.com/market-activity/stocks/screener

then read the data into a dataframe:

path1 = [replace with your path]

df = pd.read_csv(os.path.join(path1, "nasdaq.csv"))

We will get spread from rapidAPI:

url = "https://media-obsessed-market.p.rapidapi.com/transaction/"

headers = {

    'x-rapidapi-host': "media-obsessed-market.p.rapidapi.com",

    'x-rapidapi-key': [replace with your rapidAPI key]

    }

response = requests.request("GET", url, headers=headers)

Now we have some data in a pandas dataframe and some in response.text, which is a string containing json.

2.1. Remove extra fields

Let's start from the dataframe.

Here, we likely need to remove Name as it is meaningless most of the time. We need to encode Symbol to be able to refer to its human-readable interpretation later. We do not need Last Sale, Net Change, % Change, and Volume as those are time sensitive (but you can match the dates as well). We need to encode Country, Sector and Industry. Since there are many Nulls in IPO Year, we will impute those.

Of course, we could just go column by column making these changes but we will work towards doing it in one transformation. For simplicity, we'll select only 4 columns: Country, Sector, Market Cap, and IPO Year. Since the column indices are 6,9,5,7 correspondingly, we select:

df = df.values[:,[6,9,5,7]]

We'll need some extra imports:

from sklearn.compose import ColumnTransformer

from sklearn.impute import SimpleImputer

from sklearn.preprocessing import OneHotEncoder

Out of the selected column, we'll use SimpleImputer to get Median IPO Years to replace missing values:

SimpleImputer(strategy='median')

We'll use the same OneHotEncoders for country and sector:

OneHotEncoder(categories='auto')

Now, put it all together. The order of transformers defines the order in the output array.

Names of transformers, i.e. 'Country', 'Sector', 'IPO Year' are just used for convenience: it is possible to extract the transformers from the ColumnTransformer by name.

And the numbers in square brackets refer to the index of the column that will be transformed. Note, that the order has changed since we removed other columns.

Now, transform and get the results:

column_trans = ColumnTransformer(

[

 ('Country',OneHotEncoder(categories='auto'),[0]), 

 ('IPO Year', SimpleImputer(strategy='median'), [3]),

                  ('Sector',OneHotEncoder(categories='auto'),[1])],

remainder='passthrough')

res = column_trans.fit_transform(df)

res.toarray()[0]

You can notice that 2.0190000e+03 corresponds to the IPO year transformation.

All values before correspond to country, and values after correspond to Sector and Market Cap.

res variable can be now used as an X array for ML model training.

Note, that we can reverse transform everything back using the same transformers that are part of our column_trans variable. For example, to access OneHotEncoder corresponding to Sector, we can use the following command:

column_trans.transformers[2][1]

where [2] corresponds to the position of the transformer in column_trans. [1] corresponds to the position of the actual class within the transformer tuple (Name, class, subset).

Having the exact encoder class allows us to decode the data by using Encoder.decode() command.

In the next article, we'll look how to prepare Transformer for the json response we've received from RapidAPI.