- Get link
- X
- Other Apps
Why use python for your MBA data analysis? Most business school give a short course on stata, SAS, or SPSS. And these tools seem so easy to use! However, this easiness is the first reason you should consider using something different. Why? Many of my students come to me complaining that they've done the analysis using the instructions they'd got in class but they have no idea now how to write the interpretation or, even worse, they've got the results that were mentioned in the lectured as "not good", and now they do not know what to do. The easiness of using using stata, SAS, or SPSS hides the math behind simple procedures, and does not require a student to understand what is happening with the data. It is fine in simple cases where the relationship is obvious, but more difficult cases require understanding of the data and understanding why the results look the way they do. Since data processing in python requires much more thinking about the shape and relationships among the data, it is often easier to understand the results of a statistical analysis.
There is also another reason. Python is widely used in the industry. And in over 20 years of consulting I've never had a chance to work with a company using STATA or SPSS. I worked with 2 companies that used SAS, one of these companies was SAS itself. So, instead of spending time to learn a software package that you will never use in your professional career, why not start straight away with something that is likely to be useful again and again?
So, today we will go through an example panel data regression in Python, using a possible finance problem.
TOC
1. Panel data
1.1. Data
1.2. Data checks
2. Panel model
2.2. Fixed effects model>
2.3. Random effects model
Usually, it is sufficient to remove the variables that are correlated at more than 0.6. In our case, there's no variables like that, thus, we can proceed.
4. Conclusion
1. Panel data
Panel data is a time series data where observations of different entities are mixed together to find insights that are true across entities. Panel data is often used when time observations for each entity are not sufficient to derive a separate regression for it. For example, annual performance of companies during a financial crisis usually can have only several data points, which is not sufficient to test even a single relationship. Combining the data from many companies allows to increase the number of data points, thus, allowing to draw inferences.
In this example, we will use media sentiment data for a range of companies, tracked daily over several months. In this case, some of the companies are mentioned often in the media while others are mentioned seldom. Using panel data allows to see if meaningful insights about the influence of media sentiment on stock price can be drawn for all companies, whether routinely discussed by the media or mentioned once in a while.
1.1. Data
The data used in this example is the historical stock price data and media sentiment scores measured using different algorithms and presented in tone1..tone5 variables. As the media sentiment is meant to predict whether or not the stock price will go up the next day, we will calculate the change of price for each day. Also, as it is well known that individual stocks tend to move with the market, we add a column "market_change" to track the daily change in the overall market. The data then looks as follows:
| change | tone1 | tone2 | tone3 | tone4 | tone5 | market_change | |
|---|---|---|---|---|---|---|---|
| 0 | 0.005017 | 0.036539 | 0.363908 | 0.086821 | 0.007882 | 0.005130 | 0.011425 |
| 1 | 0.006349 | -0.115185 | 0.465259 | 0.155233 | 0.076364 | 0.007610 | 0.000490 |
| 2 | 0.007714 | 0.149236 | 0.451178 | 0.113196 | 0.083723 | 0.002140 | 0.002659 |
| 3 | 0.001938 | 0.034375 | 0.385901 | 0.100878 | 0.000000 | 0.002618 | 0.000980 |
| 4 | 0.002371 | -0.112875 | 0.479342 | 0.102716 | 0.000000 | 0.001348 | 0.000489 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 2528 | 0.005556 | -0.066687 | 0.439845 | 0.137625 | 0.053249 | 0.003884 | 0.001744 |
| 2529 | 0.028052 | -0.047380 | 0.512729 | 0.129501 | 0.058093 | 0.005320 | 0.000277 |
| 2530 | 0.019991 | -0.096078 | 0.419951 | 0.165278 | 0.146863 | 0.002125 | 0.000832 |
| 2531 | 0.022237 | -0.099326 | 0.474362 | 0.121675 | 0.040342 | 0.003382 | 0.008384 |
| 2532 | 0.007889 | -0.000771 | 0.446264 | 0.120838 | 0.078484 | 0.004878 | 0.002499 |
Further, we need to get our libraries and define our endogenous (dependent) and exogenous (independent) variables:
import pandas as pd
import os
import numpy as np
import scipy as sp
import statsmodels.api as sm
import matplotlib.pyplot as plt
exog_vars = ['date', 'company', 'tone1', 'tone2', 'tone3', 'tone4','tone5', 'change','market_change']
df1=df[exog_vars]
df1 = df.set_index(["company", "date"])
endog = df1['change']
df1 = df1[independent_vars]
exog = sm.add_constant(df1)
1.2. Data checks
Before running the regressions, it is necessary to do a few checks. First, we need to verify that there's no strong correlation between the exogenous variables:
independent_vars=['tone1', 'tone2','tone3', 'tone4','tone5','market_change']
df[['change']+independent_vars].corr()
The correlation table is shown below:
| change | tone1 | tone2 | tone3 | tone4 | tone5 | market_change | |
|---|---|---|---|---|---|---|---|
| change | 1.000000 | 0.008409 | -0.020838 | -0.018091 | 0.037969 | 0.035073 | 0.204896 |
| tone1 | 0.008409 | 1.000000 | -0.089915 | -0.285487 | 0.179817 | -0.043217 | 0.020401 |
| tone2 | -0.020838 | -0.089915 | 1.000000 | 0.060421 | 0.199747 | 0.008590 | 0.032540 |
| tone3 | -0.018091 | -0.285487 | 0.060421 | 1.000000 | -0.137627 | 0.031506 | -0.001985 |
| tone4 | 0.037969 | 0.179817 | 0.199747 | -0.137627 | 1.000000 | 0.100439 | -0.002575 |
| tone5 | 0.035073 | -0.043217 | 0.008590 | 0.031506 | 0.100439 | 1.000000 | -0.052801 |
| market_change | 0.204896 | 0.020401 | 0.032540 | -0.001985 | -0.002575 | -0.052801 | 1.000000 |
The next step is to check whether our data has any outliers. One of the easiest ways to check that is to make a Leverage plot. For this plot, we need to define our endogenous (dependent) and exogenous (independent) variables:
exog_vars = ['date', 'company', 'tone1', 'tone2', 'tone3', 'tone4','tone5', 'change','market_change']
df1=df[exog_vars]
df1 = df.set_index(["company", "date"])
endog = df1['change']
df1 = df1[independent_vars]
exog = sm.add_constant(df1)
Now, the Leverage plot:
from statsmodels.graphics.regressionplots import plot_leverage_resid2
results = sm.OLS(endog, exog).fit()
fig, ax = plt.subplots(figsize=(16, 12))
fig = plot_leverage_resid2(results, ax=ax)
In the leverage plot, the values on top left and top right are important outliers. From the graph above, we can see that BAC and JPM values in June are valuable outliers. It makes sense to remove them:
endog=endog.drop(('jpm', '2021-06-16'))
exog=exog.drop(('jpm', '2021-06-16'))
endog=endog.drop(('bac', '2021-06-27'))
exog=exog.drop(('bac', '2021-06-27'))
2. Panel model
Panel regression can be built using either Pooled OLS or Fixed effects or Random effects models. These models are presented below.
2.1. Pooled OLS regression
from linearmodels import PooledOLS
import statsmodels.api as sm
mod = PooledOLS(endog, exog, check_rank=False)
pooledOLS_res = mod.fit()
# Store values for checking homoskedasticity graphically
fittedvals_pooled_OLS = pooledOLS_res.predict().fitted_values
residuals_pooled_OLS = pooledOLS_res.resids
print(pooledOLS_res)
PooledOLS Estimation Summary
================================================================================
Dep. Variable: change R-squared: 0.0530
Estimator: PooledOLS R-squared (Between): 0.0179
No. Observations: 2531 R-squared (Within): 0.0538
Date: Sun, Sep 19 2021 R-squared (Overall): 0.0530
Time: 16:27:37 Log-likelihood 7706.5
Cov. Estimator: Unadjusted
F-statistic: 23.548
Entities: 51 P-value 0.0000
Avg Obs: 49.627 Distribution: F(6,2524)
Min Obs: 1.0000
Max Obs: 83.000 F-statistic (robust): 23.548
P-value 0.0000
Time periods: 83 Distribution: F(6,2524)
Avg Obs: 30.494
Min Obs: 18.000
Max Obs: 48.000
Parameter Estimates
=================================================================================
Parameter Std. Err. T-stat P-value Lower CI Upper CI
---------------------------------------------------------------------------------
const 0.0139 0.0029 4.7743 0.0000 0.0082 0.0197
tone1 0.0050 0.0032 1.5767 0.1150 -0.0012 0.0112
tone2 -0.0144 0.0062 -2.3143 0.0207 -0.0265 -0.0022
tone3 -0.0011 0.0091 -0.1221 0.9028 -0.0189 0.0167
tone4 0.0169 0.0041 4.1100 0.0000 0.0088 0.0249
tone5 0.0187 0.0157 1.1901 0.2341 -0.0121 0.0496
market_change 0.7602 0.0722 10.523 0.0000 0.6186 0.9019
=================================================================================
Normally, Pooled OLS does not work too well if the samples for each period are not taken randomly. As, this is the case in our example because the same companies were tracked over the time interval, we need to develop Fixed or Random effects models.
2.2. Fixed effects model
In a fixed effects model, we can control for time or entity effects. If time effects are set to False, the assumption is that the effect remains the same no matter the time interval. If entity effects are set to False, the assumption is that the effect remains the same across all entities. In our case, it is likely that different companies will be differently affected by the change in media sentiment, but it is not likely that the effect of the sentiment is time dependent. Thus, we choose entity_effects only.
from linearmodels.panel import PanelOLS
mod = PanelOLS(endog, exog,entity_effects=True, time_effects=False, drop_absorbed=True)
fe_res = mod.fit()
print(fe_res)
2.3. Random effects model
In a random effects model, the assumption is that the effects across entities or time are random. Generally, random effects allows to estimate the variability in the population and is more efficient.
from linearmodels.panel import RandomEffects
mod = RandomEffects(endog, exog)
re_res = mod.fit()
print(re_res)RandomEffects Estimation Summary ================================================================================ Dep. Variable: change R-squared: 0.0649 Estimator: RandomEffects R-squared (Between): 0.0271 No. Observations: 2531 R-squared (Within): 0.0572 Date: Sun, Sep 19 2021 R-squared (Overall): 0.0459 Time: 16:28:21 Log-likelihood 7951.3 Cov. Estimator: Unadjusted F-statistic: 29.187 Entities: 51 P-value 0.0000 Avg Obs: 49.627 Distribution: F(6,2524) Min Obs: 1.0000 Max Obs: 83.000 F-statistic (robust): 25.284 P-value 0.0000 Time periods: 83 Distribution: F(6,2524) Avg Obs: 30.494 Min Obs: 18.000 Max Obs: 48.000 Parameter Estimates ================================================================================= Parameter Std. Err. T-stat P-value Lower CI Upper CI --------------------------------------------------------------------------------- const 0.0121 0.0029 4.1363 0.0000 0.0064 0.0178 tone1 0.0076 0.0031 2.4575 0.0141 0.0015 0.0137 tone2 -0.0073 0.0059 -1.2373 0.2161 -0.0189 0.0043 tone3 0.0027 0.0086 0.3150 0.7528 -0.0142 0.0196 tone4 0.0079 0.0040 1.9574 0.0504 -1.427e-05 0.0158 tone5 -0.0016 0.0155 -0.1056 0.9159 -0.0320 0.0287 market_change 0.7719 0.0658 11.739 0.0000 0.6429 0.9008 =================================================================================While Random Effects is generally more efficient and in this case results in a seemingly more explanatory power, random effects model can only be used if they are as stable as the fixed effects model. The stability can be verified using Hausman test:import numpy.linalg as la
from scipy import stats
import numpy as np
def hausman(fe, re):
b = fe.params
B = re.params
v_b = fe.cov
v_B = re.cov
df = b[np.abs(b) < 1e8].size
chi2 = np.dot((b - B).T, la.inv(v_b - v_B).dot(b - B))
pval = stats.chi2.sf(chi2, df)
return chi2, df, pvalComparing fixed and random effects in our case gives:chi-Squared: 0.8483391913225743hausman_results = hausman(fe_res,re_res) print('chi-Squared: ' + str(hausman_results[0])) print('degrees of freedom: ' + str(hausman_results[1])) print('p-Value: ' + str(hausman_results[2]))
degrees of freedom: 7
p-Value: 0.9969187721379007 If p-value is small, the populations are the same, and random effects can be used. If p-value is large as in our case, the populations are different, therefore, fixed effects model should be used.3. Model evaluation
At this point, we know that our data is described best using the fixed effects model. From the parameters of the model we can see that the model is valid (p-values in the heading are below 0.05). At the same time the model is not very explanatory as it describes only 6% of the data (R-square). This is not necessarily a problem. Generally, it means that other variables exist that can describe the data better. If you are working on a panel model in finance, you can find a variety of control variables in literature that are used to improve the predictability of the model. However, this is outside the scope of this exercise.The useful information that we can extract from the model is that the stock price variability over a day is significantly and positively correlated with the tone of the media the day before (measured by tone1 and tone5, where tone1 and tone5 are simply different methods for evaluating the positivity/negativity of an article)As expected, the stock price change over the day is also significantly correlated with the move of the market.Finally, we can add some more tests to understand the deficiencies of the model better. Since we know that the model does not explain the data too well, we can look at the shape of the residuals and attempt to understand what is happening. We'll start with Q-Q plot:pplot = sm.ProbPlot(residuals_fixed) fig = pplot.qqplot() h = plt.title("Model VD - qqplot - residuals of OLS fit") plt.show()The Q-Q plot shows that the data is not normally distributed. While normality is not required in panel data, it is likely that another type of regression will work better with the data.We can also verify that the residuals are not normally distributed:n, bins, patches = plt.hist(residuals_fixed, 25, density=True, facecolor='g', alpha=0.75) plt.xlabel('residuals') plt.ylabel('Density') plt.title('Histogram of residuals') plt.grid(True) plt.show()While the non-normality can be guessed from the graph above, it is always better to confirm using a test, such as Shapiro-Wilkinson test:from scipy.stats import shapiro #The null-hypothesis of this test is that the population is normally distributed. #Thus, if the p value is less than the chosen alpha level, #then the null hypothesis is rejected and there is evidence that the data tested are not normally distributed. shapiro(res) print("Shapiro Wilkinson statistic:" + str(shapiro(residuals_pooled_OLS)[0] )+ ", p-value: " + str(shapiro(residuals_pooled_OLS)[1]) )Shapiro Wilkinson statistic:0.7917779684066772, p-value: 0.0 (Not normal).Another problem with the residuals is Homoskedastisity, i.e. changing variability of residuals with fitted values. Again, we can present it visually and confirm using a test.import matplotlib.pyplot as plt fig=plt.figure(figsize=(20,12)) fig, ax = plt.subplots() ax.scatter(fittedvals_pooled_OLS, residuals_pooled_OLS, color = 'blue') ax.axhline(0, color = 'r', ls = '--') ax.set_xlabel('Fitted Values') ax.set_ylabel('Residuals') ax.set_title('Homoskedasticity') plt.show()
Visually, the plot above is not homoskedastic. However, it is always better to confirm using a test:from statsmodels.stats.diagnostic import het_breuschpagan #The null hypothesis for this test is that the error variances are all equal. # If p-value is small, null hypothesis is rejected and error variances are all different bp=het_breuschpagan(residuals_pooled_OLS, exog, robust=True) print("Breusch-Pagan statistic: f-value: " + str(bp[2] )+ ", p-value: " + str(bp[3]) ) print("Breusch-Pagan statistic: lm: " + str(bp[0] )+ ", p-value: " + str(bp[1]) )Breusch-Pagan statistic: f-value: 2.9517126183995623, p-value: 0.007133642737474482
Breusch-Pagan statistic: lm: 17.635648032499365, p-value: 0.0072102542443691344. Conclusion
This post explains how to use python-based panel data analysis in your research paper or dissertation. While the model developed in this post is far from perfect, it is important to remember that this is also the case for many models conceived and developed in student research. This is nothing to be scared of. The knowledge is generated gradually, and having proven that the variables of interest (media sentiment measured by the tone) are significant, we can proceed to the next step and develop a better model in the future.And this is where having a clean easy-to-modify code in Python is also very handy. But this is a topic for another post.Need more help? Have questions? Contact usBibliography
Wooldridge, J. M. (2010)Econometric analysis of cross section and panel data
Comments
Post a Comment