Solving Statistical Hypothesis Testing with Machine Learning
I wanted to create a machine learning model to solve statistical hypothesis tests, given a word problem. This model would have to perform several tasks, picking the correct type of test, setting up the correct hypothesis, extracting the correct parameters from the word problem, and finally (the easy part) calculating the test statistic and giving it's conclusion.
This was not an easy task, due to a very limited of training data that I hand crafted. My dataset below consists of only about 20 problems, and many of them are made from the staff of DTSA 5003-200 in Colorado Boulder's online MSDS master's program. This is a very limited dataset that I put together just for the sake of setting up this project.
We used TF-IDF vectorization to conver raw problem text into numeric vectors based on word importance, which we then used to feed our regression and classification models.
We used a logistic regression classifier to predict the type of test (mean,proportion,variance,t-test,z-test,chi-sq,difference of means, etc), and to predict the type of alternate hypothesis (left tail, right tail, or two-tailed).
This works by fitting weights to words that best separate each class.
Then we used a MultiOutput Linear Regression model for value extraction, to try to extract and predict the numerical values (sample mean, variance, proportions, etc), where each value is treated as a separate regression problem based on the text vector.
To improve this, we could use better transformer-based NLP models (bert) for deeper text understanding. We could also train separate models to perform the test, based on the test chosen by the initial classifer.
Overall, the model performs decent on problems from the dataset as shown in the below notebook. Much more data is needed to get a better understanding of where it stands.
To see the results of the sample problem used, check out the jupyter notebook below.