Week 6: SentBERT + Model Usefulness
Johnny Y -
This week, after discussing with my advisor, I computed average SentBERT embeddings for all posts of each day. I then manually entered 2024 prices for Dogecoin, adapted my ARIMA model to work with an Excel file (which proved to be harder than I expected – pandas DataFrames are picky!), and appended the ARIMA results to each embedding. I’m currently working on a classifier for each day using PyTorch. Each average embedding has 384 features, and with the next day’s ARIMA prediction, there will be 385 features. The classifier will utilize logistic regression to make a prediction based on the social media and ARIMA data. Recall from my last post that logistic regression examines the dot product of the feature vector and a weight vector, and then adjusts the weight vector by a factor proportional to the magnitude of error (and gradient descent can be used to find the proportion).
A brief explanation of how to measure model usefulness: not all errors are alike – a false positive and false negative should be treated differently (or in this case, a false signal vs “no change”). If the model doesn’t trade, it won’t lose money, but if it makes a trade that goes wrong, it will lose money. Hence, we should put more weight on false positives when measuring the efficacy of our model. This is measured by precision, the proportion of true positives out of all positives predicted by the model. Of course, we also don’t want to miss out on potential opportunities for gains. Recall, the percentage of true positives that the model correctly predicted, measures this. We can combine these using a weighted harmonic mean to form the F1 score, which summarizes our model’s usefulness.