256 points by ml_stockwhiz 1 year ago flag hide 21 comments
financeguru 1 year ago next
Interesting project! I'd like to see more details about the types of machine learning algorithms you used. Did you consider using neural networks or random forests?
financeguru 1 year ago next
Neural networks are always a solid choice, and they tend to do well on complex data sets like stock market data. What kind of feature engineering did you do on the data?
quant4life 1 year ago prev next
Really cool to see machine learning being applied to the stock market. I'm curious about what data preprocessing you did to prepare your data for training and testing. Did you standardize or normalize the data?
quant4life 1 year ago next
Data preprocessing is crucial for ML projects. I've found that standardizing the data tends to give me slightly better results than normalization. It's also interesting that you went with neural networks, I think when you have large datasets, that can be a good choice.
quant4life 1 year ago next
If I have time, I'll also try using neural networks on my next stock prediction project. Thanks for the suggestion.
algotrader 1 year ago prev next
Very cool, I've been following this topic for a while. I'm curious, what kind of evaluation metrics did you use to decide your model's performance?
financeguru 1 year ago next
Evaluation metrics vary by use case. Did you use something like Mean Absolute Error (MAE), or Root Mean Square Error (RMSE) for regression tasks, or metrics like accuracy, precision or recall for classification tasks?
algotrader 1 year ago next
We used MAE for evaluation. I think MAE is pretty good because it gives us a sense of by how much the model is continuously wrong, whereas RMSE would penalize large errors more than smaller ones. This is helpful, because one can still make money in the market even if the prediction is wrong by a little bit, but being wrong by a large margin would be more penalizing.
anonymous 1 year ago prev next
The market is pretty unpredictable, how can machine learning predict it with higher accuracy than other models?
financeguru 1 year ago next
Machine learning essentially uses data to detect underlying patterns in the data with a degree of accuracy. While I can see how someone could think that the stock market is unpredictable, it's clear based on historical patterns that there is indeed some degree of predictability (learn more at https://quantpedia.com/). Basically, that's what we're trying to capitalize on with this project.
cryptofanatic 1 year ago prev next
Have you considered using any other ML approaches like LSTM's and GRU's for time series predictions? where X is open, high, low, close, \\ volume, and y is the next day's open value. The learning occurs on the open, high, low, close and volume columns. Time series can be tricky but RNNs and their variants show promising results.
algotrader 1 year ago next
Yes, we have tested LSTMs and GRUs, they have potential as you pointed out. The challenge we've noticed was the long training times and the need for more data to have reliable predictions. We found that simpler models were more suitable for our use case in this initial phase of the project
cryptofanatic 1 year ago next
@algoTrader Yes, we found the same, overfitting and a lot of noise with GRUs. This is a great start and I'm looking forward to seeing your project improvements
algotrader 1 year ago next
@tradingPro We currently do not make use of news data in our stock predictions. We found that there's so much noise in the data that it's difficult to extract features that are meaningful and helpful in improving the accuracy of our predictions. However I think with some elbow grease news data could yield interesting features
quant4life 1 year ago next
@tradingPro To add to that, news data can be subjective and finding the right news and timing can be crucial. We did not find any relevant data to incorporate in the model but I believe that's a good question for further research
ml_beginner 1 year ago prev next
I've been interested in machine learning for some time now, and really want to start a side project on stock market predictions. I'm not an expert but have taken some online courses on ML, I'd love to get some feedback. Would appreciate any resources/projects you recommend for a beginner like me
machinelearningmaster 1 year ago next
@ML_Beginner Get started with some small tasks, maybe try using open data-sets, and some basic ML algorithms. Kaggle competitions are a great place to start. Some good ones to check out for beginners are Titanic and House Prices. Once you're familiar with basic ML concepts, try to move on to some more complex problems, perhaps related to stock market predictions. I'd be happy to help along the way.
ml_beginner 1 year ago next
@MachineLearningMaster Thank you! I'll look into those competitions, sounds like a great starting point to apply ML theory practically and get my feet wet!
tradingpro 1 year ago prev next
Can you share your insights on how your algorithm integrates or makes use of news data for stock predictions?
the4thalgo 1 year ago prev next
Did you consider ensembling approaches like stacking/blending/boosting or was it beyond the scope of this project?
algotrader 1 year ago next
@the4thAlgo We did not currently explore ensembling approaches like stacking, blending or boosting as this was out of scope for this project. As we iterate and improve upon the current version, ensembling and combining models could certainly be a good strategy to improve model accuracy.