125 points by data_scientist23 6 months ago flag hide 10 comments
deeplearning88 6 months ago next
This is impressive, I wonder what kind of datasets they used to train the model? The 99% accuracy is intriguing.
thoughtvectors 6 months ago next
I also have questions about the evaluation process. How did they measure the accuracy? Was it on standard NLP test datasets?
logicmaster007 6 months ago next
In order for this kind of text analysis models to be accepted, they should pass various existing NLP tests, not just a custom evaluation method.
wizarrd_codez 6 months ago prev next
The model might be over-optimized for the datasets used. We need more information about their test and validation approach.
deeplearning88 6 months ago next
Sure, we also used multiple datasets, but got similar high accuracy results. We'll release the datasets and projects for reproducibility purposes soon.
polygoned_matrix 6 months ago next
Great to hear that, hopefully we can also examine the models using other evaluation techniques to ensure its accuracy. @deeplearning88
neuraljargon 6 months ago prev next
99% accuracy feels a bit too optimistic, I think there are probably some limitations and error cases that would decrease that accuracy. Has it been stress-tested?
happycoder99 6 months ago next
I fully agree, I'm eager to see how this model responds in out-of-distribution test cases.
proglinguist 6 months ago prev next
Sounds promising! Existing text analysis models are in dire need of improvement. Such results could boost NLP real-life applications significantly.
textlings 6 months ago next
Well said @proglinguist, I'm excited to see and test this innovation hands-on!