125 points by data_scientist23 11 months ago flag hide 10 comments
deeplearning88 11 months ago next
This is impressive, I wonder what kind of datasets they used to train the model? The 99% accuracy is intriguing.
thoughtvectors 11 months ago next
I also have questions about the evaluation process. How did they measure the accuracy? Was it on standard NLP test datasets?
logicmaster007 11 months ago next
In order for this kind of text analysis models to be accepted, they should pass various existing NLP tests, not just a custom evaluation method.
wizarrd_codez 11 months ago prev next
The model might be over-optimized for the datasets used. We need more information about their test and validation approach.
deeplearning88 11 months ago next
Sure, we also used multiple datasets, but got similar high accuracy results. We'll release the datasets and projects for reproducibility purposes soon.
polygoned_matrix 11 months ago next
Great to hear that, hopefully we can also examine the models using other evaluation techniques to ensure its accuracy. @deeplearning88
neuraljargon 11 months ago prev next
99% accuracy feels a bit too optimistic, I think there are probably some limitations and error cases that would decrease that accuracy. Has it been stress-tested?
happycoder99 11 months ago next
I fully agree, I'm eager to see how this model responds in out-of-distribution test cases.
proglinguist 11 months ago prev next
Sounds promising! Existing text analysis models are in dire need of improvement. Such results could boost NLP real-life applications significantly.
textlings 11 months ago next
Well said @proglinguist, I'm excited to see and test this innovation hands-on!