1011 points by jsrocker 1 year ago flag hide 16 comments
deeplearningfan 1 year ago next
This is very cool! I've read about TensorFlow.js, but never seen a real-world application of it, especially not in gesture recognition. Keep up the good work!
deeplearningfan 1 year ago next
@jsMaster I didn't run any FPS tests myself, but in the documentation of TensorFlow.js, it claims to have high performance due to its GPU support. I guess the FPS depends mainly on the hardware.
jsmaster 1 year ago prev next
I'd be curious to know more about the performance of gestures in real-time. How many FPS (frames per second) does it achieve? I'm asking because I think low latency is a key factor in any gesture recognition system.
jsmaster 1 year ago next
@deepLearningFan From early tests, I managed to squeeze about 30-40 FPS from a high-end smartphone. I've not tested any budget smartphones, but I think the FPS will remain playable at about 20-25 FPS, which I think is still great for a real-world application.
cpufan 1 year ago prev next
Is this compatible with pure CPU-based devices? Because if it is, that'd change everything. Mobile-first gesture recognition will be a GAME CHANGER!
author 1 year ago prev next
@cpuFan Yes, it is compatible with CPU-based devices, but of course, the performance drops significantly compared to GPU-powered one. Nevertheless, I believe TensorFlow.js is reshaping machine learning on the web and will be very popular in the future.
deeplearningfan 1 year ago next
@author I completely agree. How did you deal with the model optimizations and loading on weaker devices? I'd be happy to hear more about your approach ;)
mlquestioner 1 year ago prev next
I've been researching the topic for a few months, and I think the biggest challenge is to verify the accuracy of the gestures. How did you approach verification? Took pre-recordings as reference and compared them to live recordings?
author 1 year ago next
@mlQuestioner You guess right! I used pre-recorded data as reference to compare live recordings, and to make the process more efficient, I used pre-trained gestures libraries to reduce the chances of false positives and negatives. This also helped with model optimization.
ethicalhacker 1 year ago prev next
Did you consider the security implication of motion gesture applications? Could attackers exploit this technology? I'm asking because similar tools in audio processing already had many security issues reported.
author 1 year ago next
@ethicalHacker Yes, I considered the security implications and vulnerabilities. Fortunately, I found some great resources on secure machine learning (ML) models and pipeline design. I will soon release a paper discussing my security procedures to help the community minimize the risk.
interesteduser 1 year ago prev next
I want to use this in a school project, but I'm new to TensorFlow.js. Could you recommend a good starting point (guides/tutorials) for understanding the basics of TensorFlow.js?
jsnewbie 1 year ago next
@interestedUser I suggest you start with the TensorFlow.js guide for beginners. It has a lot of practical examples that'll help you understand the basics. Also, deepLearning.ai just released a (free) TensorFlow: Fundamentals course, I'd recommend that, too.
seniordev 1 year ago prev next
I want to use real-time gesture recognition for a web-based app and went through your documentation earlier today. Great job! Using this, I'll dramatically improve the user experience of our app. Thanks a bunch!
astonisheduser 1 year ago prev next
How did you even implement this? I'm experienced with TensorFlow.js, but this project is still impressive and seems incredibly complex. Teach us, please!
author 1 year ago next
@astonishedUser Thank you! I'm glad it has positively impacted the TensorFlow.js community. I've spent countless hours learning, experimenting, and fine-tuning my implementation. I recommend reading this piece: "Reducing the Complexity of TensorFlow.js with three simple steps" on my Medium blog to give back to the community that supported me.