1011 points by jsrocker 6 months ago flag hide 16 comments
deeplearningfan 6 months ago next
This is very cool! I've read about TensorFlow.js, but never seen a real-world application of it, especially not in gesture recognition. Keep up the good work!
deeplearningfan 6 months ago next
@jsMaster I didn't run any FPS tests myself, but in the documentation of TensorFlow.js, it claims to have high performance due to its GPU support. I guess the FPS depends mainly on the hardware.
jsmaster 6 months ago prev next
I'd be curious to know more about the performance of gestures in real-time. How many FPS (frames per second) does it achieve? I'm asking because I think low latency is a key factor in any gesture recognition system.
jsmaster 6 months ago next
@deepLearningFan From early tests, I managed to squeeze about 30-40 FPS from a high-end smartphone. I've not tested any budget smartphones, but I think the FPS will remain playable at about 20-25 FPS, which I think is still great for a real-world application.
cpufan 6 months ago prev next
Is this compatible with pure CPU-based devices? Because if it is, that'd change everything. Mobile-first gesture recognition will be a GAME CHANGER!
author 6 months ago prev next
@cpuFan Yes, it is compatible with CPU-based devices, but of course, the performance drops significantly compared to GPU-powered one. Nevertheless, I believe TensorFlow.js is reshaping machine learning on the web and will be very popular in the future.
deeplearningfan 6 months ago next
@author I completely agree. How did you deal with the model optimizations and loading on weaker devices? I'd be happy to hear more about your approach ;)
mlquestioner 6 months ago prev next
I've been researching the topic for a few months, and I think the biggest challenge is to verify the accuracy of the gestures. How did you approach verification? Took pre-recordings as reference and compared them to live recordings?
author 6 months ago next
@mlQuestioner You guess right! I used pre-recorded data as reference to compare live recordings, and to make the process more efficient, I used pre-trained gestures libraries to reduce the chances of false positives and negatives. This also helped with model optimization.
ethicalhacker 6 months ago prev next
Did you consider the security implication of motion gesture applications? Could attackers exploit this technology? I'm asking because similar tools in audio processing already had many security issues reported.
author 6 months ago next
@ethicalHacker Yes, I considered the security implications and vulnerabilities. Fortunately, I found some great resources on secure machine learning (ML) models and pipeline design. I will soon release a paper discussing my security procedures to help the community minimize the risk.
interesteduser 6 months ago prev next
I want to use this in a school project, but I'm new to TensorFlow.js. Could you recommend a good starting point (guides/tutorials) for understanding the basics of TensorFlow.js?
jsnewbie 6 months ago next
@interestedUser I suggest you start with the TensorFlow.js guide for beginners. It has a lot of practical examples that'll help you understand the basics. Also, deepLearning.ai just released a (free) TensorFlow: Fundamentals course, I'd recommend that, too.
seniordev 6 months ago prev next
I want to use real-time gesture recognition for a web-based app and went through your documentation earlier today. Great job! Using this, I'll dramatically improve the user experience of our app. Thanks a bunch!
astonisheduser 6 months ago prev next
How did you even implement this? I'm experienced with TensorFlow.js, but this project is still impressive and seems incredibly complex. Teach us, please!
author 6 months ago next
@astonishedUser Thank you! I'm glad it has positively impacted the TensorFlow.js community. I've spent countless hours learning, experimenting, and fine-tuning my implementation. I recommend reading this piece: "Reducing the Complexity of TensorFlow.js with three simple steps" on my Medium blog to give back to the community that supported me.