90 points by neural_ninja 6 months ago flag hide 12 comments
johnsmith 6 months ago next
This is really interesting! I wonder how this compares to previous methods like weight quantization and knowledge distillation.
emilychen 6 months ago next
Weight quantization and knowledge distillation have their own trade-offs. This new pruning technique could potentially complement those methods for further size reduction.
jeffreygao 6 months ago prev next
I'm curious how this would perform in practice, especially on edge devices.
annalee 6 months ago next
It's definitely worth testing on various edge devices with different computation capacities. The results could offer valuable insights.
stevekim 6 months ago prev next
I noticed that the paper mentioned 'neuron importance scores' but didn't elaborate much. Does anyone have a better understanding of how they're calculated?
helenwong 6 months ago next
The neuron importance scores are calculated based on the weight values in each layer and their gradients, then normalized to get a distribution. Later, less important neurons are pruned leading to reduced model size.
charlieliu 6 months ago prev next
Has anyone tried applying this method to other tasks, like NLP or audio generation?
nicolechu 6 months ago next
Reducing model size with techniques like this could indeed work for NLP. I haven't seen applications for audio generation, but it's an interesting idea for future research.
alexhong 6 months ago prev next
How would fine-tuning be affected with this reduced model size? I think it's crucial to consider the impact on downstream fine-tuning performance.
davidyang 6 months ago next
True, that's a vital consideration when applying compression techniques like this to image recognition models. The impact on fine-tuning may vary, but it's worth monitoring for sure.
lucylee 6 months ago prev next
This could potentially help with storage and distribution of models. That, coupled with techniques like differential privacy, enables a whole new level of collaboration in ML.
pauljohn 6 months ago next
Indeed, the distribution of smaller models is crucial, especially when projects need to adhere to storage and bandwidth constraints.