34 points by predictserver 1 year ago flag hide 17 comments
ml_enthusiast 1 year ago next
This is really cool! Using ML to predict server downtime can significantly reduce the impact of outages. I'm curious if anyone has tried applying this in production systems yet?
production_dev 1 year ago next
@ml_enthusiast, we've been testing out a similar solution at my workplace and the results have been promising so far. The real challenge comes in integrating the predictions with our existing monitoring and alerting systems to ensure timely action.
codewarp 1 year ago prev next
Neat idea, but I'm wondering how accurate these predictions could actually be. Server failures can be influenced by a multitude of factors, some of which might be nearly impossible to model accurately.
ml_guru 1 year ago next
True, but even moderately accurate predictions can give engineers a heads-up, allowing them to address potential issues proactively.
ops_dude 1 year ago prev next
I like the idea. I think it could also be helpful to automate the server patching process in response to the predictions. Thoughts?
infrastructure_ninja 1 year ago next
@ops_dude, that's an excellent point. Automating server patching would not only reduce the potential for manual errors but also save time. Does anyone know of any tools that automate patching based on predictions?
security_queen 1 year ago prev next
This approach could have huge benefits for security teams as well, giving them extra time to prepare and respond to potential attacks, especially if integrated with a WAF or IDS.
hacking_dude 1 year ago next
I agree, but what about false positives? Falsely alerting security teams could lead to a boy-who-cried-wolf situation.
security_queen 1 year ago next
Great point, @hacking_dude. The tradeoff between reducing false negatives and increasing false positives would need to be carefully considered. It likely would vary depending on the use case and team's needs.
quant_pred 1 year ago prev next
While it is interesting, have there been any efforts to utilize the same predictive machine learning capabilities for RAID array failure prediction, or is this a much more deterministic process?
ml4servers 1 year ago next
@quant_pred, RAID array failure prediction can and does use ML for prediction. However, there is also a more deterministic approach, using S.M.A.R.T. attributes analytics to proactively identify hard drive issues.
elixir_elite 1 year ago prev next
Has anyone attempted to implement this in a functional programming language like Elixir? Or are most people using the standard imperative languages: Python, Java, etc.?
ml_erlang 1 year ago next
@elixir_elite, I haven't seen much experience using functional programming languages for this kind of application. However, it's definitely possible and might even be easier do to immutability, pattern matching and fewer side effects.
ai_puzzler 1 year ago prev next
Could we use something more exotic, like reinforcement learning, rather than normal regression or classification techniques? Could be more adaptable and responsive to ever-changing environments.
rl_tinker 1 year ago next
@ai_puzzler, I've thought about applying reinforcement learning, but it's difficult to find a clear set of rewards to make the problem well-defined and solvable, at least in a real-world timeframe. Have you found success with this?
hybrid_learner 1 year ago prev next
Has anyone experimented with combining AI-based predictive models with traditional sysadmin heuristics/rules in a unified prediction framework?
hybrid_hunter 1 year ago next
@hybrid_learner, A great idea, but it seems challenging to effectively combine ML heuristics and sysadmin rules, as they must be quantifiable and the integration would have to be robust in the presence of various environmental vagaries.