67 points by distributedman 1 year ago flag hide 12 comments
user1 1 year ago next
Great question! Debugging complex distributed systems can be quite challenging.
user2 1 year ago next
I usually start with collecting and analyzing logs from all the services involved. centralized logging and correlation id help a lot.
user1 1 year ago next
Yeah, I agree. I use tools like ELK to make log analysis easier.
user3 1 year ago prev next
In my experience, network issues or latency problems can also cause issues in complex distributed systems.
user1 1 year ago next
Good point. Monitoring network health and latency can help catch these issues. Observability tools can be very helpful for this.
user4 1 year ago prev next
Another thing that helps me is to have a good understand of the architectural diagram, the flow of requests, data, the communication pattern and dependencies between microservices
user2 1 year ago next
That's true, knowledge about the system's design makes debugging much easier. I would also suggest to do load testing to see how system behaves under high workloads
user5 1 year ago prev next
When there's a failure mode, I like to look at the metrics for the system, enabled alerts, often ask questions like - Which service is impacted? What's the source of the recently large latency? what's the error rate(request/failure) in last few mins?
user1 1 year ago next
Those are all great point. I also would suggest have a watch dogs to automatically recover (automated remediation) in case of a services failure or an unusually slow service.
user4 1 year ago prev next
A beginner friendly approach is Grepping around systematically in log files, using tools like Splunk, ELK, Graylog or even less efficient like grep, awk, less with or without pipes and so on.
user6 1 year ago next
I prefer distributed tracing, it allows to see the whole flow and it really helps identify where the problem is.
user2 1 year ago next
I agree distributed tracing gives a better understanding. We use OpenTracing and Jaeger. You should try it. :)