I've found that the most reliable systems I've worked with have a defense-in-depth model, with reliability baked into the system architecture. That includes things like timeouts & retries for RPCs, replicas, graceful fallback, backup implementations, subsystem isolation, shut-off switches, etc. This is the Erlang/OTP model, and much of Google Search & cloud infrastructure is also built on similar principles.