Why install fail2ban? You already have SSH password auth disabled, and you only allow SSH connections from your office. Won't this just risk banning your own office if someone's SSH client is misconfigured?
If helps if a) you've misconfigured your "office only" rules, or b) someone has breached your office network/one of your local machines (but hasn't discovered any of your keys/passwords yet) and is trying to get into the server. It also helps with non-ssh services too. A belt-and-braces approach. If you accidentally lock yourself out using it, use your out-of-band access to re-enable your logins.
agreed, you can set up fancy jails for people scanning other services too, someone who probes SMTP/POP/IMAP doesn't need to hit SIP and SSH. Depending on the scenario you could choose to say block an entire netblock from hitting ssh after a single offensive IP probes a few services. Even a 10min jail time will cause most attackers to give up and move along to their next victim (unless you're being targeted.)
