I've been thinking of writing a blog post about this for a while... probably with the title "Who Watches the Watchers" :).

We use Wormly to monitor our site, email and DNS. We also have an exception reporting system which alerts us about any 500 errors in the site. We also alert if phone call or SMS messages sit in our event queues for more than 3 minutes after the scheduled delivery time.

We also have redundant data centers and rapid rollover to the backup systems in the event of a data center outage.

We don't use PagerDuty to do the alerting and schedule the on-call though. We alert everybody in our team via SMS and phone if any of the aforementioned alerts go off.