I really love this idea, but it seems like on macOS, you can't use the left/right arrows to skip forward/backward. I did some testing and this appears to be a limitation of the HTML5 video player implemented in Safari, but it would be amazing if this functionality was added (along with other keyboard shortcuts).
J for rewinding 10 seconds, K for pausing, and L for fast-forwarding 10 seconds is what i've gotten used to, along with , and . for frame-by-frame scrubbing.