Playback isn't a problem usually, but i did encounter problems with encoding H.264, on server side, using ffmpeg of which the build process wasn't trivial.
Doing a high-quality video encode is a non-trivial procedure. Even using a GUI tool like Handbrake effectively, requires understanding a lot of parameters. You could just take defaults, but the results will be nowhere near as good as if you tweak the settings for your particular source material. I encode a fair amount of video using FFMPEG, but usually I am using x264 directly.
As to compiling ffmpeg yourself - again, non-trivial. Better to rely upon good builds or packages. I've done it on occasion to take advantage of some new cutting edge stuff. The thing is, ffmpeg works really well. It is a fabulous tool with a tremendous amount of power and capability. But any complex piece of software, with many pieces and many dependencies, is going to be difficult to build.
Packages ware old, especially with regards to H.264 support, granted you have to tweak the settings to get a good result, but at present I think H.264 suffers on Linux, because of the legal baggage that comes with it.
Many services will surely rely on trans-coding on the fly, questionable support for this format on the server level might adversely effect development on open source platforms.
