What if the universe is empty beyond a certain distance from the origin/central point, like a tiny explosion in the middle of an unending emptiness? Then there wouldn't have to be duplicates of you.
(This is about the same thing as the question asked below about whether infinite size implies infinite mass.)
In a flat universe, the universe starts off infinitely large and infinitely dense at time 0. As the universe expands, it gets less dense. Imagine an infinitely dense rubber sheet that starts to stretch at time 0, and after time 0 it is no longer infinitely dense, and it keeps slowly stretching forever, getting less and less dense over time. That's the universe as it is currently envisioned.
But our "Hubble sphere" is in no way special compared to other Hubble spheres. Our Hubble sphere is the part of the universe surrounding us from which light has arrived. Outside that sphere, nothing has a causal connection to us, since nothing can travel faster than the speed of light.
I.e., it just wouldn't be the case that our Hubble sphere is full of stuff and all the other Hubble spheres are are so different as to be empty.
(This is about the same thing as the question asked below about whether infinite size implies infinite mass.)