Summary: The speed of light is a universal upper limit. Distances between stars range from 4.3 light years to Alpha Centauri to a hundred thousand light years across the Milky Way galaxy to millions of light years between galaxies. These facts are incompatible with tens of thousands of apparent visitations
The Leading Theory-Based Rejectionist Argument (which we all know)
The speed of light is a universal upper limit. Distances between stars range from 4.3 light years to Alpha Centauri to a hundred thousand light years across the Milky Way galaxy to millions of light years between galaxies. These facts are incompatible with tens of thousands of apparent visitations
Comment on the Argument
I agree completely that if the only way to get from star system A to star system B is to travel at sublight speed, this rules out frequent visitation. You might expect a visit once every ten thousand years (to cite a number I believe Carl Sagan once pulled out of his hat) even if the galaxy is teeming with civilizations. The questions is, are there conceivable alternatives to slogging through space? Maybe.
The Alternatives Worth Considering
The speed-of-light limit only applies to motion through four-dimensional spacetime. Perhaps wormholes are possible. That is a concept for which Kip Thorne gets the credit (or the blame). It is an old, but still valid, argument for how traveling through vast distances might be circumvented. Perhaps spacetime itself can be stretched as proposed by the relativist Miguel Alcubierre. There is no speed-of-light limit to spacetime stretching. After all, spacetime beyond the Hubble horizon must be receding from us at v>c. The Alcubierre "warp drive" (Class. Quant. Grav., 11-5, L73-L77, 1994) shows that spacetime warping and stretching around a bubble of flat spacetime is mathematically consistent with general relativity. This sounds promising, but the energy requirements seem to be impossible, which is of course not a good thing. Modern superstring and M-brane theory imply the existence of numerous additional dimensions. Recent work indicates that these additional dimensions may be much larger than the Planck scale. The article "The Universe's Unseen Dimensions" by Nima Arkani-Hamed, Savas Dimopoulos and Georgi Dvali in the August 2000 issue of Scientific American, for example, is a good summary of some current thinking on additional spatial dimensions as large as a millimeter: "Our whole universe may sit on a membrane floating in a higher-dimensional space. Extra dimensions might explain why gravity is so weak and could be the key to unifying all the forces of nature." Perhaps it is possible to lift off the membrane-universe constituting our four-dimensional spacetime, move in one of the additional dimensions where speed-of-light limits may not apply, and reenter our membrane-universe very far away. All of this is speculation of course, but it is worth noting that disappearing in place, changing shape or sometimes jumping discontinuously from location to location is frequently reported in UFO observations. Such behavior could conceivably be associated with motion into and out of a perpendicular dimension.
Possible Conclusion
The speed-of-light limit argument against the UFO phenomenon is a theory-based one, but even without suspending the laws of relativity it may not be valid. We simply know too little about other possibilities to rule them out, and for that reason the appropriate thing to do is to suspend judgment based on this argument.