How to pick a Turbo

[Hobo Ash]

Just wanted a bit of guidance with what to look for when getting a new turbo. I know theres loads of threads on here listing the different ones and what cars they come from but i wondered if it was as simple as just buying a turbo and then adapting the manifold so it fits? Im assuming if theres space in the engine bay and its from a diesel car, then it will work? Im also assuming that if it was a turbo off too small of a car it would be a step backwards, and off too big of a car would take forever to get going.
In my head i can just search for a diesel turbo off a 1.8 to 3L car, make an adaptor and fit it. Is it that simple? I know most things that will work well have been tried, but i thought it might be interesting to try using something random