If it's any consolidation to the new car paranoia, I bought mine (E53 facelift M57N) 18 months ago with 35k on it. Done just over 20k with it since then and it's been brilliant - I absolutely love it. Not the fastest machine in the world but I like that it encourages you to behave yourself and it's quick enough. It's brilliantly comfortable and practical: dog, luggage, car full of business people to meetings, towing 2.7 tonnes on the back. Sometimes all at once. And turning in 34-35 mpg or about 28 if towing heavy.
I had a few teething niggles: non functioning Webasto heater (solved by a legendary chap in France who re-soldered it for a very reasonable donation); slight gearbox oil leak (great excuse to do a fluid and filter change); and a couple of minor odds and ends (the little boot rubber had perished - first world problem that was).
There's a lot of keyboard warriors who will attest that they're no good in snow - but that's entirely down to running aggressive summer rubber which makes them handle better in normal conditions. On winter tyres they're right up there as X drive is fundamentally a very good system.
Watch out for gearbox oil leaks (the sump gaskets go). I have a suspicion that people not noticing leaks may have been an underlying cause of a lot of failures that they have a reputation for - as the box itself is used in hundreds of other cars so there's no particular reason why it should be unreliable in this application. And watch the engine oil level - some of them do use a bit especially if worked hard. Some consumption generally reckoned to be normal albeit people's experience is very varied without apparent rhyme or reason.