I think it really depends on the site. I have found some websites that link to me have just copied content from my website to earn adsense dollars and really have nothing to do with my content.

I find that most sites that link to mine are worth checking out. Except for the weird ones that seem to be almost random links, a lot of them either have something I'm interested in, or link to something that's up may alley.

If I have the time, I always do a bit of backlink browsing.

When I find a new link into one of my sites, I often go back to it to check it out. While I am no more likely to 'trust' the site that linked me, often such explorations lead me to new blogs or sites I hadn't seen before. Occassionally, I find things that compel me to write about in my blog, or on another site.

In essence, a site that links to me stands a fairchance of picking up another backlink.