Dance like no one's watching

We have some regional colleagues who fly in from Hong Kong from time to time and I know they’re just being friendly by making small talk since we don’t see them on a daily basis.  I’m fine with them borrowing a bit of our desk space to check their emails or do some work on their laptop.  But it gets a little annoying when I’m trying to rush work and they’re trying to make small talk.  Perhaps it’s an attempt to be polite since they are borrowing our desk space but it is really not necessary.  People really underestimate the value of "silence is golden".