In response to Mr. Hoare, below, the answer is 'yes and no.' Yes, the writer is referring to a specific inn, but no, he does not mean to say 'the inn.' In a sense, he is being coy. He wants her to remember a specific inn, perhaps among several at which the two of them stayed. But he does not want to say there was only one. In a sense his coyness is related to the title of the poem, Tarantella, which is technically a circular dance. He is dancing around the real object of his questioning. The irony is, by the time he has finished his own dance he has given so many clues that if she fails to 'get it, ' he is probably talking to the wrong woman.
I think that this poem is wonderful, I first heard it when I was about 10 and it's been my favourite ever since. It seems to have such an atmosphere about it, very lively and yet also very sad at the end. It has a nice reminiscient feel about it.