The village of Ashgrove in the suburbs of Fenton, Staffordshire, Britain
By piecing together details given during the story's first few pages, we establish that Yu Tsun is doing his spying in Staffordshire, England in 1916, during the first World War. His mission takes him to the village of Ashgrove, a rural area in the suburbs of Fenton. In other words, this is definitely not an urban setting. Yu Tsun and Richard Madden duke it out in the boondocks, miles away from London and Berlin, where all the really powerful people live. It doesn't seem like the actions of these two rural nobodies could be all that important, does it? And yet the outcome of their little spy vs. spy drama will decide the fate of entire armies. (The British setting also explains why the Germans are so vilified – they don't have a very good rap during the war, after all.)
It's unclear from the story's framing paragraph when and where the initial narrator is located. He's clearly far enough removed from the time period to be contributing to a historical discussion of an obscure event of the "World War." Placing him at the time of the story's publication in 1941 would make sense, but that's really just a guess.
Dr. Stephen Albert's Pavilion
We notice a lot of similarities between the place where Dr. Stephen Albert has secluded himself and the retreat of his literary idol, Ts'ui Pen. Ts'ui pen shut himself up in the Pavilion of the Limpid Sun, a building set in the middle of an intricate garden. Dr. Albert lives in a "kind of summer house or pavilion" accessible by a series of forking roads and a damp, zigzagging path (23).
Honestly, how many people do you know who live in a "pavilion"? Yu Tsun even imagines his great-grandfather's labyrinth as an infinite maze constructed of "eight-sided pavilions." There's definitely a connection here between the structure of the pavilion – an octagon that can be placed next to other octagons in a never-ending geometric pattern – and the idea of the infinite, which is a major theme for Borges.
At the center of Dr. Albert's pavilion we find a library, which is another favorite image in Borges' work. (Check out the short story "The Library of Babel.") The library is also connected to the idea of the infinite. Think about it – it's where Ts'ui Pen writes his novel of infinite endings, and where Stephen Albert researches the infinite book. Its very repetition (in Ts'ui Pen's story and in Dr. Albert's) suggests a space that appears over and over again, in this time and possibly in alternate ones.