Sometimes it seems like God is wherever his worshipers are, but he does live in a specific place. Worship of the Israelite God started off in the Tabernacle, which was essentially a big tent. Then, about three thousand years ago (900 BCE or thereabouts), Solomon, David's son, built a temple in Jerusalem—or Zion—that was literally God's home.
The Temple got trashed by the Babylonians in 586 BCE (Psalm 79, for example, laments this destruction), but got rebuilt in 539 BCE when the Persian Empire took control of the land and agreed to let the Israelites return home. This ushered in the Second Temple period (you can guess why it's called that), and an even more fervent defense in the post-exilic period of the Temple as the center of worship (absence makes the heart grow fonder and all that).
So, in Psalm 18, God listens to his people "from his temple" (18:6). And when the writer seeks guidance, he asks God, "O send out your light and your truth; let them lead me; let them bring me to your holy hill and your dwelling" (43:3). The temple lent Jerusalem a kind of aura: this was the place to be if you worshiped God. The whole "perfection of beauty" (50:2) thing didn't hurt either.
If you go to Jerusalem today, this language is everywhere, from government documents to city plaques. The Zionists took a ton of inspiration from these passages when they were first writing, and made sure it was part of the government's mandate when Israel took Jerusalem in 1967.