Like most things in Antigua, the library has long been abandoned by the people in charge. Sure, the cats from the Mill Reef Club say that they want to fix it, but they don't do jack when push comes to shove—after all, some businessman is planning to "develop that part of St. John's" (3.1) into a tourism district. There's a lot more money to be made off of shops selling tchotchkes than lending books, and as has been the case for generations, the investment for the powers that be on Antigua is in outsiders instead of the Antiguan masses.
There's a new library across town, but its sorry state only emphasizes this hypocrisy. The Mill Reef Club folks could easily pay to renovate the new building (or even buy a new one), but they hold on to Antigua's past by refusing to help rebuild its future. Both the narrator and people from the Mill Reef Club are nostalgic about the old Antigua, but they simply "don't have the same old Antigua in mind" (3.1)—one longs for the days of slavery, while the other most definitely does not.
The library, then, represents the chasm between these groups, and the ways in which oppression takes on new forms instead of actually disappearing.