The city is DEEPLY stretching the definition of "building" under the International Residential Code - "Any structure used or intended for supporting or sheltering any use or occupancy." Since the thing can't be occupied it would seem not to be a building at all. This is also in a city that per their own codes doesn't require permits for "swingsets and playground equipment", which this could possibly be described as, since it is a fun structure for the purpose of enjoyment by kids and their families. If they don't issue permits for mailboxes, birdhouses, milk boxes, tool boxes, or garden hose reels, then they shouldn't be lumping this tiny toy library into an inappropriate bucket.
And to the comment that they "have to do something" because there were complaints - that just isn't true. Cities aren't under any obligation to find the 5 most obnoxious people in their community and then do everything that group says (unless they are elected, haha!). My mother grows tall native prairie grasses in her Colorado lawn, and when a complaint was received about "uncut grass" in her lawn the city talked with her, learned that the grasses were native species, and added a specific exception in the grass maintenance ordinances that allow for native plants.