That’s it in a nutshell!
If you know somebody’s wallet address you can use the blockchain to determine their balance and all their inbound and outbound transactions. It’s anonymous only in the sense that the wallet address cannot be linked to the identity of an individual unless said individual publicly discloses their address.
For example this is the wallet address of the Silk Road seized coins.
There’s also an interesting Reddit thread on the anonymity of Bitcoin here. Hint it’s not anonymous.