I noticed that my ~/Library/Application Support/Address Book folder was pretty big; more than 420 MB. On an SSD, a half a gigabyte is a big hit for something that just records contact information.
So I looked in that folder, and found some interesting sub-folders. First, at the top level, there was a 69 MB “Images” folder, which contains 2,672 items. Second, there was a “Sources” folder, which contained a sub-folder with one of those long names with numbers and letters; presumably my Mac’s UUID. In that folder was another Images folder, this one at 345 MB, containing 13,353 items!
So I figured I could try and delete these folders and see what happened. When I re-opened Address Book, the application automatically re-loaded images for many of my contacts. These are contacts for whom I have iChat accounts or Twitter accounts. The new folder is 4 MB, and contains 161 items.
I don’t know why the other folders contain so many items, but if your Address Book folder takes up a lot of space, try removing these folders (don’t delete them right away, just in case) and see if this slims them down a bit, as it did for me.