Some of the unique wedding gift suggestions here also fulfill another criteria – affordable.
This list should be do-able for under ~$20, but more or less depending on your decisions. Some are even free! (Except for your time.)
New City Info Pack
Personalized Address Stamp
Post Card Collection
Recipe Card Box