I do this and I definitely get spam to addresses associated with e-mails leaked or sold in bulk to the highest bidder. I rarely _see_ it as most of the time "unsubscribe" link works (for sold e-mail lists, as the buyers usually try to maintain some sort of decency) and for those where it is useless (shameless "enlargement" type emails, usually from stolen email databases) it is usually classified correctly as spam by an e-mail provider or e-mail client.

So, I never had to explicitly filter e-mails out by "To:" field, but using this system still gives me some sense of control.