1. I would 100% forget which symbol goes with which site.
2. The length I choose for my password may be too long/short for a given site.
3. Some sites require special symbols; some sites forbid them. Now I have to pick out my symbol based on the site's requirements, not ease of memorization.
One solution to the site restrictions regarding symbols would be that you generate a non-symbol password, and a symbol substring. You add the symbol substring when you know that the site requires symbols (or if the non-symbol password fails).
I use passwordcard too and when I get a site that clashes with my algorithm, I give them one of 4-5 passwords that I used to reuse everywhere before I implemented the scheme. I have a 6-symbol password, a 10-symbol password with special characters, sure they've also been used on a dozen other sites but if their super special requirements make me extra secure who am I to blame them, I'll use my super special passwords that fit :^)
I could use a password manager but I don't want to be dependent on one piece of software, there's too many failure modes. It's already bad enough that all my accounts share a limited set of email addresses.
The main issue I have with schemes like this is that there's no repository of global identifiers for websites and services. I can build a password for blizzard.com, starting from say "bl", then I forget about it and five years later their website is now activision.com, and I wonder why I can't log in with a password built from "ac". It's a minor issue since password resets are a thing and rebrands are rare but still..
http://www.passwordcard.org/en