No one is using 100% of the space in the rest of the building either. Restrooms aren't a waste of space even if they're not constantly filled. Not everything has to be at 100% utilization to be worth the investment.
But as you make a building larger the number of restrooms increases linearly. A building twice the size uses twice the space for restrooms.
Say you design a 50 story building and it has one bank of elevators that all run from 1 to 50. What if you wanted to make it twice as high, and have the same elevator capacity for the top half?
You can add another identically sized bank that starts at the ground floor, skips up to 51 and then covers from there to 100.
In the top half of the building you used the same fraction of space for your elevator shafts as you'd used originally. Great! Except the elevator shafts to get there had to go through floors 2-50, and they ate up as much space there as the original elevator bank did. The building has 2x the original volume, but you're spending 3x the space on elevator shafts so you didnt double the useful floor area.
As you try to design taller and taller buildings, the increasing cost of vertical circulation gets worse and worse.
That's not a counter. No matter what your goal utilization is, your ability to deliver that per square foot drops tremendously. If restrooms were like elevators then adding a bunch more floors would suddenly require all two-stall restrooms in the building to be five times as big.