Different pollen have different weights. If you're seeing yellow dust laying on the ground then it's likely to be a heavy pollen that won't bother many people unless they actively stir it up. Example: People with pine allergies aren't really bothered unless they play in the pine needles and stir it up.

Not familiar with the biology of the matter. But I would assume there are advantages and disadvantages to the weight of the pollen in how it disperses and pollinates.