There are two factors - an individual difference and a general matter.
In general, musical sight reading requires specific practice most people are not exposed to. Even if you play instruments fairly well, sight-reading practice is not something you implicitly pick up. I play piano for hobby over 30 years, took lessons time to time, but it's only recently I consciously started practicing sight reading and the effect is remarkable. It's a very specific exercise, different from just practicing a piece.
And there is the individual difference. My son is very visually-oriented, that he can make sense complicated figure at one look, but he's having hard time reading long sentences. Some people may just not good at read music.
In general, musical sight reading requires specific practice most people are not exposed to. Even if you play instruments fairly well, sight-reading practice is not something you implicitly pick up. I play piano for hobby over 30 years, took lessons time to time, but it's only recently I consciously started practicing sight reading and the effect is remarkable. It's a very specific exercise, different from just practicing a piece.
And there is the individual difference. My son is very visually-oriented, that he can make sense complicated figure at one look, but he's having hard time reading long sentences. Some people may just not good at read music.