Yeah, but in my experience, the PM's response to that will be "can we get it done sooner?"

If the estimated time aligns with what they want, then you're fine. But that's rarely the case and when things slip the code cleanup is pushed to the back burner.