Finding Habitable Planets Is Even Harder Than We Thought