Indeed, that's the original location that's programmed into the firmware (the chips are manufactured I guess in Taiwan). The good news is that if you're seeing the 24/121 your software and your hardware are working correctly.
It would seem that you haven't picked up a satellite yet. I would recommend taking the GPS out to a very open location where you can see as much of the horizon as possible (top of a roof if you live in an apartment, maybe on your dashboard in a car, at least outside). Power up the Arduino and GPS, and give the GPS 1-10 minutes depending on how much you can see (let it sit there in the open sky for a few minutes).
The first satellite acquisition is always the longest, which is why you can see 1-10 minutes. It will acquire the almanac, and then your subsequent power-ups will be "hot starts" meaning that it's just looking for some satellites, it will be a really quick location update after that.