If I absolutely HAD to I'd choose Toledo over Segovia, at least if you've never visited either one. I second the motion to spend the night - you really get the feel for the town after all the day trippers have left in the evening and before they arrive in the morning.
If you could possibly cut any time in Madrid, I'd try to visit Segovia also. Although I really enjoyed Madrid for it's nightlife, tapas, and the Palace, I'd rather spend the time in Segovia and Toledo. That's just my opinion - I'm not a big museum person
but if you are Madrid is a real treat too.