If you go to Segovia by bus, the previous stop is La Granja, home of a former Royal Castle and the royal glassworks and glasswork museum, which are worth a visit. In Segovia itself, just wandering the streets and seeing the 2,000-year-old aqueduct are wonderful.
In Madrid itself, I suggest the Prado Museum and especially the Reina Sofia museum of modern art. If the weather is nice, sit in an outdoor cafe opposite the lake in the Retiro park, then wander around the park and see the street performers and the park itself. If you know Spanish and enjoy theater, Madrid offers a great many affordable options.
Be sure to pick up a copy of the weekly periodical "La guía del ocio" ("The Leisure Guide"), which includes listings for restaurants, exhibits, bull fighting, performing arts, festivals, etc. The back few pages of each issue are in English, and list attractions appropriate for English speakers.
Of course, it all depends what your tastes are. A friend of mine loves going to shoe shops, and was in seventh heaven in Madrid.