I, too, would recommend the Old City center, like around Opera, Puerta del Sol, even Gran Vía. I wouldn't go much further if you'll be pushing/carrying a small child with you everywhere. Plus, you may want someplace close by where you can take frequent breaks in your sight-seeing day.

Saludos, MadridMan