As the Spanish would put it, "Por poder se puede". Yes, it can be done, especially if you leave early enough, but it is a lot, I have to admit. The good thing is that it's all highway from Madrid to Segovia (60 miles) so it doesn't take that long. The last twenty miles is a toll highway, by the way.
But I'm inclined to say it isn't a good idea.
If you're dying to see Ávila, then go for it, but you might want to reconsider. If I had to choose between the two, I'd definitely pick Segovia. Not that Ávila isn't nice (it's pretty and the walls are impressive and all), it's just that it doesn't have as much life as Segovia. Plus Segovia has the triple combo of the aqueduct, cathedral and castle (alcazar) all there in the same place, not to mention the squares, streets and restaurants...you'll need some time to get through it all.
Another thought might be for you to go to Segovia in the morning, have lunch and then visit La Granja afterwards on your way back. It's just eight miles away from Segovia and it has a beautiful 18th Century palace and gardens at the foot of the mountains. The town where the palace is, San Ildefonso, is nice too. Then you could head back to Madrid by going over the Puerto de Navacerrada, this is the area where "For Whom the Bell Tolls" is supposed to have taken place.
You'd be retracing four or five major periods in Spanish history in a single day!
This is a route I've done numerous times for visitors and it has always worked out well. And even so, it is a full day. The good thing about going to La Granja is that in a sense you are working your way back to Madrid. If you go on to Ávila, you're actually going farther away.
As I said, it can be done, but...