100% yes in my opinion. Spent a whole day there once but would love to have spent longer.
The old sloping Plaza is fascinating, the castle has spectacular views and the church has a Goya (fresco?). The views over the village from the church are great.
When I visited, there was a wild fiesta going on - only it turned out not to be a fiesta, but a film crew filming a pretend fiesta for a Lenor fabric conditioner advert
It was fun watching that from the cafes round the Plaza.
Oh, and the Chinchon aniseed drink thing is pretty nice too
It's a great place to be totally relaxed - it's tiny, but enough to keep you interested for a stopover.
Jo