I see what you're saying. I don't know the pazo you've mentioned, but I can think of one that's not far from Ourense. It's called the Pazo de Bendoiro (
www.pazodebendoiro.com) in a town called Lalín, about 50 kilometers from Ourense on the main road to Santiago. Lalín doesn't have a lot to offer the tourist (I hope my wife's family isn't reading this!) but the pazo is really pretty and you can also go and visit the gardens of the Pazo de Oca (not a hotel, by the way, but one of the nicest pazos in Galicia). There's also another town called Agolada which has its own authentic medieval market which comes to life once a year...I think around those dates too, but I'd have to check. Halfway between Ourense and Lalín, you can visit the monastery of Oseira known as the the "El Escorial of Galicia". So, there are some nice things to visit, but it's just plain pretty countryside and very relaxing.
If you're looking for something completely different and a little farther away, you can always try Ribadeo up on the northern coast of Galicia near the border of Asturias. Beautiful town and great beaches too. A lot less crowded. I always mention this place to people and you'd think I was making a commision, but it's just one of those places people don't know but should.
Heading south, you could try someplace like Tui. I've never been there, but have heard great things about it.
I just remembered another one. Ribadavia, near Ourense and home of Ribeiro wine. Not far from the town you can stay at a winery called Viña Mein.