This is really a difficult question to ask.
However I will risk myself to give you some general tips on what to expect or to be aware of.
Japanese tourists are an easy target for various reasons: they can be easily identified (there are not many asians in Spain), they are very confident, and they never mind to carry on bags, cameras, video-cameras and maps.
Pickpocketers usually attack japanese tourists (for them you will appear a japanese no matter you are from Taiwan) in tourist areas.
In any case be aware that the pickpocketers aren't generally agressive, and that violent crimes are very unfrequent. If you are aware of your surroundings while in tourist areas and don't carry visible bags, cameras, etc you should not have any problem.
Infants are never target of any attacks.
"is ok we have our son sit on the stroller while walking on the street?"
I would say yes, it is something commonly done by us spaniards
Regards and enjoy your visit.
Fernando