I believe it's safe in the daylight except a minor danger of pickpocketing.
As for the night, I have walked recently almost alone with a friend only (it can get kind of romantic), and I became a little worried about the rumours, so we came back.
However, Madrid majors are conservative, and they have had a campaign of harassment towards prostitutes in other areas and parks of the city that could have had drown some to Retiro. They only mention that the park was not safe at night when they decided to close, but for me it's absoultely different if prostitutes go there (unpleasant as it can be) from drug sellers or assaultants.
That said, I never saw prostitutes nor violence, maybe because this mesures were early, but I don't go there very often. But I believe there is not much (or any) of this.
To summarize, besides of taking care of your wallet or purse, I would go there in the daylight and in the night in the areas with more people. Not that they would help, but the criminal is not sure and usually refrains himself.