The sign relates to the Southwark neighbourhood area which is adjacent to this bit of Lewisham. This area is right on the borders of the two boroughs. Peckham Rye (address and Park) itself is quite a bit further down Forest Hill Road. One Tree Hill and Brenchley Gardens belong to Southwark. All three green areas can be joined together in a nice green walk.

To largely reiterate what others have said I don't think you have anything at all to worry about. Nowhere is completely safe, but after living near where you mention for close to 25 years I can't say I have ever felt threatened. I walk through Brenchley Gardens often and then loop round into Peckham Rye. Both are very pleasant and popular. If you like a bit of a climb, One Tree Hill is also a great place to sit and ponder whilst looking at the London views and the Horniman is close by too. I think there is a surprising amount of greenery within a 10 minute walk (or less) for somewhere so close to London.