Ive lived near to the Bird in Hand for around five years and aside from having to put up with the constant mess created around the pub, and the language (I once caught 5 'f's walking past) - havent really felt threatened.

Forest Hill is a lot more friendly than in most of the other places around London that I've lived. If you do move Im sure you wont regret it.