Forest Hill is very safe. My wife often walks the mile from the station to our house late at night and I don't ever recall feeling threatened.
Of course it is still London so crime does happen but not to any great extent.