Please do contact the police on 101.

Given that the man is known to them, it is helpful for them to be aware of any other activities in the local area. Much better for them to be aware now rather than after something nasty happens. And if there is nothing to be concerned about, then at least they will be aware of this and may be able to give you some reassurance.

Police are reliant on information from members of the public and there is usually plenty they can do to prevent the situation escalating, if they are aware of such issues.