That's exactly what happened with me many years ago. I was banging away at the floorboards (not a euphemism) at around 11pm or later, not realising that my next door neighbour in our semi-detached was being kept awake. When he called to point this out very courteously, I was mortified to realised what I'd been doing - and stopped immediately, of course!

So it's worth a try: just ask them to please not do such noisy work so late.