The map of 1874, shows this plot as part of one of the 2 gardens belonging to the last house on Westbourne Drive. (Fairly likely to be the same house as standing now.)

There is a driveway leading to the back of this house from Perry Vale, and this was sold off shortly after, as the map of 1875 shows that a little terrace cottage and garden strip had been developed on the drive. (This is fairly likely to be the same terrace cottage next to the empty plot now.)

The map of 1914 shows a separate empty plot as is today.
I've nothing to hand from that point on - something may have been built then flattened in the war, but your neighbour thinks this is not the case.

So it is more than possible that planning permission has been turned down for this empty plot for a hundred years.