rshdunlop,
You should consider an extra change to the Jubilee Line at Westminster (or Green Park if coming from further west than South Ken) and then Canada Water interchange. The District line is very slow through zone 1.

But if coming from Victoria I would probably prefer the journey on a single train rather than three trains for the sake of 5 minutes. But it does depend on the timing as the direct trains only run twice an hour. And the non-underground service is cheaper.

I once did the calculations in preparation for leaving a meeting in Victoria and there was only about 10 minutes in the hour when heading for Victoria station would have been worthwhile according to my personal cost/time/laziness analysis.