I'd avoid Borough market at the weekend as it has never been the same since Jamie Oliver put it back on the map.
Surprised that no one had mentioned Peckham on a Sunday outside the library.
East Street market is another good alternative as mentioned as is Lewisham (or Penge or Catford).
We had a farmers market a few years ago on Dartmouth Road but too few of you out there used it (see regular posts criticising the lack of people shopping locally.
And of course also worth visiting Wood Vale where you can admire the ex-railway line, ex-post office, ex hair dressers, ex offie an of course pay a pilgrimate to the Moor Park Tavern.