It is a great idea but you may need to diversify. A friend of mine has started doing a cakes/cupcakes/cookies company in South Africa for parties etc and it is fab. So you could have the shop based business but also cater for parties and deliver from the shop.
You would need to have more things to buy in the morning: Muffins, flapjacks, croissants, etc and be open from 7am.
I am wheat intolerant and lactose intolerant so I may not be very representative of your target clientelle anyway but do sometimes indulge in cupcakes and bare the consequences.
Fresh fruit juices would be nice too.