'I could die in here and no one would know for days,' he says. Joking aside, it would be ironic if Vicary-Smith found comfort in seclusion.
He is, after all, the head of an organisation on the frontline of championing consumer rights.
The 42-year-old is only five weeks into his role, and he has a hard act to follow. He replaces high-profile chief Dame Sheila McKechnie, who made her name as a campaigner against 'rip-off Britain'. As debates rage over levels of consumer debt and obesity, Vicary-Smith faces considerable pressure to measure up.
The scale of his remit is vast and comprises two elements. He is the public face of the Consumers' Association, the charity's campaigning arm, and also heads Which?, the organisation's publishing service. Last month both divisions were rebranded as Which? after research revealed that only 5% of consumers were aware of the Consumers' Association brand, compared with 83% who recognised the Which? name.
The challenge facing Vicary-Smith is to continue McKechnie's legacy as a thorn in the side of poor customer service, and to drag the Which? brand into the 21st century. But his 'poacher-turned-gamekeeper' CV (he has worked for Procter & Gamble and Mars and, more recently, for Cancer Research UK and Oxfam), means he is well-qualified for his new role.
An affable family man, Vicary-Smith has an uncanny ability to talk and smile simultaneously, which could come across as sinister were it not for his obvious sincerity. He also radiates a strong sense of social justice, particularly when discussing the misdemeanours of the financial services sector.
'Consumers are deluged with information that is often misleading and certainly inaccessible,' he says. 'A lot of marketing spend goes on telling people to take out new credit cards and personal loans so they can spend, spend, spend. But there is little going on to encourage consumers to save.'
He also bemoans poor levels of customer service, particularly in retail. 'Shops are one of the most complained about sectors in the economy and that is predominantly because of customer service,' he says. 'A lot of the problems are to do with consumers trying to return goods, and shop assistants not knowing what the law is on doing so - or knowing full well what the law is, but not caring to communicate it.'
To address such issues, Which? has launched a campaign called Bite Back, to encourage hundreds of thousands of consumers to register their customer-service gripes. The results will enable the organisation to identify more precisely the areas that could benefit from Which?'s campaigning muscle.
The other challenge facing Vicary-Smith is to broaden the appeal of the Which? brand from its current older, middle-class heartland. He acknowledges it is harder for poorer consumers to access Which? reports and believes that there is room to grow its 950,000 subscriptions. One option is to explore alternatives to the traditional subscription model, such as pay as you go, which would enable consumers to access reports via their mobiles.
Looking ahead, consumers are likely to see a lot more of the Which? brand as Vicary-Smith is considering launching its first TV brand advertising campaign. However, he admits that, as with all charities, Which?'s ultimate aim is to fulfil its remit so well that it's no longer needed. 'Sadly I suspect we won't be out of business in my lifetime,' he muses.
On a more positive note, he believes UK consumers are now more likely to complain than ever - 15m did so last year, according to Which? research.
But he believes there is more potential: 'I'd like to see people seeking information from us at the point of purchase. I'd also like them to see Which? as a friend that is working hard to make all markets fair.'
1988-1991: Consultant, McKinsey & Company
1991-1996: Various fundraising and marketing roles, Oxfam
1996-2002: Director of fundraising and communications, Imperial Cancer
2002-2004: Commercial director, Cancer Research UK
2004-present: Chief executive, Which?