Clarke, who became chief executive in 2011 and joined Tesco in 1974, will step down on 1 October but will continue in a support role until January 2015.
In a statement, Tesco said that trading conditions had been "more challenging" than the supermarket had anticipated at the time of its first quarter interim management statement on 4 June.
Sales and trading profit are "somewhat below expectations", said Tesco, thanks to a combination of a weak overall market and increasing investments being made to improve the brand’s customer offer and build long term loyalty.
It follows a number of senior management changes at Tesco over the past six months. Earlier this year, finance director Laurie McIlwee stepped down after 15 years at the company, while the top marketing job was handed to newly-appointed chief customer officer Jill Easterbrook. Former CMO Matt Atkinson was appointed chief creative officer.
Clarke’s replacement Lewis has been with Unilever since 1987, starting as a trainee at Lever Brothers. He held a number of marketing roles, including marketing operations manager and marketing director for Unilever’s River Plate business in Argentina, Uruguay and Paraguay.
Following a string of senior management positions, Lewis became chairman of Unilever UK and Ireland, the president of the Americas in 2010, before taking up his current role in 2011.
Tesco chairman Sir Richard Broadbent said Clarke had agreed with the board that this was an "appropriate moment" for the supermarket to hand over to a new leader: "Philip has done a huge amount to set a clear direction and re-position Tesco to meet the rapid changes taking place in the retail market.
"He has achieved a great deal across all areas of the business in the face of considerable pressures. The Board are deeply grateful to Philip for his contribution to Tesco, over the last four decades, as well as more recently as Chief Executive. His has been an outstanding achievement."
Clarke added: "Having taken the business through the huge challenges of the last few years, I think this is the right moment to hand over responsibility and I am delighted that Dave Lewis has agreed to join us."