By Monday afternoon, shares in the firm, which issues a trading update on Thursday, were down 15% at £15.42.
A petition, launched by employees, accuses 62-year-old Mr Kelvin of inappropriate comments and behaviour.
The firm says a "thorough independent investigation" will be conducted into the allegations, which it disputes.
'Part of the culture'
The petition on the workplace website Organise says that more than "200 Ted Baker staff are breaking the silence", and speaking out about "50 recorded incidents of harassment at Ted Baker".
"It is part of a culture that leaves harassment unchallenged," the petition claims.
But the company says hugs are "part of Ted Baker's culture, but are absolutely not insisted upon".
It said Mr Kelvin greeted many people with a hug, "be it a shareholder, investor, supplier, partner, customer or colleague".
Staff claim that as well as engaging them in unwelcome embraces, the brand's founder asked young female members of staff to sit on his knee, cuddle him, or let him massage their ears.
They also allege that millionaire Mr Kelvin, who launched Ted Baker more than 30 years ago, regularly uses sexual innuendo.
'Robust HR policies'
But Ted Baker says that the matters raised in the petition were "at odds with the values of our business and those of our CEO".
Another complaint made in the petition is that previous complaints have been ignored.
"Please set up a way of reporting harassment to an independent, external body - HR has done nothing with the reports of harassment to date," says the petition.
In response, the company said there were "very clear and robust HR policies in place to provide all employees with a wide range of reporting options for any issues that they might encounter, including a completely confidential and anonymous, 24/7 helpline".
Ted Baker, named after a fictitious alter ego of Mr Kelvin, opened its first store in Glasgow in 1988 and now has 544 outlets worldwide selling fashion and accessories.
Mr Kelvin is worth £522m according to the Sunday Times Rich List.