Staff at the Waterstone's bookshop in Union Street are in a 30-day consultation period with the company after being told last week that the outlet would close on August 14, with HMV moving into the 7,500sq ft retail space immediately.
Waterstone's has been on the site since 2004, providing a wide range of books and a Costa Coffee shop.
The Torquay store annually supplies the Ways With Words literary festival at Dartington with books.
It is not known whether this will continue, although it has been suggested that the Newton Abbot Waterstone's will take over the role.
No one from Waterstone's was able to comment on the closure, although a spokesman confirmed that staff had been spoken to.
HMV, part of the same company as Waterstone's, is expected to undertake a massive refit at the store, costing more than £100,000.
Lucy Ball, Torbay Town Centres Company chief executive, said: "It's a great shame that we're losing the name of Waterstone's from our high street.
"But at least HMV will have a more prominent home, which they've wanted for a long time.
"It would have been tragic if a store that size was left empty, so it's swings and roundabouts really.
"It would be good if Costa Coffee is retained at the site, because its another big high street name, but we will have to wait and see on that one."
Matthew Clarke, who owns the independent Torbay Bookshop in Paignton, said national literature outlets were finding it hard to compete with internet sites such as Amazon.
He said: "It's different for independent bookshops, many of which are seeing their sales go up. But we're all having to work hard to stay in business.
"It's devastating news for the Waterstone's staff, who are specialised in their field."
Author David Lawrence Jones, who launched his debut novel The Amazing Adventures of Bradley Baker at the Torquay outlet last year, said: "We are losing a great book store and, although WHSmith will supply Torquay, it is very sad to lose a very dedicated group of people who know and care about books and their readers.
"Highlighting the closure won't overturn the decision, but it will demonstrate what the store staff have done for the community."