A brutalist concrete swimming pool building and a dilapidated glass orchid house, were swept away to be replaced by a residential annex and snooker room for the writer Roald Dahl.