Duffy was married to former ballet dancer Carlyn Rosser for over four decades before her death from cancer in 2017.