In addition to Fitzroy Square and nearby Fitzroy Street, there are numerous locations named for the FitzRoy family and Devonshire/Portland family, both significant local landowners. Charles FitzRoy was the grandson of Charles FitzRoy, 2nd Duke of Grafton , hence Grafton Way and Grafton Mews.
Lord Hertford married Lady Isabella Fitzroy, daughter of Charles FitzRoy, 2nd Duke of Grafton, on 29 May 1741. Her grandfather was Henry FitzRoy, 1st Duke of Grafton (1663-1690), an illegitimate son of King Charles II. By his wife he had thirteen children: Francis Seymour-Conway, 2nd Marquess of Hertford (12 February 1743 – 28 June 1822)