The earliest reference to the name "Tower Hamlets" was in 1554, when the Council of the Tower of London ordered a muster of "men of the hamlets which owe their service to the tower". This covered a wider area than the present-day borough, and its military relationship with the Tower is thought to have been several centuries earlier than the 1554 record. 
Seit 1686 umfassten die Tower Liberties nicht nur Tower und Tower Hill, sondern auch drei weitere Flächen im Stadtgebiet. Seit dem frühen 19. Jahrhundert schränkten diverse Polizeigesetze die Rechte der Liberties ein und übertrugen die Befugnisse an die Stadt London.
Much of this was intensively rebuilt, cramming the extra housing required by London's burgeoning population into every corner. On the death of Edward VI in 1553, Lady Jane Grey was received at the Tower of London as queen, but the lord mayor, aldermen, and recorder soon changed course and proclaimed Mary I of England queen instead.