The earliest reference to the name "Tower Hamlets" was in 1554, when the Council of the Tower of London ordered a muster of "men of the hamlets which owe their service to the tower". This covered a wider area than the present-day borough, and its military relationship with the Tower is thought to have been several centuries earlier than the 1554 record. 
Seit 1686 umfassten die Tower Liberties nicht nur Tower und Tower Hill, sondern auch drei weitere Flächen im Stadtgebiet. Seit dem frühen 19. Jahrhundert schränkten diverse Polizeigesetze die Rechte der Liberties ein und übertrugen die Befugnisse an die Stadt London.
The Charter was issued from the Norman Chapel in the Tower of London in 1100. After a traditional greeting, the Charter of Liberties contained fourteen declarations, summarised as follows: Henry, king of the English, to Bishop Samson and Urso de Abetot and all his barons and faithful, both French and English, of Worcestershire, [copies were sent to all the shires] greeting.