The town was originally named Ellicott Mills for the three Quaker brothers who founded it as the site of their gristmill. Charles Carroll of Carrollton, whose Doughoregan Manor can still be seen nearby, lent financial help to the Ellicotts and the town eventually became the site of ironworks, rolling mills, and the first railroad terminus in the United States. The famous Tom Thumb locomotive race with a horse took place near here. Many of the town's original stone houses and log cabins, on hills above the Patapsco River, have been preserved.