Located on Shutes' Folly, an island one mile offshore from Charleston, the Castle fort was built over the ruins of an older fortification called 'Fort Pinckney'. The original log and earthen fort, named after the Revolutionary War hero Charles Cotesworth Pinckney, was built beginning in 1797 and was intended to protect the city from a possible naval attack when war with France seemed imminent. Completed in 1804, it saw no hostilities and was virtually destroyed by a severe hurricane in September of that year. A replacement brick and mortar structure called 'Castle Pinckney' was erected in 1809–1810 and was garrisoned throughout the War of 1812, but saw no action. Afterwards, Castle Pinckney was abandoned and fell into disrepair.
Two decades later, a sea wall was completed and the fort was re-garrisoned during the Nullification Crisis of 1832, when President Andrew Jackson prepared to collect a controversial tariff using military force if necessary. After that brief period of activity, the fort again fell into disuse and was primarily a storehouse for gunpowder and other military supplies.