The place had been in possession of the Bayard family, but William Bayard, who lived there at the time of the Revolution, was a Loyalist, and his house on Castle Point was burned down and his estate confiscated.

After the Revolution Stevens acquired the property.

He laid it out as a town in 1804, made it his summer residence, and established there the machine shops in which he and his sons carried on their mechanical experiments.
These shops were easily the largest and bestequipped in the Union when in 1838 John Stevens died at the age of ninety.

The four brothers, John Cox, Robert Livingston, James Alexander, and Edwin Augustus, worked harmoniously together.

"No one ever heard of any quarrel or dissension in the Stevens family.