Dolley Madison (1804). Gilbert Stuart (American, 1755–1828). Oil. White House. Dolley Madison, portrayed with her hair arranged in a neoclassical style and wearing a fashionable high-waisted Empire-style dress, was the popular wife of James Madison, then Secretary of State under President Thomas Jefferson. She often served as hostess at the White House for the widowed President.