Madison Square was laid out in 1837 and named for the fourth president of the United States, James Madison. There is a large monument dedicated to native Savannahian Sgt. William Jasper in the center of the square, in memory of heroism during the Revolutionary War. At the southern margin of the square, two cannons are featured, marking the starting points of Georgia’s first two highways.

• Savannah College of Art & Design – (south side of the square) – In the spring of 1979, The Savannah College of Art and Design purchased and renovated the Savannah Volunteer Guard Armory in Madison Square to serve as the school’s first classroom and administration building. Today, SCAD, as it is known, is one of the largest art and design universities in the world.
• The Sorrel-Weed House (northwest corner of the square) is famous not only for its dazzling architecture but also due to its reputation as the most haunted house in America. The Greek Revival structure was built in 1841 for Frances Sorrel, a merchant from the West Indies. The story is told that Sorrel’s wife jumped to her death from the second floor balcony after learning that Francis was having an affair with a slave girl. The girl was found hanged two weeks after Mrs. Sorrel’s suicide. An episode of the popular TV show “Ghost Hunters” explored the home in depth. The ghost hunters claimed to have captured an African-American woman screaming in the second floor bedroom where the hanging supposedly occurred. The basement of the home once housed the home’s slaves and contained the kitchen facilities where many of the slaves toiled, unseen by those above. Ghost stories aside, the property has great historical significance. It was once home to Brig. Gen. Moxley Sorrel, the youngest general officer in the Confederate army. Sorrel served as a staff officer to Lt. Gen. James Longstreet and was decorated for heroism at the Battle of the Wilderness in May 1864. Additionally, Robert E. Lee visited the home in 1861 before being appointed to command the Army of Northern Virginia the following year. In 1953, the Sorrel-Weed House was the first house in Georgia to be designated a state landmark. The home is also a National Trust Historic Landmark and is currently operated as a bed and breakfast. The home is available to the public for tours.
• Green-Meldrim House – (west side of the square) This house is one of the nation’s best examples of Gothic Revival architecture and has great historic significance due to its use as headquarters by Gen. William T. Sherman during his occupation of Savannah. The home was built for Charles Green, who came to Savannah from England in 1833 and made a fortune in the cotton trade. In hopes of sparing his new and elaborately constructed home from pillaging by Union troops, Green rode out to meet Gen. Sherman and offered his house as headquarters for the Union forces. The story is recounted in Sherman’s own memoirs: “While waiting there, an English gentlemen, Mr. Charles Green, came and said that he had a fine house completely furnished, for which he had no use, and offered it as headquarters. He explained, moreover, that Gen. Howard had informed him, the day before, that I would want his house for headquarters. At first I strong disinclined to make use of any private dwelling, lest complaints should arise of damage and loss of furniture and so expressed myself to Mr. Green; but, after riding about the city and finding his house so spacious, so convenient, with large yard and stabling, I accepted his offer and occupied that house during our stay in Savannah. He only reserved for himself the use of a couple of rooms above the dining room, and we had all else, and a most excellent house it was in all respects.” – Gen. William T. Sherman The home remained in the possession of the Green family until its sale in 1892 to Judge Peter Meldrim, a former Savannah mayor and past president of the American Bar Association. In 1943 the Green-Meldrim House was sold once again, this time to St. John’s Episcopal Church, and the building now serves as the rectory for the church. The church has graciously made the home available to the public for daily tours.
• St. John’s Episcopal Church was built in 1852 in the Neogothic style and is well-known for its lovely stained glass windows. A nod to the city’s seafaring heritage, a ship’s mast is located in the center of its only spire. The church’s leaders preached unsuccessfully against secession before the Civil War, and so it is fitting that Sherman’s old headquarters, the Green-Meldrim House, now serves as the parish house.