In 1778, John Parke Custis, George Washingtons stepson, bought the Abingdon Estate and other tracts from Gerard Alexander. George Washington Parke Custis, son and heir of John Custis and step grandson of George Washington inherited the northern 1100 acres of this land and began construction of a mansion in 1802 on the high ground overlooking the Potomac River and the City of Washington. The building was finally completed in 1817. At first named Mount Washington, it was soon renamed Arlington after the original Custis estate established before 1680 in Northhampton County, Virginia. G.W.P. Custis, ever conscious of his role as the child of Mount Vernon (he spent his boyhood years living there), made his home a museum of Washington heirlooms and relics.
In 1831 Mary Anne Randolph Custis, the heiress of Arlington, married her third cousin, Lieutenant Robert Edward Lee of the U.S. Army Corps of Engineers. Although born at Stratford in Westmoreland County, Robert Lee had grown up in Alexandria. After his marriage, having no home of his own, he came to feel at home at Arlington. He spent some part of almost every year on leave there and lived there during a three-year tour of duty in Washington, 1834-1837.
G.W.P. Custis died in 1857, leaving his estate to his daughter and naming his son-in-law to be his executor. Thus Lieutenant Colonel Lee became the master of Arlington. It took him a two-year leave of absence from the Army to get the estate in order.
In 1860 Lee returned to his command in Texas and from there, in agony of spirit, witnessed the disintegration of the Union. The theme of his letters of this period was that there was no validity in the constitutional doctrine of the secessionists and no occasion for a revolution, but that a Union that could be held together only by force of arms would have already ceased to exist. If the Union should be dissolved in this way, his only obligation would be to defend Virginia. Early in 1861 Lee was ordered back to Washington, where in April he was offered command of the newly levied Union army. He declined, stating that he could not take part in an invasion of the Southern States, although he disapproved of their course of political action. After resigning his commission in the U.S. Army, he accepted the command of seceded Virginias military forces. Summoned to Richmond, he left Arlington on April 22 never to return to the mansion.
In May 1861 the Arlington estate was occupied by federal troops. Three years later, the estate was confiscated for failure of the taxpayer to appear in person to pay $92.07 in taxes. The property then was purchased by the U.S. Government at public auction for $26,800. The seizure of the estate was challenged in the courts in 1873, and after ten years of litigation, Custis Lee, son of Robert E. and Mary Anne Randolph Lee, received $150,000 compensation. The federal government received clear title to the lands it had seized and occupied twenty-two years earlier. In 1955 Arlington House was officially designated by an Act of Congress as a permanent national memorial to Robert E. Lee.
On December 4, 1863, The Freedmans Village settlement was established by the federal government on the Arlington Estate, south of the cemetery and the mansion. This village was created to provide homes and employment opportunities for the numerous slaves who had gained their freedom by fleeing from nearby Maryland and Virginia and entering the District of Columbia, in which slavery had been abolished on April 16, 1862. Freedmans Village consisted of 100 frame houses, a school with five teachers, and two chapels. The chapels were the predecessors of the Mount Zion Baptist Church at 19th Street, South and Mount Olive Baptist Church now located on 13th Street, South. Most of the residents of Freedmans Village later moved to other sites in the County.