Davis was admitted to the Virginia bar in 1895 but returned to his birthplace two years later. In 1899 he was elected to the West Virginia House of Delegates and in 1910 to the U.S. House of Representatives. From 1913 to 1918 he served as solicitor general of the United States, and he was one of President Woodrow Wilson's advisers at the Paris Peace Conference following World War I (1919). He also served as ambassador to Great Britain (1918-21), after which he accepted a partnership in a New York law firm.

At the Democratic National Convention of 1924, neither the supporters of New York governor Alfred E. Smith nor those of William G. McAdoo would yield their votes in order to settle on a presidential candidate. After 102 ballots the party compromised by choosing Davis, who went down to overwhelming defeat that fall before Republican Calvin Coolidge.

Returning to private law practice, Davis appeared in 140 cases before the U.S. Supreme Court, more than anyone had argued to that time. A capstone of his career was his victory in 1952, when the Supreme Court ruled that President Harry S. Truman had exceeded his constitutional powers in seizing control of the nation's steel mills. His legal career is also noted for his defense of the "separate but equal" doctrine in Brown v. Board of Education. Davis died March 24, 1955, in Charleston, S.C.