Max Taylor/The Atlantic

In the heat of the civil-rights fight, when told he shouldn't push too hard for racial equality because of political backlash, Lyndon Johnson famously shot back, "What the hell's the presidency for?"

Ted Olson had a similar question for the Supreme Court Wednesday, pondering why the justices had opted not to take a single case on same-sex marriage this term.

"I agonize over the court not making a decision," said Olson, an attorney on one of the cases. "We give them lifetime appointments, and you’re supposed to make hard decisions. It brings tears to my eyes, actually physically, when I see people suffering ... It seems inhuman to make people wait just for the Court to decide."

Olson—an experienced Supreme Court litigator and solicitor general during the George W. Bush administration—allowed he might have made the case differently with the judges present, but his commitment to the point was clear. Nonetheless, a talk by Olson and Evan Wolfson at the Washington Ideas Forum Wednesday had the tempered optimism of fighters who know the end is in sight.

"We are winning, but winning is not won," Wolfson said. "It's not a done deal until it’s done. It's not going to waft in on waves of inevitability."

Wolfson has been leading the fight for gay marriage for more than three decades; Olson, alongside Democratic lawyer David Boies, has recently been one of its most high-profile advocates. The men were chummy, despite a kerfuffle this spring over Jo Becker's Forcing the Spring, a book some LGBT advocates felt aggrandized Boies and Olson at the expense of Wolfson and others.

There was no rivalry—just a mix of hope and frustration. On the one hand, about two-thirds of states now have gay marriage, and about two-thirds of American citizens live in those states. On the other hand, that means one third do not, and as long as same-sex marriage is not legal everywhere in the United States, they argued, the rights of gay people and their families are painfully compromised.

There are currently three federal appellate circuits where cases are in the pipeline, and Wolfson said the message from the Supreme Court to lower-court judges—in refusing to hear challenges to pro-gay-marriage decisions in lower courts, and in Windsor v. United States—seems to clearly favor marriage rights. On the other, he said that an actual Supreme Court decision would resolve any questions, and noted that this year's appeals were unusual in that same-sex-marriage backers, the winners in lower courts, asked the Supreme Court to hear their cases.

"The strategy our movement has always been using was not that we were going to have to win in every state or every court," Wolfson said. "The strategy was that we had to win a critical mass of support. That would give the comfort and impetus to the appellate courts or the Supreme Court to finish the job."

There are two main scenarios for the future. Either every federal circuit will rule that same-sex-marriage bans are unconstitutional and marriage equality will be the law of the land, or else one or more of the remaining circuits—which tend to be more conservative—will rule in favor of bans, creating a split among federal appellate courts and presumably forcing the Supreme Court to intervene and decide for the nation.

That may sound positive for advocates, but Wolfson and Olson aren't ready to rest just yet. Wolfson recalled how early victories for the movement, in which courts in states including California and Hawaii legalized gay marriage, were reversed when popular ballot initiatives rolled it back. Olson cited the danger of a more conservative Supreme Court in the future. But the main reason to keep fighting, they agreed, was to hasten the end of unequal rights.

"Every day of delay is a day of real hardship for real people," Wolfson said. "It’s time, and the American people don’t need a delay."