Julia Pierson called it quits Wednesday, stepping down as director of the U.S. Secret Service just 18 months after she took the job and one day after her dismal performance in front of Congress failed to appease lawmakers angry about the recent White House security breach. In announcing the news, Secretary of Homeland Security Jeh Johnson promised an independent review of the failures that allowed Omar Gonzalez to hop the fence at 1600 Pennsylvania Ave. and dash into the executive mansion, with a “review of broader issues concerning the Secret Service” a possibility as well.

The decision to launch an external review was long overdue. But Johnson shouldn’t need an outside panel to tell him what should be obvious by now: Concerns about the Secret Service extend far beyond the Sept. 19 incident at the White House, and Pierson’s departure alone won’t fix problems that began before her tenure. While this week’s string of high-profile embarrassments has made that abundantly clear, it shouldn’t have taken someone hopping a fence to put the spotlight on the troubled agency.

The Department of Homeland Security’s Inspector General’s Office provided a rare—albeit somewhat redacted—peek inside the Secret Service this past December when it released a report following the agency’s Colombian prostitution scandal the previous year. The report should have raised a giant red flag for anyone paying attention. Instead, it went largely overlooked thanks to its official conclusion that there was nothing to suggest that “misconduct [was] widespread” within the Secret Service.

Dig past the executive summary and you’ll find a huge disconnect between the IG’s conclusion and the evidence that comes before it. The report found that between 2004 and 2012, there had been nearly 800 documented cases of misconduct within the agency, ranging from disruptive behavior to problems with drugs or alcohol. The report doesn’t provide details of the incidents, but 257 of the offenses were classified as “neglect of duty,” a rather broad catchall that includes both the failure to carry out orders on time and the failure to carry them out at all. Other offenses that fall under that category include: “sleeping on the job,” “inattention to duty,” and “negligent or careless performance of assigned duties.”

In other words, a member of the Secret Service has been caught sleeping on the job, either literally or figuratively, on average nearly every other week for the past decade. There’s reason to believe that the true total is higher: An electronic survey given to members of the Secret Service found that 18 percent of respondents—almost one in five—said they believe “management tolerates violations of misconduct.” While the Secret Service has taken a few small steps to address the problems documented in the report, it’s telling that officials were so eager to believe there wasn’t anything to really worry about.

It’s likely that many of the offenses that are enumerated in the IG report turned out to be relatively harmless. But we could have said the same thing about any number of the mistakes made on Sept. 19 if it weren’t for the fact that they overlapped with Omar Gonzalez’s decision to hop the White House fence, dash across the North Lawn, barge through the North Portico’s doors, and push past an on-duty officer before finally being tackled by a serendipitously placed off-duty agent who had just finished his work day.

Perfection may be an unfair standard to demand of the agents and officers willing to give their lives in the line of duty. But it’s clear that the Secret Service hasn’t come anywhere close to meeting that standard, and hasn’t seemed all that concerned about that fact. For all the agency’s behind-the-scenes screw-ups, it took a prominent public event for anyone to notice this broader dysfunction. According to the Secret Service, in the past five years, 16 people have jumped the White House fence, including six this year alone. All were nabbed by security. Yet, it was the agency’s failure to extend its shutout streak by one last month that rightly prompted a crisis of confidence in the Secret Service, one that ultimately cost Pierson her job. The release of an IG report 10 months ago that documented hundreds of individual failures, though, prompted the nation to shrug its shoulders and move on.

Of course, reports don’t often drive action in Washington, and sensational anecdotes do. The Secret Service has offered plenty of those, too. Earlier this year an agent was found drunk and passed out in a hotel hallway in Amsterdam the day before Obama was set to arrive. In 2012 a dozen agents spent a boozy night in Cartagena in the company of prostitutes. And in 2009 an agent on a presidential detail in Asia had to be retrieved by a superior after spending a drunken night in a Thai brothel. The theme of alcohol isn’t a coincidence: Excessive drinking was one of a handful of problems a plurality of respondents labeled “more than isolated but less than systemic” in the IG’s survey.

Take all that together, and no one should have been shocked to learn this past week that it took officials four days—and the help of a housekeeper—to figure out that someone had deliberately unloaded more than half a dozen rounds at the White House in 2011. Or that only a few days before Gonzalez hopped the fence, the Secret Service had unknowingly allowed Obama to share an elevator with a gun-toting security contractor with a criminal record.

The Obama administration is hinting at major reforms to an agency that clearly needs them. Let’s hope it delivers. Because while Pierson’s resignation couldn’t come fast enough given her misleading testimony before Congress, there’s little to suggest her departure will make the Secret Service any less secretive—or the president any safer.