(Bloomberg) -- In the corridors and break rooms of Equifax Inc.'s giant Atlanta headquarters, employees used to joke that their enormously successful credit reporting company was just one hack away from bankruptcy. They weren't being disparaging, just darkly honest: Founded in the 19th century as a retail credit company, Equifax had over the years morphed into one of the largest repositories of Americans' most sensitive financial data, which the company sliced and diced and sold to banks and hedge funds. In short, the viability of Equifax and the security of its data were one and the same.
Nike Zheng, a Chinese cybersecurity researcher from a bustling industrial center near Shanghai, probably knew little about Equifax or the value of the data pulsing through its servers when he exposed a flaw in popular backend software for web applications called Apache Struts. Information he provided to Apache, which published it along with a fix on March 6, showed how the flaw could be used to steal data from any company using the software.
The average American had no reason to notice Apache's post but it caught the attention of the global hacking community. Within 24 hours, the information was posted to FreeBuf.com, a Chinese security website, and showed up the same day in Metasploit, a popular free hacking tool. On March 10, hackers scanning the internet for computer systems vulnerable to the attack got a hit on an Equifax server in Atlanta, according to people familiar with the investigation.
Before long, hackers had penetrated Equifax. They may not have immediately grasped the value of their discovery, but, as the attack escalated over the following months, that first group—known as an entry crew—handed off to a more sophisticated team of hackers. They homed in on a bounty of staggering scale: the financial data—Social Security numbers, birth dates, addresses and more—of at least 143 million Americans. By the time they were done, the attackers had accessed dozens of sensitive databases and created more than 30 separate entry points into Equifax's computer systems. The hackers were finally discovered on July 29, but were so deeply embedded that the company was forced to take a consumer complaint portal offline for 11 days while the security team found and closed the backdoors the intruders had set up.
The handoff to more sophisticated hackers is among the evidence that led some investigators inside Equifax to suspect a nation-state was behind the hack. Many of the tools used were Chinese, and these people say the Equifax breach has the hallmarks of similar intrusions in recent years at giant health insurer Anthem Inc. and the U.S. Office of Personnel Management; both were ultimately attributed to hackers working for Chinese intelligence.
Others involved in the investigation aren't so sure, saying the evidence is inconclusive at best or points in other directions. One person briefed on the probe being conducted by the Federal Bureau of Investigation and U.S. intelligence agencies said that there is evidence that a nation-state may have played a role, but that it doesn't point to China. The person declined to name the country involved because the details are classified. Mandiant, the security consulting firm hired by Equifax to investigate the breach, said in a report distributed to Equifax clients on Sept. 19 that it didn't have enough data to identify either the attackers or their country of origin.
Wherever the digital trail ultimately leads, one thing is clear: The scant details about the breach so far released by Equifax—besides angering millions of Americans—omit some of the most important elements of the intrusion and what the company has since learned about the hackers' tactics and motives. Bloomberg has reconstructed the chain of events through interviews with more than a dozen people familiar with twin probes being conducted by Equifax and U.S. law enforcement.
In one of the most telling revelations, Equifax and Mandiant got into a dispute just as the hackers were gaining a foothold in the company's network. That rift, which appears to have squelched a broader look at weaknesses in the company's security posture, looks to have given the intruders room to operate freely within the company's network for months. According to an internal analysis of the attack, the hackers had time to customize their tools to more efficiently exploit Equifax's software, and to query and analyze dozens of databases to decide which held the most valuable data. The trove they collected was so large it had to be broken up into smaller pieces to try to avoid tripping alarms as data slipped from the company's grasp through the summer. In an e-mailed statement, an Equifax spokesperson said: “We have had a professional, highly valuable relationship with Mandiant. We have no comment on the Mandiant investigation at this time.”
The massive breach occurred even though Equifax had invested millions in sophisticated security measures, ran a dedicated operations center and deployed a suite of expensive anti-intrusion software. The effectiveness of that armory appears to have been compromised by poor implementation and the departure of key personnel in recent years. But the company's challenges may go still deeper. One U.S. government official said leads being pursued by investigators include the possibility that the hackers had help from someone inside the company. “We have no evidence of malicious inside activity,” the Equifax spokesperson said. “We understand that law enforcement has an ongoing investigation.”
The nature of the attack makes it harder to pin on particular perpetrators than either the Anthem or OPM hacks, said four people briefed on the probe. The attackers avoided using tools that investigators can use to fingerprint known groups. One of the tools used by the hackers—China Chopper—has a Chinese-language interface, but is also in use outside China, people familiar with the malware said.
The impact of the Equifax breach will echo for years. Millions of consumers will live with the worry that the hackers—either criminals or spies—hold the keys to their financial identity, and could use them to do serious harm. The ramifications for Equifax and the larger credit reporting industry could be equally severe. The crisis has already claimed the scalp of Richard Smith, the chief executive officer. Meanwhile, the federal government has launched several probes, and the company has been hit with a flurry of lawsuits. "I think Equifax is going to pay or settle for an amount that has a 'b' in it," says Erik Gordon, a University of Michigan business professor.
When Smith became Equifax CEO in 2005, the former General Electric Co. executive was underwhelmed by what he found. In a speech at the University of Georgia last month, he described a stagnating credit reporting agency with a “culture of tenure” and “average talent.” However, Smith also saw enormous potential because Equifax inhabited a uniquely lucrative niche in the modern global economy.
In the speech, Smith explained that the company gets its data for free (because regular consumers hand it over to the banks when they apply for credit). Then, he said, the company crunches the data with the help of computer scientists and artificial intelligence and sells it back to the banks that gave Equifax the data in the first place. The business generates a gross margin of about 90 percent. "That's a pretty unique model," Smith said.
And one that he fully exploited. Smith acquired two dozen companies that have given Equifax new ways to package and sell data, while expanding operations to 25 countries and 10,000 employees. Business was good—the company’s stock price quadrupled under Smith’s watch, before the breach was announced—and its leaders lived well. Equifax executives were prone to bragging about their mansions and expensive gadgets. They took lavish trips to Miami, where they stayed in luxury hotels costing as much as $1,000 a night. Last year, Smith's compensation was almost $15 million.
But the man who transformed Equifax was plagued each and every day by the fear that hackers would penetrate the company's firewall and make off with the personal data of millions of people. By the time he gave the speech on Aug. 17, Smith knew of the hack but the public didn't. He told the audience the risk of a breach was "my No. 1 worry" and lingered on the threats posed by spies and state-sponsored hackers.
Not long after becoming CEO, he hired Tony Spinelli, a well-regarded cyber expert, to overhaul the company's security. The new team rehearsed breach scenarios, which involved 24-hour crisis-management squads taking turns to address each given issue until it was resolved. Protocol included alerting the chief of security, who determined the severity of the breach, and then telling the executive leadership if a threat was considered serious.
Apparently, gaps remained. After the breach became public in September, Steve VanWieren, a vice president of data quality who left Equifax in January 2012 after almost 15 years, wrote in a post on LinkedIn that "it bothered me how much access just about any employee had to the personally identifiable attributes. I would see printed credit files sitting near shredders, and I would hear people speaking about specific cases, speaking aloud consumer’s personally identifiable information."Spinelli left in 2013, followed less than a year later by his top deputy, Nick Nedostup. Many rank and file followed them out the door, and key positions were filled by people who were not well-known in the clubby cybersecurity industry. The company hired Susan Mauldin, a former security chief at First Data Corp., to run the global security team. Mauldin introduced herself to colleagues as a card-carrying member of the National Rifle Association, according to a person familiar with the changes.
Two people who worked with Mauldin at Equifax say she seemed to be putting the right programs in place, or trying to. “Internally, security was viewed as a bottleneck,” one person said. “There was a lot of pressure to get things done. Anything related to IT was supposed to go through security." Mauldin couldn’t be reached for comment.
Besides amassing data on nearly every American adult, the hackers also sought information on specific people.
The company continued to invest heavily in state-of-the-art technology, and had a dedicated team to quickly patch vulnerabilities like the one identified by Zheng. Overseeing technology for Equifax was David Webb, a Kellogg MBA and Russian-language major hired in 2010 from Silicon Valley Bank, where he had been chief operations officer. But one former security leader said he finally joined the talent exodus because it felt like he was working with the “B team.”
Lapses in security began to catch up to the company in myriad ways beginning early this year. Since at least Feb. 1, Equifax had been aware that identity thieves were abusing a service that manages payroll data for companies, according to notices sent to victims.
Criminals were feeding stolen Social Security numbers and other personal information into login pages for Equifax Workforce Solutions, downloading W-2 and other tax forms for dozens of employees of clients including Northrop Grumman Corp., Whole Foods Market Inc. and Allegis Global Solutions Inc., a human resources company. They accessed the data freely for over a year to file fraudulent tax returns and steal the refunds before Equifax learned of the incidents. (KrebsOnSecurity.com, a cybersecurity blog, first reported the thefts in May.)
Equifax hired Mandiant in March to investigate any security weaknesses related to the scams, and in notifications mailed to victims throughout the summer, Equifax eventually said its systems weren't breached to acquire the personal data used in the fraud.
However, there are signs that Smith and others were aware something far more serious was going on. The investigation in March was described internally as "a top-secret project" and one that Smith was overseeing personally, according to one person with direct knowledge of the matter.
The relationship with Mandiant broke down sometime over the next several weeks—a period that would later turn out to be critical in how the breach unfolded. Mandiant warned Equifax that its unpatched systems and misconfigured security policies could indicate major problems, a person familiar with the perspectives of both sides said. For its part, Equifax believed Mandiant had sent an undertrained team without the expertise it expected from a marquee security company. A Mandiant spokesman declined to comment on the March investigation.
Although the hackers inside Equifax were able to evade detection for months, once the hack was discovered on July 29, investigators quickly reconstructed their movements down to the individual commands they used. The company's suite of tools included Moloch, which works much like a black box after an airliner crash by keeping a record of a network's internal communications and data traffic. Using Moloch, investigators reconstructed every step.
Once the hackers found the vulnerability Zheng reported, they installed a simple backdoor known as a web shell. It didn't matter if Equifax fixed the vulnerability after that. The hackers had an invisible portal into the company's network. The Moloch data suggests the initial group of hackers struggled to jump through internal roadblocks like firewalls and security policies, but that changed once the advanced team took over. Those intruders used special tunneling tools to slide around firewalls, analyzing and cracking one database after the next—while stockpiling data on the company's own storage systems.
Besides amassing data on nearly every American adult, the hackers also sought information on specific people. It's not clear exactly why, but there are at least two possibilities: They were looking for high-net-worth individuals to defraud, or they wanted the financial details of people with potential intelligence value.
Eventually the intruders installed more than 30 web shells, each on a different web address, so they could continue operating in case some were discovered. Groups known to exploit web shells most effectively include teams with links to Chinese intelligence, including one nicknamed Shell Crew. Some investigators within Equifax reached the conclusion that they were facing Chinese state hackers relatively quickly after analyzing the Moloch data, according to a person briefed on those discussions. If the Equifax breach was a purely criminal act, one would expect at least some of the stolen data, especially the credit card numbers that were taken, to have showed up for sale on the black market. That hasn’t happened.
What’s more, banks are typically asked to shut down all stolen cards if investigators are near certain who is behind a hack. In this case, they still aren’t sure. That’s why on Sept. 11, the FBI asked several major banks to monitor the credit card accounts of small batches of consumers—in one case just 20 people—for suspicious activity. Investigators were still looking for anything that could give them insight into the hackers' identity and motives, according to security experts.
“This wasn't a credit card play," said one person familiar with the investigation. "This was a 'get as much data as you can on every American’ play.” But it probably won’t be known if state hackers—from China or another country—were involved until U.S. intelligence agencies and law enforcement complete their work.
That could take weeks or months, but Equifax is already a changed company. Smith has handed the reins to Paulino do Rego Barros, who will be interim CEO until the board finds a permanent replacement. Smith's departure was preceded by the early retirement of the company's two top security officials, chief information officer Webb and chief security officer Mauldin. Federal investigators are probing suspicious stock sales by other executives that happened not long after Equifax discovered the breach. And lawmakers are making ominous noises about boosting oversight of the credit reporting industry, which is largely unregulated.
“What member of Congress can vote against tighter regulation when every congressional district has nearly half its voters affected by this?” says Gordon, the Michigan business professor. “The lobbying wins when there is no organized group fighting back, but you don't need an organized group when you have 143 million angry people.”