Apache had to scramble at the beginning of December 2021 to be ready to release patches for Log4Shell when it publicly disclosed the situation on December 9 of last year. As a result, researchers quickly found edge cases and workarounds to the patches, and Apache was forced to release multiple iterations, which added to the confusion.
“This thing was everywhere, truly everywhere,” says Jonathan Leitschuh, an open source security researcher. “Attackers were jumping on it, the security community was jumping on it, payloads were flying everywhere.”
Researchers say, though, that Apache’s overall response was solid. Nalley adds that Apache has made changes and improvements in reaction to the Log4Shell saga and hired dedicated staff to expand the security support it can offer to open-source projects to catch bugs before they ship in code and respond to incidents when necessary.
“In a short period of time, two weeks, we had fixes out, which is great,” Nalley says. “In some ways, this is not a new situation to us, and I would love to say we dealt with it perfectly. But the reality is, even at the Apache Software Foundation, this highlighted what a responsibility we have to everyone who consumes our software.”
Going forward, the more concerning aspect of the situation is that, even a year later, roughly a quarter or more of the Log4j downloads from the Apache repository Maven Central and other repository servers are still full of vulnerable versions of Log4j. In other words, software developers are still actively maintaining systems running vulnerable versions of the utility or even building new software that is vulnerable.
“The reality is that the majority of the time when people are choosing a vulnerable open-source software component, there’s already a fix available,” says Brian Fox, cofounder and chief technology officer of the software supply-chain firm Sonatype, which operates Maven Central and is also a third-party Apache repository provider. “I’ve been around for a long time, and I’m jaded, but that really is shocking. And the only explanation is that people really do not understand what’s inside their software.”
Fox says that after the initial scramble to address Log4Shell, version downloads in Maven Central and other repositories hit a shelf where roughly 60 percent of the downloads were of patched versions and 40 percent were still of vulnerable versions. Over the last three months or so, Fox and Apache’s Nalley say they’ve seen the numbers fall for the first time to roughly a 75/25 percent split. As Fox puts it, though, “After a year, a quarter of the downloads is still pretty terrible.”
“Some people feel Log4j was a big wake-up to the industry, a collective freak-out and awakening,” he says. “And it has helped us really expand upon the message about software supply-chain security, because no longer were people in denial. The thing we were all talking about was real now’ we were all living it. But the peer pressure alone of Log4j should have forced everyone to upgrade, so if we can’t get this one to 100 percent, what about all the other ones?”
For security researchers, the question of how to address the long tail of a vulnerability is always present. And the issue applies not just to open-source software, but proprietary systems as well. Just think about how many years it took to move the last 10 percent of Windows users off of XP.
“With these worst-case scenarios—black swan events in open source—you just know they’re going to keep happening, because the community has gotten a lot better at reacting, but the pace of open-source development is even faster,” ChainGuard’s Lorenc says. “So we have to find the balance of prevention and mitigation, and keep coming up with efforts to reduce the frequency as much as possible. It’s like The Simpsons meme when Bart says, ‘This is the worst day of my life.’ And Homer says no, ‘The worst day of your life so far.’”
