If you have anything to do with technology, the chances are pretty good that you wasted at least part of the month of December patching, upgrading, or reconfiguring software that uses Log4J.
Now that the dust has settled, we can look at the vulnerabilities that were announced in December, what factors led to the problem, and why it should never have occurred.
Table of Contents
What is Log4J?
Log4J is a logging utility for Java, written and maintained by the Apache Software Foundation.
Uh… do what, now?
Yes, the entire purpose of Log4J is to take information from a Java-based application and write it to a log file.
What is a log file?
A log file can take many different forms, but for most Java applications, a log file is just a text file. As an application does something, it writes information about something to the end of that text file. So the log file is just a record of everything the application did, from the time it started, to the time it stopped (or crashed). What ends up in a log file is largely up to the application’s developers.
Every well-written application, especially one that runs as a server or service, should write detailed information to a log file, in order to tell you what happened and when. Here are some things you might see in a well-thought out log file:
- At Date, Time: INFO [User] Bob logged in from IP address 1.2.3.4 and was assigned Application Role 1
- At Date, Time: INFO [User] Bob submitted a transaction consisting of these details: {$14,2,3,A1,B1}, ID: 12345
- At Date, Time: INFO [TransactionCruncher] 12345 Processed Normally, 200ms
- At Date, Time: WARN [FraudAlert] 12345 contains an invalid transaction code – transaction cancelled.
A good log file tells an administrator what happened, the criticality of the event, when it happened, and why. It also tells the administrator what module within the application processed what transaction, and the outcome of that transaction.
Log files are typically used for:
- troubleshooting
- debugging
- performance monitoring
And, although log files are not typically used for non-repudiation, they can be. For example, Amazon isn’t going to scrape a bunch of log files to figure out your order history – that comes from a database. However, when you buy something, the transaction itself might get logged.
OK, got it: Log4J is used by Java applications to write diagnostic information generated by the application out to a text file.
Correct.
Why is Log4J So Popular?
If all it does is write to text files, isn’t that something the developers can do directly within their applications?
Yes, and no.
Log4J provides a lot more than just “writing to text files” – it is a “logging facility” that provides a standard set of capabilities to any application that uses it:
- Filter for specific events or type of events
- Log to various destinations, such as console, text files, or even to a database
- Split out certain events to specific log destinations – for example, log all alerts (and only alerts) to alerts.log
In addition, most of this can be configured by an administrator at runtime without having to recompile, nor build in localized logging options. Moreover, any administrator who supports multiple Log4J-enabled applications can use standardized options so that every application’s logging configuration and management are similar.
To a developer, this represents hundreds or thousands of lines of code that they don’t have to include in to their application, and then test and debug.
For example, even if you want to write to a simple text file without Log4J, your application needs to determine where to write the log, how big the log file can grow, and what to do when the log file reaches its maximum size.
Just figuring all of that out using parameters read from a config file might be a few hundred lines of code, but if you use Log4J, all of that is handled transparently, independent from the application, and can even be customized by an administrator, all without having to add very much code in to your application.
For example, if the administrator decides that they want everything logged to a database, Log4J handles everything, including the database connection, all without having to write any code to support database connections.
So Log4J often appears to be a win-win – developers don’t have to waste time and resources on logging, and administrators get to benefit from a robust and standardized set of features and functionality.
Recap of the Log4J Problem
On December 9, 2021, a “zero-day” vulnerability (software is deployed in the wild, exploit exists, no mitigation currently exists) was published by the Alibaba Cloud Security Team, allowing arbitrary code execution using specially-formulated URLs that contain embedded Log4J directives.
Security research is big business these days, because you can make a name for yourself, and build an entire career by discovering and publishing a single, new security vulnerability, so there is a drive to publish as quickly as possible, because there is a lot of competition.
However, it’s considered standard practice to go to the software vendor before publishing, so that the software vendor has time to develop a patch before the vulnerability goes public.
It’s hard to tell the exact sequence of events, but Log4J 2.15.0 was also released on 12/9. Unfortunately, Log4J 2.15.0 didn’t effectively remediate the vulnerability.
Also, the first thing that happens when a new vulnerability gets announced is that the bad guys immediately start scanning the web for unpatched servers.
The second thing that happens is, like sharks who taste blood in the water, other security researchers jump in, and immediately start tearing apart the software component in question, looking for other vulnerabilities.
Both of these happened with Log4J.
Within hours, hackers began scouring the internet for vulnerable servers, which forced administrators to have to move quickly to patch them.
Also, other security teams quickly uncovered a litany of related vulnerabilities, necessitating a quick succession of Log4J patches, and each patch had to be applied as quickly as possible. After 2.15, there was 2.16, 2.17, and 2.17.1.
This made for a MISERABLE month of December for just about everyone who works in IT.
Remember – Log4J just writes log files… so how does writing log files make your system vulnerable?
Hint: It’s not the log files.
Log4J Is Insidious
As a project of the Apache Software Foundation, other Apache software projects utilize and therefore require Log4J, including Apache, one of the world’s most popular web servers, and Tomcat, one of the world’s most popular application servers.
If you do online shopping or banking, it’s fairly likely that you’ve used Apache or Tomcat or both, and haven’t even realized it.
Further, many consumer-grade and even commercial-grade technology “appliances” often include Apache web server, or Tomcat, or both, including camera systems, routers, firewalls, and many others. Therefore, all of this stuff has Log4J potentially embedded in its firmware, unless the vendor specifically disabled it.
So, even if your bank’s online banking platform isn’t written in Java, it might run on Apache web server or Tomcat application server (or both), and even if not, your bank’s infrastructure might include firewalls and routers that have Apache web server embedded inside them.
But the problem doesn’t stop there…
Even your cable modem, residential gateway, or wireless router might have Apache web server (and Log4J) embedded within it.
In fact, the Apache web server is so incredibly popular, that it’s often baked in to IoT (Internet of Things) devices as a convenient way to provide a user interface to a device that wouldn’t normally have a screen or buttons.
Just within your house, you might have a dozen copies of Apache running, and not even realize it. Apache could be running on any of these devices:
- Smart TV
- Smart phone
- Tablet
- Camera system
- Smart thermostat
- Smart devices such as light switches
- Health tracker / smart watch
- Home Security system
- Smart Door Lock
- Bluetooth Gun Safe
- Vehicle information / navigation system
- Washer and dryer
- Robot vacuum cleaner
- Smart assistant (Alexa / Google)
And the list goes on.
So Apache web server is easy and free, and because it’s easy and free, it’s everywhere, and because Apache is everywhere, Log4J is potentially everywhere as well.
Worse, none of this gets automatically updated. In most cases, you-the-user have to initiate a software update. And in many cases, older IoT devices might not even be currently supported by the manufacturer, or the manufacturer might have even gone out of business, which means that a Log4J update for some devices might not even be available.
And IoT vendors are not known for “hardening” their products by removing unused services and components, such as Log4J that might not be needed on your smart toothbrush, but it’s THERE because no one thought to remove it.
Log4J is Certainly Ubiquitous, But Why Do You Say Insidious?
Log4J is insidious because of feature-creep.
Stated best is Zawinski’s Law:
Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.
Or Justin’s Corrolary:
Eventually every online platform turns in to Tik-Tok
If we look at the oldest stable release, 1.0.4, the entire download is around 825K (under 1 meg), and there are no good date stamps within the file, but the files themselves are dated in the range of 2000 to 2001.
Comparing to 2.14 (the last pre-vulnerability release), it was released in March, 2021, and the file size is 14 meg.
Interestingly, Log4J versions 1.x are not vulnerable, which gives us another clue.
So basically, Log4J started in 2001 as a standard way to allow Java applications to write to text files, but 21 years and +13 meg of code later, it now has a bunch of unnecessary features that can be exploited by hackers.
As the Log4J developers kept cramming in new features, they were slowly increasing the attack surface until an exploit was inevitable.
And THAT is why Log4J is insidious.
A long series of bad decisions led to the point where my smart bowling ball can get hacked remotely, even if I’m not using it, because the user interface runs Log4J.
Lessons That Need to be Learned
New features and functionality are necessary, but feature-bloat absolutely led to this problem.
In the past, I’ve written about “Top Developer Mistakes”, so I’m adding this to the series.
Here are the lessons that the Log4J developers need to learn:
Unnecessary Features Should Be Disabled By Default
This is the top 1 thing that could have prevented this mess.
The original vulnerability had to do with JNDI, which is a way for Java programs to communicate with other Java programs, and has absolutely NOTHING to do with writing to a text file.
Likewise, one of the vulnerabilities had to do with JDBC, which is a way for Java programs to talk to a database, and has NOTHING to do with writing to a text file.
So if all of this extra +13 megs of crap had simply been disabled by default at runtime, MOST SYSTEMS would not have been vulnerable. Only the 1% to 2% of systems who used the features in question would even be vulnerable.
Most people use Log4J to write to a text file. That’s it. That’s the only thing that should be enabled by default.
Additional functionality should be DISABLED unless specifically enabled within the configuration.
This is a tough one because developers are proud of their work, and they want their products to be easy to use, but even if most people don’t even know your code exists, it’s better to be silent and anonymous than vulnerable.
Don’t Install Unnecessary Components By Default
Better yet, include your core functionality in the core module. Most people write to text files. But, rather than include all of the optional features within the core module, PUT THEM IN A SEPARATE MODULE THAT DOESN’T EVEN GET INSTALLED BY DEFAULT.
Code that isn’t there can’t be exploited.
Put all of that crap in to an “optional-features” folder, and if the administrator needs one of these “optional features”, he or she can go to the “optional-features” folder and install it!
But if not, there is nothing to exploit because the feature is optional and never got installed.
Stay In Your Lane: Steve Moose Lawnmowers (A Hypothetical Fable)
By way of analogy, let’s say your company, Steve Moose, makes lawnmowers. Steve Moose has been making lawnmowers since the turn of the LAST century. Steve Moose lawnmowers represent decent quality at decent prices, and has acquired such a loyal following that owning a Steve Moose lawnmower is now synonymous with Being American.
But… some of your loyal customers keep coming to you and saying, “Steve! I like drinkin’ a beer while I mow my lawn. And my damn beer gets warm, because I have a big damn lawn.”
The answer should be: “Thank you for your loyalty to Steve Moose. We specialize in grass-decimation hardware, not thermal-management hardware. We do, however partner with a company that sells Steve Moose-branded beer coozies, and I’m sure a Steve Moose coozie will keep your damn beer cold.”
Instead, Steve Moose toils for months developing a tiny generator that sits on top of the gasoline engine inside the lawnmower. The generator charges a battery, and the battery runs a cooling system, and the cooling system has a cup holder, and now you can keep your damn beer cold.
This new feature revolutionizes lawnmowing, and soon everyone agrees that the ONLY way to be a true, loyal American is to buy a Steve Moose lawnmower with a damn beer cooler on it. Your company sells hundreds of thousands of units.
But… after a few months, these lawnmowers start catching on fire.
People are injured, there is a massive recall, your company is now being sued for property damage caused by runaway, flaming lawnmowers that found their way in to garages and yoga gardens all over this great nation.
WHAT HAPPENED?
The battery that runs the beer cooler is prone to catch on fire.
The same battery that has nothing to do with mowing lawns!
STAY IN YOUR LANE!
(Talking to YOU, WinZip!)
Pingback: Monitor and TV Buying Guide – 2022 | Justin A. Parr - Technologist