Ecommerce merchants are usually very quiet about their servers getting hacked. Obviously, they face some serious legal liabilities in situations like this, and the potential for lost sales if their customers lose faith in them. This may be good for business on the one hand, but it’s really bad for business on the other. Sharing security information with people who need it helps us all. We just faced a hack that didn’t impact our business in any way, so I want to share it to help other people. I hope it’s useful to other merchants.
On November 4, 2015, a Magento test server got hacked with a novel attack that I haven’t seen before. This was not a production machine and caused no problems for us, other than being an inconvenience as we quarantined the machine and rebuilt a new test server. The hack occurred in the morning just before I drove into work. It was immediately obvious and we took the affected machine offline within an hour.
The Affected Server
This server contained a full copy of our Magento source code, which we had set-up to test some new shipping code. It had been online for several months. Initially, it was firewalled from the rest of the world and only accessible on our office subnet. But when we had some bugs, we opened up our firewall to allow an outside vendor access to the system. We created an admin account for the vendor and pointed them to the Magento admin URL.
The best practice for Magento admin URLs is to change them from the default site.com/admin to something else. This doesn’t offer great security and is really security through obscurity. But it does make it harder for brute-force password guessing on a public-facing server. I’m not a fan of security through obscurity, so on our live site we use access control mechanisms to block /admin from any IPs that aren’t authorized to hit it. Since it’s only accessible from authorized IP addresses, we haven’t changed from the default /admin. What this meant, though, is that /admin was accessible on our test server, now exposed to the public Internet.
Initially, we assumed we’d only have this test site online for a few weeks max. But a few weeks dragged on into a few months due to several bugs we were trying to resolve that took longer than expected.
A week before the attack, as I was doing some Google searches on brand terms, I discovered that Google had found this server and was indexing pages on it. This was a big problem from an SEO perspective since the test server had all the content from our live site and was now a complete copy of our website with duplicate content. I set up a robots.txt file to tell Google to stop indexing the site, jotted down a note to add it to Google Webmaster Tools and de-list all the URLs at that domain, and figured I’d get to that in a few more days.
robots.txt:
User-agent: *
Disallow: /Discovering the Attack
I woke up on the morning of November 4th and took a shower. When I got out of the shower, my phone had blown up with a dozen emails from the cron daemon on the test server. It was complaining that the Magento cron job wasn’t executable. “Weird,” I thought. Maybe someone on my team had done something on the test site to break it and I would need to ask once I got into the office. I threw on some clothes and drove in.
When I got to the office, the first thing I did was mention seeing some errors, to see if the admins had anything to say about that. None of them had a comment. So I logged into the server to check the logs. Cron was failing as it tried to run the Magento cron job, so I dropped to a shell to check the Magento files, and I was greeted with this:
The README file looked like this:
Basically all the files under /var/www that were owned by the apache2 process (www-data user on a Debian jessie server) were encrypted to a file named the same as the base filename with a .encrypted extension, and the originals were removed.
We checked a couple of things, then immediately shut the virtual machine down and copied images of the disks to my laptop so I could bring up an isolated VM to figure out how it happened.
How the Attack Occurred
The last log entries in /var/logs/apache2/access.log before were:
It appears as though a PHP file had been previously uploaded to the staging site at /skin/error.php, and commands were sent to it with a POST.
Here’s a directory listing of the /var/www/html/skin folder:
What’s immediately interesting is that there is no file here named 404.php. But the apache log showed that 404.php was accessed, with a 200 status code, indicating that it was previously there. Perhaps it executes and then removes itself? Also, it’s interesting to note that there is an additional file here named host.encrypted.
A quick check for other .encrypted files anywhere on the filesystem revealed nothing:
So we’ve got three suspicious files: /skin/host, /skin/error.php, and /skin/404.php. Unfortunately, this was supposed to be a quick staging server for a module integration that would go away quickly, so most settings were the Debian defaults, and I only have log files going back to late October. But here are all references to those files earlier in the apache2 access.log files from October 27 through November 4:
Basically, no references to those files before the day of the attack. No references to /skin/host at all. And no references to /skin/404.php before the final attack. Perhaps 404.php is the “do it now!” command, after error.php is used to upload multiple commands, or do whatever else it’s going to do before the final website encryption.
Here’s the information on the IP addresses, with reverse lookups on the hosts that perpetrated these attacks.
I don’t have any evidence for how the extra files were uploaded to /skin, and I suspect it may have happened weeks ago. At first I thought this was an attack related to the Magento SUPEE-6788 patch, which I’d just installed on Monday, November 2. That was fresh in my mind. And perhaps it was. But if this test server hadn’t been updated in a number of months, since it was installed, there were previous patches it was also vulnerable to. It could have been any of them.
So what security hole was responsible for this attack? I have installed every Magento patch released in 2015 within 1 week of release. I originally installed this site from a backup taken of our live site in late July 2015. Since then, only the following patches have been released:
- SUPEE-6482 (August 4, 2015)
- SUPEE-6788 (October 27, 2015)
I checked the dated backup that I used to install this test server in late July, and verified that there were no extra files in /skin:
So they didn’t come from the backup. They were inserted between July and November. And the backup had been patched for all previous Magento security releases. This site had already been patched for Shoplift and everything earlier in the year. This site did not have Magmi installed, or any other known extension with security holes.
The implication of this is that this attack was either from security holes patched by SUPEE-6482 or SUPEE-6788… Or it’s from a new hole that we are not yet aware of.
Googling Staging Servers
As I previously mentioned, I’m no fan of security through obscurity. But how did hackers find this site? It was only up for a few months. There were no inbound links to this site from anywhere, as far as I know. The only people with the URL were internal employees and one external vendor.
I think that the answer is Google. For whatever reason, Google found the site. And Google indexed away. Checking the logs, it looks like Bing found it as well. Since it had a mirror of the live site’s robots.txt (until I changed it the week before the attack), there were no instructions to the spiders to tell them not to index the site. I previously mentioned the issues with duplicate content impacting SEO. But after this attack, I realized that being in Google also makes a staging or test system easier to find for the bad guys. There are a number of URL patterns and content snippets that are unique to Magento, and it’s easy to find sites with simple Google queries.
I’m going to be a lot better about hiding non-production servers from the search engines in the future.
Key Takeaways
So what did I learn? What should we do differently going forward? What should you do right now?
- Check your /skin folder on all production and staging sites and if you find any files other than the “install”, “frontend”, or “adminhtml” folders, take action right now.
- If your test or staging servers are public, check them immediately at MageReport.com and install any necessary patches.
- Going forward, ensure staging servers get prompt updates just like live servers do when a critical security patch is released.
- Firewall your staging servers by IP so only the vendors you’re working with can access them.
- Change your /admin URL, even if you don’t think you need to.
- If your staging server must be public, hide it better through robots.txt, which is also an SEO requirement.
- Outside development agencies and vendors should proactively tell merchants their IP ranges, encourage them to lock down staging servers, and warn them when access is open to the public.
- If you’re not running routine offsite backups, start immediately.
- Drop everything and make Magento security patches your top priority every time they are released.
- If you’re still waiting on external vendors to install SUPEE-6788, hit the warpath and beat them up over it. Do you really want to do business with suppliers who expose you to security risks?
- If an external agency hosts your staging server, make sure it’s locked down. Test it yourself to make sure it can’t be accessed from public locations.
Nov 10 2015 Update: Decryption Option Confirmed
Apparently this Crypto-Ransomware is now named Linux.Encoder.1.
One of my systems adminstrators sent me this link today:
Decryption Instructions and Script
I just installed the scripts on my backup of the hacked server and worked through their procedure. Initial results are in. IT WORKS!
Because of this, I know a bit more about the files /skin/error.php and /skin/host.
/skin/host is a binary file. It’s an ELF executable, statically linked and stripped.
/skin/error.php is a set of PHP functions to decrypt a string, then a long string assigned to a variable that’s encrypted and base64 encoded. The last line of the file wraps an eval() around a function call to decrypt the string, turning it back into PHP source code and executing it. Dumping the decrypted string is now trivial, so I’ve got that source code.
I haven’t read through the entire source code, but I’ve skimmed it. It’s definitely not specific to Magento. The original vector of attack was likely a bug in Magento. But the uploaded files are a generic Linux remote control system. It’s poorly-written old-school PHP that allows a remote attacker to run local commands, search the filesystem, and probe various services. It looks like it’s set up to allow remote downloads of your database if it’s not password protected, or if they can guess your password. There doesn’t appear to be any mechanism to search Magento-specific files for that information, however. But since Magento passwords are in a known location in most installs, it would be trivial for the attacker to manually check those files.
Nov 12 2015 Update: Initial Attack Likely ShopLift
In my original post, I said that there were only two patches missing from my site. Well, digging a bit deeper, I realized that’s not the case. When I set up this server, it was from a fully-patched backup of a Magento 1.9.0.1 site. But after installing it, I updated it to Magento 1.9.1.0. And I neglected to re-install the patches that 1.9.1.0 were missing. ShopLift was patched in 1.9.1.1, so this test server was missing that patch.
Based on a couple of other conversations I had with people telling me that all observed attacks with this software were on sites that were not patched for ShopLift, I’d say that was the most likely attack vector, and this probably wasn’t due to a more recent security hole.