From an infected website to a serious security incident - a case study
Serious data leaks and security incidents do not necessarily have to be the result of deliberate actions by intruders aimed at a specific target. They are not always noticed immediately by the victims of the attack. Often a serious breach of security occurs as a result of a combination of several events and its detection may be the result of the inquisitiveness of a random person. The following is a transcript of an interesting investigation, as a result of which we discovered a very serious threat to customer data of a large hosting company.
WARNING:
The description below includes links to websites that have been attacked or maintained by reputable organizations. Opening them may be dangerous.
Case study
On one of the websites maintained at AZ.pl (a hosting company belonging to Home.pl that supports the largest number of domains in Poland according to http://top100.wht.pl/) I noticed suspicious behavior: typing the website address in the browser caused redirection to the address http://semanticore.com.pl/admin/dropbox/proposal/which opened a page pretending to be Dropbox and asking you to log in - a classic phishing. The first thing that occurred to me was that I missed my domain renewal and someone took over. But no, domain is paid for. So I log in to the hosting panel and check the website files. Several of them have today's modification date, although I have not made any changes today. The website has therefore been modified in an unauthorized manner. Quick analysis of possible attack vectors:
- Outdated, leaky CMS - no longer available, the latest version of WordPress, updated a few days earlier
- An infected computer and a password leak stored in the FTP client - no longer possible, apart from the fact that I know something about security and I am rather sure of my computer, I did not use FTP access at all and I only had the password saved in KeePassX
- Hosting account password leak (cPanel) - dropped for reasons as above, password nowhere remembered except for KeePassX.
- Brute-force or dictionary attack on FTP or cPanel password - as I mentioned a bit about security, I know the password is non-dictionary and quite strong (several characters), so the probability is rather small.
- Someone else knew the passwords and was not careful - no one, the site was managed only by me
So there is probably only 0-day for WordPress to consider, but let's be serious, who would pay for it to make phishing on a zero-audience website? Having no better ideas at the moment, I look to the directory of another website that I have on the same hosting account and here's a surprise: in the page folder consisting only of the index.html file, wp-apps.php file appeared out of nowhere. More importantly, I don't even have an FTP account for this site, so basically the only way to upload a file there was via cPanel. Or maybe a compromise of the hosting server? Quite a bold thesis, but almost at the same time I notice that the phishing page to which I am redirected turns out to have the same IP as mine, so it is on the same server. And suddenly the bold thesis becomes more plausible. It's time to call AZ and share your thoughts. The conversation goes something like this:
- I present the problem point by point, as described above
- to which address are you redirected when you enter your website?
- http://semanticore.com.pl - I answer
- ok, we will block this user's account for breaking the regulations
- ??? But wait, it's not his fault that someone modified his page. What about me?
- We can restore your files from a backup from 4-5 days ago, please send us such a request.
"What if someone changes them again the same way?"
- Please run an anti-virus scan to be sure
- But I tell you that I am sure that the problem is not on my side. Maybe it's something for you, since this phishing takes place on the same server? Maybe it's some sort of massive problem?
- Impossible, we don't have similar reports
So I sent a request to restore the files from the backup, as requested, describing the whole problem once again there.
Meanwhile, I started browsing the modified files and assessing the damage. Found wp-apps.php turns out to be a variation of quite popular web shell called "WSO". In the first few lines, access to it was secured with a password written in the form of MD5 "9cdfb439c7876e703e307864c9167a15" (the first online breaker finds the password in a few seconds, it reads "lol"). The rest of the file is code obfuscated by a fairly simple and popular technique:
eval (gzinflate (base64_decode ("HZ3HkqPagkX / 5Y3uDQZ4Fx09wA ……… etc.")))
ie php code execution, which is first base64 encoded and then deflate packed. It is worth mentioning that a good practice in hardening web servers is to disable eval (). The PHP tutorial warns against it, and in the first comment from 12 years ago we read:
If eval() is the answer, you're almost certainly asking the
wrong question.
But let's go back to the web shell. I go to the address http: //moja_strona/wp-apps.php to check what interesting we have at our disposal. A few clicks and I'm in the / backup / sqlbackup / mysql / directory !!! and in it dumps of all databases from the server with "644" permissions! (reading for everyone). I am a bit distrustful, so I find my Wordpress database and download it to be sure. I unpack, look inside, there is everything and more. In the wp_users table, apart from my administrative account, I find another one called "-" with a set password and full rights to the database. And at this point I realize what really happened. It was enough for one of the thousands of pages hosted on the server to be compromised (leaked CMS, weak password, virus, whatever) and the intruder gained access to all databases from the entire server. He could download them and calmly break passwords, then log into CMSs as admin and add his accounts there, with the help of which he later modified the content of the pages and browsed the databases. Probably one of the databases also stores the passwords of all clients to their hosting accounts. If an intruder gained access to it and began to break passwords, he could turn thousands of customers' services upside down at any time.
The next day, I decide to check it out. I go back to the database dump directory and look for some name pointing to the cPanel user base. Most databases have names that start with the customer's login, which means that they are user databases. There are only a few with other names, including "mysql", the main database server base. Let's check. But wait, here the permissions don't allow reading (600). I look at other bases, powers also limited. Did someone find out and correct the backup scripts? I leave the case open for a few days because the rush of various responsibilities does not allow me to come back to the topic temporarily. One evening, however, I decide to check if something has changed. I visit the backup directory and again a surprise: again the permissions to all databases are 644. But by the way, I also notice that there are much less databases in the directory. Maybe I came across the moment when the database dump files were created? I refresh the directory, a few files have arrived. Bingo. I look at the process of creating files and their dates. The first ones are open at 11.30 pm, the last ones around 04.30 am. Dropping over 1,800 databases takes about five hours, after which the access permissions for all files are modified from 644 to 600. And now it's all clear. The backup script modifies the rights after taking the screenshot, but as the number of customers and databases increased, the process began to take several hours. And these few hours every night, each of the server users and intruders who have taken over any page on it, have the opportunity to copy all databases. Back to the point where I was looking for the cPanel user base. Mysql turns out to be a good lead. The "user" table contains logins and passwords of all clients to their own databases and the main hosting panel. If anyone but me discovered it, the results could be catastrophic.
In the meantime, I find a ".lastlogin" file in my hosting account. A quick overview of the log and, unfortunately, the assumptions are confirmed. I have logged into my hosting account several times with an IP address that does not belong to me. So someone has already downloaded the databases and uses their content. Whois shows that the suspicious IP address belongs to the Home.pl network - AZ.pl's majority shareholder. Maybe they have a similar problem there too? Maybe the infrastructure is shared or managed by the same team? It's time to report the problem to AZ.pl again. Due to the late night hours and the helpline closed, I use the contact form:
I have detected a critical security error on your servers that allows you to hijack, among others, all hosting accounts and databases and leakage of personal data on a massive scale. Please take this information very seriously (I have evidence) and pass it on to the person responsible for security in the company (CISO) or directly to the management board. I am ready to cooperate in identifying and removing the source of the problem. I expect contact from a person empowered to take appropriate steps.
The contact takes place after more than 32 hours. You call me from the hotline asking if I have any evidence, because they have hundreds of similar reports. Please be advised that I can send a screenshot of the main mysql database or any other user. Therefore, the Lord asks for my login to the hosting panel and informs that the administrators will follow the history of activities on it. I get an additional hint from me that the databases are read to everyone. After another 30 hours, I get a short message by e-mail:
Based on our conversation and the information you provided, as well as on the verification of the settings on the servers, changes were made to improve the security of AZ.pl servers.
Could use some "thank you", but never mind, it remains for me to check the effects. I put it back on the shell web server, which I removed earlier and I find that they did it. This time I can't go beyond my home directory anymore. There remains, however, quite an important issue to be solved. So I reply to the last e-mail:
Are you going to inform your customers about this incident? In my opinion, they deserve information about what could, have already happened or may still happen:1. Possibilities of taking over their databases
2. Possibility of unauthorized creation of users in databases
3. Possibility of unauthorized access to CMSs and modification of web services
4. Possibility of unauthorized access to the hosting account and modification of services
5. Possibility of unauthorized access to e-mail
6. Possibility of personal and payment data leakage
There is no answer yet. However, two very important questions remain: how long has this been the case and whether the problem occurred on only one server where I have an account, or on all of them (then not thousands, but tens or hundreds of thousands of customers could have lost their data) . Google helps quite quickly in determining the answer. I get the idea that since some pages on AZ have been compromised and the same web shell was placed on them, maybe Google has indexed them. One of the information returned by the web shell is the output of the Linux 'uname -a' command. On my server it is: "Linux az0103.srv.az.pl 4.0.6 brocade-gut #31 SMP Fri Nov 20 22:05:10 CET 2015 x86_64". So I ask Google for the string "srv.az.pl 4.0.6 brocade-gut”And the first results return links to compromised sites on AZ servers, not only with the WSO web shell, but several others. In most of them, however, the web shell has already been replaced with options.php, which loads a flash demo informing about the page being hijacked by someone AYYILDIZ TIM (example page: http://www.poczta.kukow.pl/options.php ). There are also downloaders and uploaders ( http://hosting6203918.az.pl/ , http://www.hosting3298174.az.pl/loc ). But the most interesting post on the RedTurk forum turns out to be: http://redturk.org/konu-12-site-hacked.html dated May 13, 216, reading as follows:
Addressers Zone http://golgeler.net/shadow!Tar.gz!1
Linux az0014.srv.az.pl 4.0.6 brocade-gut #31
Easy server 🙂
The first link contains a list of hijacked sites, most of which are hosted on AZ's servers. The comment and the second link (although no longer working) unfortunately confirm my concerns. More than four months ago, on a forum with a dubious reputation, someone posted a link who listed the contents of the directory with databases through a web shell on a server other than mine. So it can be assumed that all AZ servers were affected. We can only guess who is in possession of these databases and what to do with them.
It took me several evenings in installments to write down my notes and compose this article. In the meantime, I was looking closely at my services at AZ, being aware that someone could get to them at any time. Another disturbing thing I noticed: Loginizer (a WordPress plugin that detects failed login attempts) reported to me a huge number of access attempts from different IP addresses. In addition, the website logs contain an equally large number of entries proving attempted brute force attacks on the WordPress password by xmlrpc.php. It can therefore be assumed that intruders who discovered a vulnerability on any AZ server, attacked in bulk all sites located on other servers in order to gain access also to databases located there. I am a bit surprised that AZ.pl does not have tools to detect this type of activity. Such attempts should probably be noted, for example, by WAF (Web Application Firewall)?
I am also not entirely convinced that AZ.pl is aware of the seriousness of the situation. Or maybe they are counting on the fact that all their clients have strong enough passwords that dictionary and brute force attacks will be useless? In my opinion, clients should definitely be informed that as a result of carelessness on the part of AZ.pl, it is possible to access all their services at any time, including DNS configuration, e-mail browsing, website modification. There are, among others, online stores, forums, blogs ... The amount of personal data in customer databases is probably countless. Can there also be payment details, maybe even card numbers?
Finally, a small tip for people who use AZ.pl services. List of steps that each user should take to secure their services:
- Changing the password to your hosting account (cPanel).
- Change of passwords for all services associated with a given account (mail, ftp, databases).
- Database review for the presence of unknown users. In my case, it was the name "-", but other names may have appeared as well.
- Review of websites for the presence of web shells or unauthorized files.
- In the case of stores and other service websites - notifying their customers about the possibility of their personal data being stolen and recommending / enforcing a password change.