Seeing an IP blacklisted is part and parcel of life as a web host, a daily occurrence for the biggest and most permissive ones. Nevertheless, this is a first for LRob in its 10 years of existence!
The more positive among you will say that this is the ransom of glory... Inevitably, as the volume increases, so does the risk of unauthorized activity on a site. The most critical will cause a scandal. Whatever the case, at LRob we're committed to transparency. So we're here to answer all your questions.
What happened, and what solutions are we implementing? Here are the answers.
Discover blacklisting on AbuseIPDB
On Sunday October 13, I was alerted by my client Weather Centre that its site was blocked by the MalwareBytes paid antivirus (thanks to them for this alert).
Faced with this anomaly, LRob was in a hurry. Whether it was Sunday or Christmas Day, I immediately checked his site, which had no problems whatsoever, then investigated with MalwareBytes. The cause of the blockage was given in just 1 hour (well done MalwareBytes support): the IPv4 of the server hosting this site is blacklisted on AbuseIPDB for malicious requests. But obviously, the problem originates from another site.
Favorite AbuseIPDB
Not all blacklists are created equal (CF: SPFBL), nonetheless, AbuseIPDB is a real project, well done and serious.
In the midst of all this bad news, the discovery of this community blacklist was a stroke of genius: not only had it enabled me to detect an anomaly on my server, but it was also easy to contribute. That's why I LRob immediately joined AbuseIPDB to help report attacking IPs and help the global sysadmin community in return.
But the most important thing was to stop the source of the malicious requests. So where were they coming from?
An unexpected cause
Of course, I immediately looked for the cause of these requests emanating from one of my servers. If it's a customer, it's not tolerated. If it's a hacked site, it has to be shut down and repaired immediately... If it's a server intrusion, it's very serious. So I started investigating, and what I found was both harmless and unusual.
It will not have escaped your notice that LRob carries out the repairing and securing hacked WordPress sites for several years now. This leads many customers to choosing LRob web hosting to benefit from discounted repairs and rock-solid support.
But recently, a newly repaired site played a nasty trick on me: a recurrence. A repaired site was hacked again, something I've never had happen to any of my hostings before.
This fell within the 90-day warranty, so I repaired the site again at no charge. But the attacks, though less frequent, continued to arrive on AbuseIPDB.
After 3 days of fruitless log searches during the day and evening, I finally analyzed all the network traffic on the server (a titanic task), until I traced one of the abnormal processes (programs) launched as the system user of this site. Eureka! I've found the root cause of the problem.
How is this possible?
I now know that, as soon as it arrived, this site actually contained malicious programs (not just PHP scripts, but actually programs), which managed to run as soon as the site was imported. This meant that even when deleted, and even with the site deactivated, the malware continued to run. Hackers were therefore able to use this as an attack vector even after the site had been repaired. They were also able to rewrite malicious files, generating visible recurrence.
As these programs were not running directly via the usual web server, but independently of it, it was impossible to detect their activity via the web server logs, making detection more difficult (which is why network traffic had to be analyzed as a last resort, which is unusual). What's more, this type of program isn't even supposed to be able to run, and for good reason: the "PHP exec()" function that launches them is supposed to be disabled... But it wasn't for this client, due to a specific configuration (since corrected). I'd already seen this problem on other servers, and if it hadn't been mine, I'd have looked directly at the processes launched and found the problem in 30 minutes... But here, as it wasn't supposed to happen, it took me 3 days. The joys of the job!
The immediate solution
All suspicious processes (running programs) were of course saved for investigation, then stopped. A complete site checkup was redone, and a manual check 3x a day was set up to ensure that no recurrences took place.
Current situation
At the time of writing, no more attacks have been reported on AbuseIPDB, but the blacklist has not yet been lifted.
Delisting
The delisting request was made on the day the problem was discovered, and can take between 7 and 10 days to be lifted. This is slow, and would be AbuseIPDB's only flaw. Fortunately, the impact of this blacklisting is fairly low until the request is lifted.
No other sites affected
Thanks to the excellent website isolation applied to my hosting, the hacked site was confined to itself and had no impact on other hosted sites.
Additional long-term solutions
The impact of this blacklisting is very minimal, as evidenced by the fact that only Weather Centre alerted me to this concern, as this is probably the most popular site I host, which means they can quickly be notified of this type of information. Thanks again to them and to MalwareBytes for their responsiveness in reporting relevant information.
Nevertheless, some customers may one day wish to avoid having their site hosted on a blacklisted IP because of a third-party website. The solution is a dedicated IP address.
That's why I'll soon be launching a dedicated IP option available to all LRob web hosting.
This requires some preparation, as adding multiple IPs to a dedicated server is not trivial. I also need to estimate and define the price of the option. When the option is launched, each LRob customer will be able to benefit from a dedicated IPv4 and IPv6 if they wish.
With a little imagination, this could also enable server migrations without IP changes, thanks to "IP Failover" (more expensive but more flexible, they can be transported from one server to another, as long as the server provider remains unchanged). This should please my resellers who don't have control over all their customers' DNS zones...!
The importance of transparency
There's no such thing as a trouble-free service. Of course, I could have hidden this minor problem and only one customer would have noticed. But at LRob, we choose to show our commitment to handling any anomaly with rigor and devotion.
Transparency is the basis of trust. And I thank you for your trust.
Leave a Reply