I had occasion recently to try and figure out how to use the Firewall built into Mac OS X to prevent a very bad mannered “bot” from hitting one of my sites – at times at the rate of 10 hits per second, and 2 seconds later another 8-10 hits!!! So I needed to be able to block certain IP numbers, or ranges of IP numbers. Mac OS X comes with the FreeBSD firewall programme called IPFW. This is a very powerful feature that can be accessed from Terminal.
Some Googling later I came up with several helpful sites that got me up and running with this.
First, using the Apache server logs identify the IP number, or range of IP numbers you want to block. I used TextWrangler to open the log file and do some preliminary editing, and then imported that into FileMaker Pro to get only the log lines applicable to PiplBot (BAD ROBOT!!!!). Over the course of about 5 hours it used 84 different IP numbers as it hit away at one of my sites over 19,000 times.
So once I had a list of these numbers, I was able to break them down into a number of shorter lists that had the first 2 or 3 octets of the IP number the same. With this done, this site http://www.mikero.com/misc/ipcalc/ provides a VERY handy calculator that will take the starting and ending IP numbers in a range, and convert it to a range in the CIDR notation (very technical explanation here) which takes a range of numbers like
184.108.40.206, 220.127.116.11, 18.104.22.168, 22.214.171.124
which potentially covers 14 different numbers and converts it to 126.96.36.199/28 which represents 16 numbers without the need to list them all out. And simlarly the range from 188.8.131.52 to 184.108.40.206 covers 2,572 addresses, and is represented by 220.127.116.11/18 – a range of 16,384 addresses.
So, armed with this knowledge and ability, I’m now able to understand the instructions on this page http://www.dancatts.com/articles/dealing-with-bad-bots-at-the-firewall-level.php and this page http://www.ibiblio.org/macsupport/ipfw/ which in their simplest form are saying that you can use Terminal with this instruction
sudo /sbin/ipfw add 02010 deny ip from 18.104.22.168/28 to any in
to add a block into the Firewall for the range of numbers 22.214.171.124/28. You can see the current status of your ipfw with this Terminal command
sudo /sbin/ipfw list
which will return a list in this form
02010 deny ip from 126.96.36.199/28 to any in
02020 deny ip from 188.8.131.52/27 to any in
02030 deny ip from 184.108.40.206/16 to any in
02040 deny ip from 220.127.116.11/16 to any in
02050 deny ip from 18.104.22.168/24 to any in
65535 allow ip from any to any
Changes made by Terminal only last as long as your Macintosh is running – they are not saved to be used on a Restart unless you write a startup script to do this. This site http://www.ibiblio.org/macsupport/ipfw/ provide details on this, including a number of sample scripts, but frankly this was way over my head, so I turned to MacUpdate to see if there was an application that would do this via a GUI (Graphical User Interface). I found several, and settled on WaterRoof by Hany El Imam. This allows you to define the rules you want to implement, and then takes care of creating the script that will activate these rules each time your Macintosh is started up.
WaterRoof Screen Shot
This seems much easier to deal with 🙂
So for now PiplBot is banned, even though they seem to be honouring their statement that they would remove all of my sites from their list of sites to crawl.
I hope this helps someone else – I’ve written it partly to help me remember what I did, but also to help others.