I had occasion recently to try and figure out how to use the Firewall built into Mac OS X to prevent a very bad mannered “bot” from hitting one of my sites – at times at the rate of 10 hits per second, and 2 seconds later another 8-10 hits!!! So I needed to be able to block certain IP numbers, or ranges of IP numbers. Mac OS X comes with the FreeBSD firewall programme called IPFW. This is a very powerful feature that can be accessed from Terminal.

Some Googling later I came up with several helpful sites that got me up and running with this.

First, using the Apache server logs identify the IP number, or range of IP numbers you want to block. I used TextWrangler to open the log file and do some preliminary editing, and then imported that into FileMaker Pro to get only the log lines applicable to PiplBot (BAD ROBOT!!!!). Over the course of about 5 hours it used 84 different IP numbers as it hit away at one of my sites over 19,000 times.

So once I had a list of these numbers, I was able to break them down into a number of shorter lists that had the first 2 or 3 octets of the IP number the same. With this done, this site http://www.mikero.com/misc/ipcalc/ provides a VERY handy calculator that will take the starting and ending IP numbers in a range, and convert it to a range in the CIDR notation (very technical explanation here) which takes a range of numbers like

67.228.42.161, 67.228.42.162, 67.228.42.169, 67.228.42.174

which potentially covers 14 different numbers and converts it to 67.228.42.160/28 which represents 16 numbers without the need to list them all out. And simlarly the range from 208.43.23.227 to 208.43.33.238 covers 2,572 addresses, and is represented by 208.43.0.0/18 – a range of 16,384 addresses.

So, armed with this knowledge and ability, I’m now able to understand the instructions on this page http://www.dancatts.com/articles/dealing-with-bad-bots-at-the-firewall-level.php and this page http://www.ibiblio.org/macsupport/ipfw/ which in their simplest form are saying that you can use Terminal with this instruction

sudo /sbin/ipfw add 02010 deny ip from 67.228.42.160/28 to any in

to add a block into the Firewall for the range of numbers 67.228.42.160/28. You can see the current status of your ipfw with this Terminal command

sudo /sbin/ipfw list

which will return a list in this form

02010 deny ip from 74.86.25.192/28 to any in
02020 deny ip from 67.228.42.160/27 to any in
02030 deny ip from 74.86.0.0/16 to any in
02040 deny ip from 75.126.0.0/16 to any in
02050 deny ip from 174.36.22.0/24 to any in
65535 allow ip from any to any

Changes made by Terminal only last as long as your Macintosh is running – they are not saved to be used on a Restart unless you write a startup script to do this. This site http://www.ibiblio.org/macsupport/ipfw/ provide details on this, including a number of sample scripts, but frankly this was way over my head, so I turned to MacUpdate to see if there was an application that would do this via a GUI (Graphical User Interface). I found several, and settled on WaterRoof by Hany El Imam. This allows you to define the rules you want to implement, and then takes care of creating the script that will activate these rules each time your Macintosh is started up.

WaterRoof Screen Shot

WaterRoof Screen Shot

This seems much easier to deal with 🙂

So for now PiplBot is banned, even though they seem to be honouring their statement that they would remove all of my sites from their list of sites to crawl.

I hope this helps someone else – I’ve written it partly to help me remember what I did, but also to help others.

 

Saturday 20 November was the “Smart Move Regional LEGO League Qualifying Tournament” of “FIRST Lego® League” at Grandville Middle School, Grandville, Michigan.

The Team from Byron Center is Team <default>. The ended up finishing 6th at this tournament, including 1st place in Research, so are off to the State Finals in Flint on 12 December. Congratulations Guys and Girls!!!

Here are a few photos taken by Uncle Roger.

© 2000 – 2019 Roger’s Ramblings Powered by WordPress Suffusion theme by Sayontan Sinha

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close