GStone Posted September 4, 2012 Share Posted September 4, 2012 Googlebot is slurping up my bandwidth like crazy. I nearly went over my bandwidth limit last month, so I've been watching the online users periodically through the day, and there are always dozens of instances of the googlebot agent all over the board, all with the same IP (66.249.67.55), and they're there 24/7. I checked the stats for the site over at Google's webmaster tools, and the reported crawl rate was WAY off, saying that the high end of the average was only 17 pages per day. I just counted 35 google-guests. Obviously I don't want to disallow the thing, and I did put a delay in the robot.txt, but it seems weird to me that the bot just literally moved into my site permanently! One thing I did notice was that the robots.txt reads: Disallow: /index.php?act=Post&CODE=02 Disallow: /index.php?act=Print&client=printer Disallow: /index.php?app=calendar Disallow: /index.php?app=calendar$ ..... etc. My board, however is not in the root directory, but is at mysite.com/forums/index.php Should I prepend the /forums/ directory to all of those urls? Link to comment Share on other sites More sharing options...
Management Matt Posted September 4, 2012 Management Share Posted September 4, 2012 You just need to throttle Googlebot. Link to comment Share on other sites More sharing options...
GStone Posted September 4, 2012 Author Share Posted September 4, 2012 I have it limited to 2.5 seconds between requests. This is a screen shot of my current online users that I took just a few minutes ago. I have 7 pages like this (this is just the top 1/2 of the page), but only six actual users are on line right now and 12 legitimate guests. Link to comment Share on other sites More sharing options...
Dmacleo Posted September 4, 2012 Share Posted September 4, 2012 You just need to throttle Google. I gladly fixed this for you :) Link to comment Share on other sites More sharing options...
GStone Posted September 5, 2012 Author Share Posted September 5, 2012 Ain't it the truth. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.