scriptygoddess

16 Jul, 2003

Get an email when Google visits

Posted by: Jennifer In: Script snippet

When googlebot comes (if it does!) to pages that contains this code, you get an e-mail sent to the address you specify.

Insert the PHP code below in all pages where you want to track whether google is visiting the page or not.

<?php
//Change the following to your email address
$email = "myemail@something.com";
if(eregi("googlebot",$_SERVER['HTTP_USER_AGENT']))
{
mail($email, "Googlebot detected",
"Google has crawled : ".$_SERVER['REQUEST_URI']);
}
?>

If you have include a file like header.php, footer.php,.. etc. everywhere then you can simply place this code in any of those files to track all those pages. MT users can place it in their template and all their pages will be tracked.

PERL users can use:

# give path of sendmail
$mailprog = '/usr/sbin/sendmail';
#Change the following to your email address
$email = myemail@something.com;
$browser = $ENV{'HTTP_USER_AGENT'};
if ($browser =~ m/googlebot/i) {
open (MAIL, "|$mailprog -t") || die "ERROR: Problem Opening – $mailprog!\n";
print MAIL "From: $email\n";
print MAIL "To: $email\n";
print MAIL "Subject: Googlebot detected\n\n";
print MAIL "Google has crawled : ".$ENV{'REQUEST_URI'};
close (MAIL);
}

33 Responses to "Get an email when Google visits"

1 | Fil

July 17th, 2003 at 7:57 am

Avatar

I get a parse error everytime I call this piece of code on my site.
It seems that the line "$email " is failing.

Any ideas?

2 | Jayant Kumar Gandhi

July 17th, 2003 at 8:00 am

Avatar

Thanks to the editors for inserting that bug 😀

Change
$email = myemail@something.com;

to

$email = "myemail@something.com";

3 | Fil

July 17th, 2003 at 8:04 am

Avatar

Had I been paying attention this morning, I would've noticed that!
Thanks for the quick response.

4 | Jayant Kumar Gandhi

July 17th, 2003 at 8:17 am

Avatar

Go get some caffiene 😀

5 | Jennifer

July 17th, 2003 at 8:28 am

Avatar

"Thanks to the editors for inserting that bug :D"

Actually I posted exactly what you submitted (I'll forward it to you).

Post is now updated with your correction.

6 | Jayant Kumar Gandhi

July 17th, 2003 at 9:18 am

Avatar

ah ya. probably if was just sleep taking over me when i posted it late @ night.

7 | lotas

July 17th, 2003 at 9:54 am

Avatar

hmmmm. only one thing. what happens when google comes in and visits about 800+ pages, or in my case 35000+ pages! you go to check your mail and BANG! mailserver explodes! 😛

8 | Jayant Kumar Gandhi

July 17th, 2003 at 11:05 am

Avatar

Just do it for you main page or important ones or new ones.
wot say :)

9 | endoid [at] endoid [dot] net

July 21st, 2003 at 4:26 pm

Avatar

Well done scripties, this is really useful.
There is one thing that the MovableType users will need to remember though: it's so trivial that we will easily forget it.

To add the PHP code to the MT templates you need to make sure that MT generates it's files (or at least the main index) as PHP files and not simply as HTML. That way the PHP code is executed instead of being returned as text.

In the MT administration area, click on "Templates > Main Index " and change the "Output File" to "index.php" if it's "index,html". If you call it "main.html" or indeed anything, just make sure that the file's extension is ".php" and test it out just to make sure.

For those of you using Linux and you've access to the "wget" program you can have it act as the Googlebot engine by running the following:

wget -U googlebot http://www.put-your-site-url-here.com

That will let you know that the test is working.

ps: my *might* need to remove the old "index.html" file (I recommend you rename it first) in case your web server looks for ".html" files before it looks for ".php". And remember, you *do not) need to change your archived files: they can remain as ".html" if you have many of them.

10 | thomas

July 25th, 2003 at 1:31 pm

Avatar

does it work on .html file? What if the webhost doesn't support .php or PERL ? can I still get notified when Googlebot come?

11 | endoid [at] endoid [dot] net

July 25th, 2003 at 1:58 pm

Avatar

To Thomas:
Normally no, this code (in either PHP or Perl) will not work in a .html file. Webservers know to execute code that lives inside .php files *before* they are sent our to your web browser and this is how thye web server is able to send you the email to tell you that Google has visited. This is known as "server-side" code becasuse it's code that runs on the server. If your web server is not set up to run code and only serve up .html code then this won't work for you.
The other code that works on the web is JavaScript or VBScript. With this case though it runs in your web brower, but that's code running on your browser machine and not on the server. Your browser machine doesn't have any way of knowing what other borwsers have come to the website. Other browsers in this case includes the GoogleBot so there's no easy way that client side JavaScript or VBScript will work.

[I say no easy way, but it would be possible if your webserver was set up to be frightfully insecure and there's no way I'm going to get into how to do that. You'll be so much better off finding a server with php or a hostings sote that allows php instead.]
/e

12 | Jayant Kumar Gandhi

July 26th, 2003 at 12:24 am

Avatar

It maybe be possible using SSI if your host allows it. You can exec the sendmail program with the parameters of the email.
I am not sure how will one compare that the useragent is google or not.
perhaps you can make a shell script or a executable program and pass all the related parameters and that shell script/program will invoke sendmail.

Presently I have no plans of implementing it myself. If a lotta people request it, I might think of it.

13 | Fil

July 26th, 2003 at 5:37 pm

Avatar

Would there be anyway to have the email also send the user's agent and IP address?

Im sort of using this as a notification of when google visits so I know how often they're crawling my site. I for one, don't want my site crawled and would like to have the IP's of the google crawlers so that I can ban them.

Any other suggestions to aid me in this task?

14 | Jayant Kumar Gandhi

July 27th, 2003 at 12:18 am

Avatar

In PHP $_SERVER['HTTP_USER_AGENT'] will have the user-agent while $_SERVER['REMOTE_ADDR'] has the IP address.

The better option is:-
See http://www.google.com/bot.html

15 | Fil

July 27th, 2003 at 4:15 pm

Avatar

Its too bad that the google bot doesnt seem to obey those tips.

I use both a robots.txt and the META tag for robots to no avail.
GoogleBot just doesnt seem to care!

16 | mike

October 30th, 2003 at 10:28 pm

Avatar

Well 1st off i been reading over this php stuff and looking at the codeing and i see stuff i get and all but well i am lost i have a web server called savant and i use it to host my web site and i been trying to do some stuff like Fil has posted and i cant get it to work if you could tell me how to get it to work i would o you one i have a cgi-bin well i can add what i like to it what i would like to do is when someone comes on my site i would like to send them a email thanking to for comeing to my site and they no that this email is comeing to them so it is not spam if you could tell me how i could go about doing this would be grate i been doing html have not got in to all this other stuff but I AM NOW thanks you can email me at itsmcfly32@ma.rr.com

17 | Jenni

January 5th, 2004 at 11:14 pm

Avatar

Where would you put the following pieces of info in the script?

$_SERVER['HTTP_USER_AGENT']
$_SERVER['REMOTE_ADDR']

Thanks!

18 | Jennifer

January 6th, 2004 at 1:06 pm

Avatar

You mean to have those pieces of information mailed to you too?

19 | Jenni

January 6th, 2004 at 4:59 pm

Avatar

Yes, but I figured out a better way. I just denied from googlebot.com in my .htaccess file. But thanks to the script, I knew they were crawling my site, so I could block them.

Thank you anyway!!

20 | Anh-Kiet Ngo

March 18th, 2004 at 2:47 pm

Avatar

I would like to comment for Perl users.

$email = myemail@something.com;

After that is changed into:
$email = "myemail@something.com";

Be sure to add a "\" in front of the @ sign. Otherwise, it will not work. The end result would be:

$email = "myemail\@something.com";

Computer Central

21 | Ks Weblog

July 16th, 2003 at 11:12 pm

Avatar

Get Email if Googlebot Comes
From scriptygoddess When googlebot comes (if it does!) to pages that contains this code, you get an e-mail sent to the address you specify….

22 | Wandering Around

July 16th, 2003 at 11:16 pm

Avatar

Get Email if Googlebot Comes
From scriptygoddess When googlebot comes (if it does!) to pages that contains this code, you get an e-mail sent to the address you specify.

23 | Bloggins

July 17th, 2003 at 8:41 am

Avatar

Google Schiester
Reading the scriptygoddess I saw a post authored by Jayant Kumar Gandhi. In this post, Jayant states how using simple php, you can make your server email you whenever the famed "GoogleBot" visits your page. This was particularly interesting t…

24 | Savage Nomads

July 18th, 2003 at 10:30 am

Avatar

Monitoring your site
This is a pretty cool script that will email you when the googlebot indexes your pages. I think I may try to add this script this weekend, I'll let you know how it goes. Nope I don't have a life….

25 | [ entrivia ]

July 21st, 2003 at 4:36 pm

Avatar

Get notified when Google Crawls
The scriptygoddesses have a very simple and very effective bit of PHP code to let you know that Google has crawled your site. It depends on your webserver running PHP and that you've access to sendmail (but you *may* be…

26 | Daisyhead

July 26th, 2003 at 11:22 pm

Avatar

Scriptygoddess goodness
I haven't perused Scriptygoddess lately so I'm missing out on all kinds of little gems like this handy little piece of code which notifies you by email when a Googlebot visits your site and this code which allows you to…

27 | pMachine HowTo

August 7th, 2003 at 7:24 am

Avatar

HowTo: Tell if the GoogleBot Has Visited
Originally found at Scripty Goddess, here's a code snippet you can add to your pages to tell you when the GoogleBot has found your site.

<?php
//Change the following to your email address
$email = "myemail@something.com";

if(eregi("googleb…

28 | Untitledlog

August 10th, 2003 at 7:29 pm

Avatar

Sabe cuando Google revisa tu pagina
Visitando scriptygoddess me encontre este script que te brinda la posiblidad de…

29 | Glimpse of a Grrl

August 11th, 2003 at 12:05 am

Avatar

Playing with Google
Adi's trying to see if Google crawls his site often. So we both installed this code snippet from :SG:. Since Google keeps hitting my site, I'm going to see if I can redirect them to him by the force of…

30 | Nibble Guru

November 3rd, 2003 at 2:13 pm

Avatar

Get an email when Google comes visiting
One of my friend wanted me to do this for him. I thought I will share this with you too. See the following PHP code below to achieve it. <?php if(eregi("googlebot",$_SERVER['HTTP_USER_AGENT'])) { mail("myemail@something.com", "Googlebot detected", "…

31 | Rizwan Kassim's Public Log

November 5th, 2003 at 7:05 pm

Avatar

Look Mom!
See what I've done? The side bars (just left for now) are collapsable, and not in the stupid way that the blogrolling is collapsable… It actually doesn't transmit the data if it doesn't need to, meaning an even quicker download…

32 | Daisyhead

January 6th, 2004 at 12:47 am

Avatar

RSS feed updates
I've updated my RSS feeds. You can now choose from posts only or posts with comments. Although I'm not sure the posts with comments feed is working. I used Jennifer's template and I just copied and pasted it, but still…

33 | Glimpse of a Grrl

May 23rd, 2004 at 10:38 pm

Avatar

Playing with Google
Adi's trying to see if Google crawls his site often. So we both installed this code snippet from :SG:. Since Google keeps hitting my site, I'm going to see if I can redirect them to him by the force of…

Featured Sponsors

Genesis Framework for WordPress

Advertise Here


  • Scott: Just moved changed the site URL as WP's installed in a subfolder. Cookie clearance worked for me. Thanks!
  • Stephen Lareau: Hi great blog thanks. Just thought I would add that it helps to put target = like this:1-800-555-1212 and
  • Cord Blomquist: Jennifer, you may want to check out tp2wp.com, a new service my company just launched that converts TypePad and Movable Type export files into WordPre

About


Advertisements