Use of robots.txt

[Simon Harvey]

In a few posts on this community I have ready about renaming robotstxt.txt to use as a robots.txt template file however I am not able to find robotstxt.txt on my installation.

Is this not used anymore? Is it advised to create a robots.txt file?

Thanks

[opentype]

There is no need for a robots.txt. You can create one, if there is public content you don’t want indexed. 

[Simon Harvey]

Great thanks.

[MeMaBlue]

Hello @opentype. 

what would you do with links like these appearing on top of results page,    (as the only links of your site, on that search)

http://parents.org.gr/forum/topic/15147-institouto-gonimotitas-loytradis/?do=email&comment=1048539

http://parents.org.gr/forum/notifications/?do=followers&follow_app=forums&follow_area=topic&follow_id=75243

Currently  if i search for my domain name + something else in greek, 

parents.org.gr ρωτήστε την ειδικό   

or 

δοκιμή parents.org.gr 

those links appear  exactly on top of search results, instead of a normal, (with content) link that should appear, or just the domain name. 

what is your recommendation ?    I have seen a 5% increase in bounce traffic since the migration. 

I am discussing it right now with support, they recommend adding a specific  robot.txt to hide those.    

/*?do=email&*

Doing a search now, to see what others are doing, ...is it just me who has this problem?.....??

 

[opentype]

You seem to allow a lot of activity for guests. That’s why these links are even crawled. In a typical setup, they don’t appear for guests. Allowing emailing for guests is pretty problematic. Every spammer could use your page as a spam distribution center. 

But anyway. As long as you want to keep your current settings and permissions, then yes, you would want to set up robots.txt rules to filter out those URLs. 

[MeMaBlue]

Thank you very much for your reply. 

no i didnt get that those links were the result of that.     

following and emailing.... ofcourse i dont want that for guests.      these things are very new for us  because of the huuuuge  difference  between   vbulletin 3,8  and  ips....! 

I will go and change that asap! . 

But i think those things should be something like a default setting when you migrate? ... no ?