Now, if you are using Google Ads, then you will probably want to add in two more User-agents;
Code: [ Select all ]
User-agent: Googlebot
Disallow:
User-agent: Googlebot-Image
Disallow:
User-agent: AdsBot-Google
Disallow:
User-agent: Mediapartners-Google
Disallow:
Now, me wanting to tidy up my site, I decided to combine them into a single grouping than having multiple. I also wanted to define where my site's sitemap was because it is suggested by both Google and Mozilla that it really should be there for crawlers to find it. But know, that if you do copy how I have mine set up, you will need to change your sitemap URL. If you don't have a sitemap, then go to this module: Your-Site-URL/modules.php?name=Site_Map. After visiting that module, then you should see a sitemap.xml file in your root of your site that has been just created/updated. I have that link in my menu on the left of my site for easy access.
Before I show you my code, I do have a warning.
!!WARNING!! - Don't go crazy with things in the robots file, for you can easily destroy what SEO you may have.
Now, here is my code that I use on my site.
Code: [ Select all ]
User-agent: Baidu
User-agent: *alexa*
Disallow: /
User-agent: Googlebot
User-agent: Googlebot-Image
Disallow:
User-agent: *
Disallow: /admin/
Disallow: /blocks/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /themes/
Disallow: /admin.php
Disallow: /config.php
Disallow: /ips.php
Disallow: /trap.php
Sitemap: https://www.headshotdomain.net/sitemap.xml
All I did was combine the User-agent: into single groupings instead of having separate. Went from:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-Image
Disallow:
to
User-agent: Googlebot
User-agent: Googlebot-Image
Disallow:
But it really doesn't matter if you do that type of clean up. Also, I have the sitemap defined at the bottom. This can get a bit tricky, but since I have my site set up using the force WWW and HTTPS, which I go over in , that is how I needed to put it. What I have read is you need to be absolute with the path, but I am not sure if you could get away with doing something like:
Sitemap:
Sitemap:
Which could cover both the HTTP and the HTTPS. But on my part, that would only be speculation.
HeadShot Extreme
-> PC & Codeing Talk
output generated using printer-friendly topic mod. All times are GMT - 7 Hours
Page 1 of 1