Request: Edit Robots.txt, and Add XML Sitemap to Robots.txt

[imFORZA]

#321

IDX Broker has a robots.txt file which loads for all of its clients. It would be great if developers/clients had the option to add or edit this file. This can be useful in telling robots to ignore specific IDX pages for example. It also should always have the link to the idx xml sitemap.

(+2 rating, 2 votes, rated)
You need to be a registered member to rate this post.

Loading...

[imFORZA]

#443

So reviewing Webmaster tools from many of our clients accounts, and I noticed something. Google typically doesn’t want us blocking css, javascript, or images, however for the IDX pages you are blocking some via the robots.txt file.

If I check the robots.txt file for any client I will see this two lines:

Disallow: /graphical/
Disallow: /images/

Google Webmaster Tools is reporting the issue of these urls being blocked on IDX Pages:

/graphical/frontend/css/idxstyles.php
/images/ajaxLoadBar.gif

The first one will show up with various strings/variables on the url. Here is a screenshot example: https://www.evernote.com/l/AAGk_sQLMe1IwqtMEwV5hTsT34lQlSyXUv0

Can we please make sure these resources are not blocked via the robots.txt file? Google needs to see these resources to properly render the IDX pages with any custom css.

http://googlewebmastercentral.blogspot.com/2014/10/updating-our-technical-webmaster.html

(+2 rating, 2 votes, rated)
You need to be a registered member to rate this post.

Loading...

[Nick Leech]

#531

We do have a feature request ticket in for us to edit the robots.txt file to remove: Disallow: /graphical/ , Disallow: /images/

(+2 rating, 2 votes, rated)
You need to be a registered member to rate this post.

Loading...

[Author]

Posts