Request: Edit Robots.txt, and Add XML Sitemap to Robots.txt 2015-03-23T15:55:05+00:00

Build with IDX Broker Forums Enhancement Requests Request: Edit Robots.txt, and Add XML Sitemap to Robots.txt

  • Author
    Posts
  • imFORZAimFORZA
    Participant
    Post count: 68

    IDX Broker has a robots.txt file which loads for all of its clients. It would be great if developers/clients had the option to add or edit this file. This can be useful in telling robots to ignore specific IDX pages for example. It also should always have the link to the idx xml sitemap.

    2 ratings, 2 votes2 ratings, 2 votes (+2 rating, 2 votes, rated)
    You need to be a registered member to rate this post.
    Loading...
  • imFORZAimFORZA
    Participant
    Post count: 68

    So reviewing Webmaster tools from many of our clients accounts, and I noticed something. Google typically doesn’t want us blocking css, javascript, or images, however for the IDX pages you are blocking some via the robots.txt file.

    If I check the robots.txt file for any client I will see this two lines:

    Disallow: /graphical/
    Disallow: /images/

    Google Webmaster Tools is reporting the issue of these urls being blocked on IDX Pages:

    /graphical/frontend/css/idxstyles.php
    /images/ajaxLoadBar.gif

    The first one will show up with various strings/variables on the url. Here is a screenshot example: https://www.evernote.com/l/AAGk_sQLMe1IwqtMEwV5hTsT34lQlSyXUv0

    Can we please make sure these resources are not blocked via the robots.txt file? Google needs to see these resources to properly render the IDX pages with any custom css.

    http://googlewebmastercentral.blogspot.com/2014/10/updating-our-technical-webmaster.html

    2 ratings, 2 votes2 ratings, 2 votes (+2 rating, 2 votes, rated)
    You need to be a registered member to rate this post.
    Loading...
  • Nick Leech
    Keymaster
    Post count: 109

    We do have a feature request ticket in for us to edit the robots.txt file to remove: Disallow: /graphical/ , Disallow: /images/

    2 ratings, 2 votes2 ratings, 2 votes (+2 rating, 2 votes, rated)
    You need to be a registered member to rate this post.
    Loading...

You must be logged in to reply to this topic.