# /robots.txt file for http://www.travelnsw.com.au/ # # Allow everything except /about/stats which may contain links to # non-existant pages as the site organization changes. # Include the trailing slash, else we will exclude visits to # all directories beginning with "stats" in /about # # Also disallow /cgi-bin as this area is not searchable anyway # User-agent: * Disallow: /advertise/accommodation/results.html Disallow: /search/results.html Disallow: /about/stats/ Disallow: /data/ Disallow: /cgi-bin/