Adding robots.txt to a Django project the easy way

Googling a website I own, I found out Google indexed parts of my site erroneously. After inspecting the errors in Google Webmaster Tools, I found the culprit in my code. However, after I cleaned up the error, Google still had the wrong URL’s in it’s index.

To solve this problem, I needed to declare the wrongly indexed URL’s in a robots.txt file to tell Google to not index those pages. In this post I’ll show a quick way of adding a robots.txt file to your Django application.

1. Create a robots.txt file in your projects templates folder

Search engines look for robots.txt file and read and follow the rules defined in it (if they are playing nice). This wikipedia page and robotstxt.org explain the different rules you can create pretty nicely.

So, first create a plaintext robots.txt file, write your rules and save it in your project’s template folder.

For example I used this rule:

User-agent: *
Disallow: /en-us/

2. Add a robots.txt URL pattern in urls.py

In order for Google to find your robots.txt file at the standard yourwebsite.com/robots.txt location, you need to create a new URL pattern in your urls.py file.

The most easy way to do this is to import and use the direct_to_template helper as follows:

# urls.py
from django.views.generic.simple import direct_to_template

urlpatterns += patterns('',
    url(r'^robots\.txt$', direct_to_template,
    {'template': 'robots.txt', 'mimetype': 'text/plain'}),
)

That’s it!

If implemented correctly, you will see your robots.txt file when you visit yourwebsitecom/robots.txt. You can now tell Google to reindex your site through Webmaster Tools and check if the wrong URL’s have disappeared.

Alternative methods

The above method is well-suited for when you have a relatively simple website. When things get more complex, you might want to check out a more sophisticated solution such as django-robots.

If you have any other methods, or libraries you use, please do let me know in the comments!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>