How to generate (and automatically update) a Rails sitemap

Building a sitemap index file

There’s a limit to what a search engine can index and that limit is 50,000 URLs or sitemaps that are larger than 50MB uncompressed. If that’s your site, you’ll need to break your sitemap up into multiple maps and reference them in a sitemap index file that you’d submit Google Search Console and the like.

It’s also not unheard of to separate sitemaps by topic, category or content type to help with debugging, analytics, sitemap stats and monitoring indexing.

So a sitemap index is like a table of contents for your sitemaps, organizing multiple sitemaps to avoid server slowdowns and easing management.

The sitemap_generator gem gives us the flexibility to create separate configurations for different website parts, like a distinct sitemap file for blog content and another for the main website pages.

Let’s get to the building:

1. Configure sitemap generator

Open your config/sitemap.rb file and configure the sitemap generator to generate multiple sitemap files.

# config/sitemap.rb
SitemapGenerator::Sitemap.default_host = "https://www.example.com"
SitemapGenerator::Sitemap.sitemaps_path = 'sitemaps/'

SitemapGenerator::Sitemap.create do
  # Add your URLs or use dynamic generation here
end

2. Generate sitemap files

Run the following command to generate your sitemap files:

rake sitemap:refresh

This will generate multiple sitemap files and store them in the specified path.

3. Create sitemap index

Create a new XML file for the sitemap index. You can manually create it in the public directory or use a controller action to generate it dynamically.

# app/controllers/sitemap_controller.rb
class SitemapController < ApplicationController
  layout nil

  def index
    headers['Content-Type'] = 'application/xml'
    render 'index.xml'
  end
end

# app/views/sitemap/index.xml.builder
xml.instruct!
xml.sitemapindex xmlns: 'https://www.sitemaps.org/schemas/sitemap/0.9' do
  (1..n).each do |i|  # Replace n with the number of sitemap files generated
    xml.sitemap do
      xml.loc root_url + "sitemaps/sitemap#{i}.xml.gz"
      xml.lastmod Time.now.utc.strftime('%Y-%m-%dT%H:%M:%S%:z')
    end
  end
end

Replace (1..n) with the range that corresponds to the number of sitemap files generated.

4. Configure routes

Add a route for the sitemap index in your config/routes.rb file.

# config/routes.rb
get '/sitemap.xml', to: 'sitemap#index', defaults: { format: 'xml' }

Now, when you visit /sitemap.xml, it should serve the sitemap index file, which references the individual sitemap files.

Structuring the index file

A sitemap index file is structured as an XML file. It must contain the <sitemapindex> root tag, which encloses individual <sitemap> elements. Each <sitemap> element contains a <loc> tag that specifies the location of the sitemap. Think of it as a family tree, with thetag as the root,tags as the branches, andtags as the leaves.

To add sitemaps to the index file, we use the add_to_index method. This method can even include sitemaps hosted on a different domain specified by the host option. However, it’s important to remember that sitemap index files should only reference sitemaps hosted on the same site, and these referenced sitemaps need to be in the same directory or lower than the index file in the website hierarchy.

Automating sitemap updates with rake tasks

Wouldn’t it be nice if there were a way to keep your sitemap up to date without doing it manually every time?

That’s where rake tasks come into play.

Custom rake tasks for sitemap updates can be created within the lib/tasks directory of your Rails application, automating the update process. It’s like setting a recurring reminder that ensures your sitemap is always fresh and updated.

Building these tasks is like writing instructions for an automated robot.

Creating a custom rake task for updating a sitemap

A rake task to generate a sitemap can be defined with the following steps:

1. Create a rake task

In your Rails application, create a new Rake task file. For example, you can create a file named update_sitemap.rake in the lib/tasks directory.

touch lib/tasks/update_sitemap.rake

Edit the update_sitemap.rake file and define your task:

# lib/tasks/update_sitemap.rake
namespace :sitemap do
  desc "Update the sitemap"
  task :update => :environment do
    # Add the code to update the sitemap here
    Rake::Task["sitemap:refresh"].invoke
  end
end

This task depends on the sitemap:refresh task, which is provided by the sitemap_generator gem.

2. Configure sitemap generator:

Make sure you have the sitemap_generator gem in your Gemfile:

# Gemfile
gem 'sitemap_generator'

Run bundle install to install the gem.

Ensure that your config/sitemap.rb file is properly configured to generate the sitemap files.

3. Run the rake task

You can now run your Rake task to update the sitemap. In the terminal, execute:

rake sitemap:update

This will invoke your custom Rake task, which, in turn, invokes the sitemap:refresh task to update the sitemap.

Integrating with cron for scheduled updates

Cron jobs are like alarm clocks for our tasks, automating the execution of rake tasks and plain ruby scripts, including rb file handling, allowing sitemaps to be refreshed regularly without manual intervention.

On Unix-like systems, the crontab utility schedules rake tasks and RVM wrappers are recommended for invoking rake when RVM is part of the project.

For instance, a cron job can be configured to execute rake -s sitemap:refresh silently every day at 5 am. This ensures your sitemap is always up to date, even while you’re catching up on your beauty sleep!

Enhancing sitemaps with additional resources

At this point, you’ve got a fully functioning sitemap that’s regularly updated. But why stop there? We can further improve indexing by incorporating additional resources such as images and videos, as well as alternate links for a URL for international versions of your site.

Sitemap extensions can convey additional data about various media types, enhancing their visibility to search engines.

The SitemapGenerator gem is equipped to handle various sitemap extensions, including those for News, Image, Video sitemaps and more. By including these in yours, you’re not only providing comprehensive information about our website but also enriching the user experience by showcasing multimedia content effectively.

The most common multimedia content on any site? Images and videos, so let’s discuss how to integrate those into your map.

Supporting videos and images

To include image and video URLs in your sitemap in Rails, you need to modify your config/sitemap.rb file to specify these additional entries. Here’s an example of how you can add image and video entries to your sitemap using the sitemap_generator gem:

# config/sitemap.rb
SitemapGenerator::Sitemap.default_host = "https://www.example.com"
SitemapGenerator::Sitemap.sitemaps_path = 'sitemaps/'

SitemapGenerator::Sitemap.create do
  # Add your regular URLs
  add '/page1'
  add '/page2'
  
  # Add image URLs
  add '/image1.jpg', :images => [
    { :loc => '/image1.jpg', :title => 'Image 1 Title', :caption => 'Image 1 Caption' },
    # Add more image entries if needed
  ]

  # Add video URLs
  add '/video1.mp4', :videos => [
    { :thumbnail_loc => '/video1_thumbnail.jpg', :title => 'Video 1 Title', :description => 'Video 1 Description', :content_loc => '/video1.mp4' },
    # Add more video entries if needed
  ]
end

In this example:

• For image entries, you use the add method and provide an :images option as an array of hashes. Each hash represents an image with its location :loc, title :title, and caption :caption.
• For video entries, you use the add method and provide a :videos option as an array of hashes. Each hash represents a video with its thumbnail location :thumbnail_loc, title :title, description :description, and content location :content_loc.

Adjust the URLs and metadata according to your actual image and video details.

Advanced sitemap customizations

We can customize things further with priority values, change frequencies and managing last modification dates to improve SEO. While Google doesn’t take into account priority values or change frequency tags for indexing pages or determining search results, they can be significant for other search engines that use this information to understand website content hierarchy.

Sitemap priority

Sitemap priority values range from 1.0 (highest priority) to 0.0 (lowest priority) to indicate the relative importance of pages within a site. Think of it as a grading system for your web pages, guiding a search engine to understand which pages you deem more important.

Change frequency

Setting realistic changefreq sitemap tags can guide other search portals in determining the update cycle of website content.

It’s basically telling a search engine how frequently you update your content, giving them a schedule to follow for indexing your website. Again, Google doesn’t use the change frequency attribute but giving a little more information never hurts.

To further optimize your site, consider using the Google Search Console and Google Webmaster Tools for additional insights and recommendations.

Managing last modification dates

Finally, last modification dates, this one Google does consider.

The lastmod tag can be used to indicate the last modification time of the URL in the W3C datetime format. You can automatically set the lastmod attribute by passing an ActiveRecord object that responds to updated_at to the sitemap generator.

For dynamic content, database triggers or Rails model callbacks can be used to update a 'last modified' timestamp, ensuring the lastmod value remains accurate.

Once you’ve done all of this, you can use a sitemap validator w3c to check your file for any errors before submitting it to a search engine.

For a deeper dive into what the sitemap generator for Rails can do, see the comprehensive guide on GitHub.

Summary

The power of a well-structured, regularly updated sitemap cannot be overstated. It’s a vital tool for improving your website’s SEO and visibility. So, go ahead, use the knowledge you’ve gained from this guide and create a Rails sitemap that perfectly represents your Rails application. Remember, a well-crafted sitemap is more than just a map; it’s a blueprint for success.

Frequently Asked Questions