Slickplan

How to generate (and automatically update) a Rails sitemap

To enhance your Rails application’s SEO, you must have an effective sitemap in place. This article demystifies the process of generating a Rails sitemap using the 'sitemap_generator ' gem, configuring your routes for accessibility and keeping the sitemap updated with rake tasks. We’ll walk you through each step, so search engines have no trouble finding and indexing your content.

Key takeaways

  • Use the 'sitemap_generator' gem to easily create and automate sitemap generation for Rails apps, which includes setting up, installing the gem and making the sitemap accessible through routing.
  • Employ a sitemap index to manage multiple sitemaps efficiently for larger websites, making sure to structure the XML properly, and consider adding alternate links for multi-language sites.
  • Automate sitemap updates by creating custom rake tasks and schedule them using cron jobs to keep the sitemap current, enhancing SEO with additional resources like images and videos.

Setting up your sitemap with a gem

What is a gem?

A gem is a nice little bundle of prepackaged Ruby code that developers can use for specific tasks — like creating a sitemap — without having to write out all the code. The easiest way to get something down in Ruby.

Sweetness.

In this case, you’ll need the 'sitemap_generator' gem, which allows you to easily generate sitemaps for your Rails app.

For larger sites, having a dynamic sitemap that generates automatically is a huge timesaver, so we’ll also break down how to do that with a rake task.

First, we explore the steps needed to generate your sitemap using the gem.

Installing the sitemap_generator gem

Getting the 'sitemap_generator' gem into the gemfile of your Rails app couldn’t be easier.

The gemfile, for the uninitiated, is like a shopping list for your Rails project, containing all the gems it needs to function properly. Once you’ve added the Rails sitemap generator gem to this list, it’s time to install it.

Just run gem install sitemap_generator in the command line and voila! You’ve got the gem installed.

gem install sitemap_generator

Now that you’ve got the sitemap generator gem installed and ready, it’s time to make the sitemap accessible.

Configuring routes for sitemap accessibility

Newsflash: a sitemap is pointless if it’s not accessible to search engines.

To make yours accessible, you need to add a specific route in the routes.rb file of your Rails app. This file is like the traffic controller of your site, directing requests to different parts.

The route should point to the sitemap’s location, which is typically located in the public/ directory of the Rails application.

It’s critical that the route pointing to the sitemap.xml.gz file is public and can be accessed without the need for authentication. It’s not of much use otherwise. Routing correctly guarantees that search engine crawlers can reach your sitemap without any hurdles.

Crafting the sitemap config file

With the routing in place, it’s onto the sitemap configuration file, which is called sitemap.rb. Note that SitemapGenerator will look for it in the /config directory.

To have this file generated and placed automatically, run this task:

rake sitemap:install

Now in its correct home, config/sitemap.rb is where we set your sitemap’s default host to your website’s hostname.

Within the SitemapGenerator:Sitemap.create block is where the magic happens, where we tell the SitemapGenerator how exactly to stitch together our sitemap.

Use add to include specific URLs and omit any files or directories you want excluded from the sitemap.

You have a host of other customization options at your disposal in the config file, including:

  • Setting change frequencies
  • Priorities
  • ast modification timestamps for each URL
  • Using a custom namer for sitemap files
  • Compressing sitemap files
  • Specifying a sitemap index
  • Setting a custom location for sitemap files

With these sitemap configs options, you can craft a map that perfectly represents your sitemap structure and hierarchy.

🎬 Learn what Slickplan can do!

We filmed a short video to show you exactly how to use Slickplan

Building a sitemap index file

There’s a limit to what a search engine can index and that limit is 50,000 URLs or sitemaps that are larger than 50MB uncompressed. If that’s your site, you’ll need to break your sitemap up into multiple maps and reference them in a sitemap index file that you’d submit Google Search Console and the like.

It’s also not unheard of to separate sitemaps by topic, category or content type to help with debugging, analytics, sitemap stats and monitoring indexing.

So a sitemap index is like a table of contents for your sitemaps, organizing multiple sitemaps to avoid server slowdowns and easing management.

The sitemap_generator gem gives us the flexibility to create separate configurations for different website parts, like a distinct sitemap file for blog content and another for the main website pages.

Let’s get to the building:

1. Configure sitemap generator

Open your config/sitemap.rb file and configure the sitemap generator to generate multiple sitemap files.

# config/sitemap.rb
SitemapGenerator::Sitemap.default_host = "https://www.example.com"
SitemapGenerator::Sitemap.sitemaps_path = 'sitemaps/'

SitemapGenerator::Sitemap.create do
  # Add your URLs or use dynamic generation here
end

2. Generate sitemap files

Run the following command to generate your sitemap files:

rake sitemap:refresh

This will generate multiple sitemap files and store them in the specified path.

3. Create sitemap index

Create a new XML file for the sitemap index. You can manually create it in the public directory or use a controller action to generate it dynamically.

# app/controllers/sitemap_controller.rb
class SitemapController < ApplicationController
  layout nil

  def index
    headers['Content-Type'] = 'application/xml'
    render 'index.xml'
  end
end
# app/views/sitemap/index.xml.builder
xml.instruct!
xml.sitemapindex xmlns: 'https://www.sitemaps.org/schemas/sitemap/0.9' do
  (1..n).each do |i|  # Replace n with the number of sitemap files generated
    xml.sitemap do
      xml.loc root_url + "sitemaps/sitemap#{i}.xml.gz"
      xml.lastmod Time.now.utc.strftime('%Y-%m-%dT%H:%M:%S%:z')
    end
  end
end

Replace (1..n) with the range that corresponds to the number of sitemap files generated.

4. Configure routes

Add a route for the sitemap index in your config/routes.rb file.

# config/routes.rb
get '/sitemap.xml', to: 'sitemap#index', defaults: { format: 'xml' }

Now, when you visit /sitemap.xml, it should serve the sitemap index file, which references the individual sitemap files.

Structuring the index file

A sitemap index file is structured as an XML file. It must contain the <sitemapindex> root tag, which encloses individual <sitemap> elements. Each <sitemap> element contains a <loc> tag that specifies the location of the sitemap. Think of it as a family tree, with thetag as the root,tags as the branches, andtags as the leaves.

To add sitemaps to the index file, we use the add_to_index method. This method can even include sitemaps hosted on a different domain specified by the host option. However, it’s important to remember that sitemap index files should only reference sitemaps hosted on the same site, and these referenced sitemaps need to be in the same directory or lower than the index file in the website hierarchy.

Automating sitemap updates with rake tasks

Wouldn’t it be nice if there were a way to keep your sitemap up to date without doing it manually every time?

That’s where rake tasks come into play.

Custom rake tasks for sitemap updates can be created within the lib/tasks directory of your Rails application, automating the update process. It’s like setting a recurring reminder that ensures your sitemap is always fresh and updated.

Building these tasks is like writing instructions for an automated robot.

Creating a custom rake task for updating a sitemap

A rake task to generate a sitemap can be defined with the following steps:

1. Create a rake task

In your Rails application, create a new Rake task file. For example, you can create a file named update_sitemap.rake in the lib/tasks directory.


touch lib/tasks/update_sitemap.rake

Edit the update_sitemap.rake file and define your task:

# lib/tasks/update_sitemap.rake
namespace :sitemap do
  desc "Update the sitemap"
  task :update => :environment do
    # Add the code to update the sitemap here
    Rake::Task["sitemap:refresh"].invoke
  end
end

This task depends on the sitemap:refresh task, which is provided by the sitemap_generator gem.

2. Configure sitemap generator:

Make sure you have the sitemap_generator gem in your Gemfile:

# Gemfile
gem 'sitemap_generator'

Run bundle install to install the gem.

Ensure that your config/sitemap.rb file is properly configured to generate the sitemap files.

3. Run the rake task

You can now run your Rake task to update the sitemap. In the terminal, execute:

rake sitemap:update

This will invoke your custom Rake task, which, in turn, invokes the sitemap:refresh task to update the sitemap.

Integrating with cron for scheduled updates

Cron jobs are like alarm clocks for our tasks, automating the execution of rake tasks and plain ruby scripts, including rb file handling, allowing sitemaps to be refreshed regularly without manual intervention.

On Unix-like systems, the crontab utility schedules rake tasks and RVM wrappers are recommended for invoking rake when RVM is part of the project.

For instance, a cron job can be configured to execute rake -s sitemap:refresh silently every day at 5 am. This ensures your sitemap is always up to date, even while you’re catching up on your beauty sleep!

Enhancing sitemaps with additional resources

At this point, you’ve got a fully functioning sitemap that’s regularly updated. But why stop there? We can further improve indexing by incorporating additional resources such as images and videos, as well as alternate links for a URL for international versions of your site.

Sitemap extensions can convey additional data about various media types, enhancing their visibility to search engines.

The SitemapGenerator gem is equipped to handle various sitemap extensions, including those for News, Image, Video sitemaps and more. By including these in yours, you’re not only providing comprehensive information about our website but also enriching the user experience by showcasing multimedia content effectively.

The most common multimedia content on any site? Images and videos, so let’s discuss how to integrate those into your map.

Supporting videos and images

To include image and video URLs in your sitemap in Rails, you need to modify your config/sitemap.rb file to specify these additional entries. Here’s an example of how you can add image and video entries to your sitemap using the sitemap_generator gem:

# config/sitemap.rb
SitemapGenerator::Sitemap.default_host = "https://www.example.com"
SitemapGenerator::Sitemap.sitemaps_path = 'sitemaps/'

SitemapGenerator::Sitemap.create do
  # Add your regular URLs
  add '/page1'
  add '/page2'
  
  # Add image URLs
  add '/image1.jpg', :images => [
    { :loc => '/image1.jpg', :title => 'Image 1 Title', :caption => 'Image 1 Caption' },
    # Add more image entries if needed
  ]

  # Add video URLs
  add '/video1.mp4', :videos => [
    { :thumbnail_loc => '/video1_thumbnail.jpg', :title => 'Video 1 Title', :description => 'Video 1 Description', :content_loc => '/video1.mp4' },
    # Add more video entries if needed
  ]
end

In this example:

  • For image entries, you use the add method and provide an :images option as an array of hashes. Each hash represents an image with its location :loc, title :title, and caption :caption.
  • For video entries, you use the add method and provide a :videos option as an array of hashes. Each hash represents a video with its thumbnail location :thumbnail_loc, title :title, description :description, and content location :content_loc.

Adjust the URLs and metadata according to your actual image and video details.

Advanced sitemap customizations

We can customize things further with priority values, change frequencies and managing last modification dates to improve SEO. While Google doesn’t take into account priority values or change frequency tags for indexing pages or determining search results, they can be significant for other search engines that use this information to understand website content hierarchy.

Sitemap priority

Sitemap priority values range from 1.0 (highest priority) to 0.0 (lowest priority) to indicate the relative importance of pages within a site. Think of it as a grading system for your web pages, guiding a search engine to understand which pages you deem more important.

Change frequency

Setting realistic changefreq sitemap tags can guide other search portals in determining the update cycle of website content.

It’s basically telling a search engine how frequently you update your content, giving them a schedule to follow for indexing your website. Again, Google doesn’t use the change frequency attribute but giving a little more information never hurts.

To further optimize your site, consider using the Google Search Console and Google Webmaster Tools for additional insights and recommendations.

Managing last modification dates

Finally, last modification dates, this one Google does consider.

The lastmod tag can be used to indicate the last modification time of the URL in the W3C datetime format. You can automatically set the lastmod attribute by passing an ActiveRecord object that responds to updated_at to the sitemap generator.

For dynamic content, database triggers or Rails model callbacks can be used to update a 'last modified' timestamp, ensuring the lastmod value remains accurate.

Once you’ve done all of this, you can use a sitemap validator w3c to check your file for any errors before submitting it to a search engine.

For a deeper dive into what the sitemap generator for Rails can do, see the comprehensive guide on GitHub.

Summary

The power of a well-structured, regularly updated sitemap cannot be overstated. It’s a vital tool for improving your website’s SEO and visibility. So, go ahead, use the knowledge you’ve gained from this guide and create a Rails sitemap that perfectly represents your Rails application. Remember, a well-crafted sitemap is more than just a map; it’s a blueprint for success.

Refine UX with superior sitemaps

Use our easy drag-and-drop interface to ensure people can get where they want to go.

14-day free trial
No credit card required

Frequently Asked Questions

  • What is a sitemap used for?

    A sitemap is a blueprint of your website that helps search engines find, crawl and index your site's content. There are different types, an HTML sitemap is a page of links that helps visitors navigate your website, for example, while a UX sitemap focuses on the entire user journey. To make organization easier, start with an example of a sitemap using our sitemap template

  • What's the importance of a sitemap generator gem for Rails applications?

    Using a sitemap generator gem like 'sitemap_generator' for Rails applications makes the process of creating sitemaps easier and more efficient, especially for larger sites with dynamic content. It helps keep the sitemap updated as the website grows and changes.

  • How can I make my sitemap accessible to search engines?

    You can make your sitemap accessible to search engines by adding a specific route in the routes.rb file of your Rails application, pointing to the sitemap's location in the public/ directory. You can also add a sitemap directly to Google Search Console.

  • What is a sitemap index and why is it beneficial?

    A sitemap index is like a table of contents for your sitemaps, making it easier to manage multiple sitemaps and prevent server slowdowns, which is especially useful for large websites.

  • What's the difference between a Rails sitemap and an ASPX sitemap?

    Both a sitemap in Rails and an ASPX sitemap use the XML format for search engines but the creation process differs. Rails often uses gems or background tasks in Ruby, while ASPX relies on manual configuration or plugins in ASP.NET. Ultimately, they achieve the same goal: helping a search engine discover your site's content.

Steve Tsentserensky

Want more free content like this?

Tips & tricks, how-to’s and deep dives delivered to your inbox 🚀

Refine your UX with superior sitemaps

14-day free trial
No credit card required

You might also like

What’s an ASPX sitemap & why your ASP.NET site needs one

What’s an ASPX sitemap & why your ASP.NET site needs one

An ASPX sitemap is an essential tool for any website built using ASP.NET. This structured file provides a map of all the pages and content on your website, making it…

Refine UX with a superior sitemap

Sign up