Understanding the sitemap protocol is essential for any website owner looking to optimize for search. Why? Because it’s what tells search engines about the pages on your site and their importance, directly affecting your site’s visibility.
Let’s look into what sitemap protocol is so you can create, manage and submit your XML sitemap and improve your SEO results.
Key takeaways
- XML sitemaps are what search engines use to crawl and index a website, with a maximum limit of 50,000 URLs or 50MB per sitemap, and should include metadata such as the last update date and page priority.
- Creating an effective sitemap involves organizing URLs into a logically structured hierarchy, using a tree layout with the homepage at the top, and ensuring the inclusion of canonical URLs to prevent content duplication.
- For large websites, utilizing multiple sitemaps and a sitemap index file is necessary to manage and ensure efficient crawling of all pages, while keeping individual sitemaps under the size and URL limits.
Understanding the sitemap protocol
Your XML sitemap is chock full of info about your site’s content and the sitemaps protocol is what turns that code into meaningful data for search engine crawlers.
Each URL in these sitemaps carries a wealth of information in the metadata, which includes details like the last update date, change frequency and priority relative to other pages on your site.
Sitemap protocol format
For a search engine to pick up what you’re putting down in an XML sitemap, you need to be speaking their language and working within their constraints.
That language, as you might’ve gathered, is XML or extensible markup language.
The sitemap protocol format consists of these must-have tags:
| Required tags | Purpose |
|---|---|
<urlset> | Defines the protocol format and XML schema in use, i.e "xmlns= https://www.sitemaps.org/schemas/sitemap/0.9" |
<url> | Denotes the starting point of all information you’ll give about a page |
<loc> | The literal location of a web page, i.e. www.yoursite.com |
Additionally, you can include optional tags:
| Optional tags | Purpose |
|---|---|
<lastmod> | The date when content on the page last received a significant update |
<changefreq> | Indicates how often a page’s content changes, possible values are: always, hourly, daily, weekly, monthly, yearly and never |
<priority> | Describes the relative importance of a page on a 1.0 to 0.0 scale, 1.0 being most important |
Of those three, the <lastmod> tag is the one you’ll want to include and each tag above will need to have an associated end tag as well; </urlset>, </url>, </loc>, </changefreq>, </priority> and </lastmod>.
Accuracy in encoding and careful use of tags are paramount for your XML file to be read correctly, to that end, these special characters &, ', ", < , > require entity escape codes to be interpreted correctly. Otherwise, they’d be read as code, causing indexing issues — not good:
| Character | Literal | Escape code |
|---|---|---|
| Greater than | > | > |
| Less than | < | < |
| Apostrophe | ' | ' |
| Quotation mark | " | " |
| Ampersand | & | & |
Lastly, make sure your XML sitemap file is UTF-8 encoded, which is the default if you don’t add any other encoding information. Same goes for other formats like a Txt file sitemap.
The XML sitemap format is pretty versatile, supporting extensions for various content types like images, videos and news.
But why are these sitemaps so important? And why does your website need one? Time to dig a little deeper.
The purpose of sitemaps
Sitemaps guide search engines in the right direction.
They facilitate the crawling and indexing process, making it easier for a search engine to understand your website’s structure and content. Providing detailed information about your site’s pages and files, including update times and alternate language versions, sitemaps serve as a detailed guide to all the important work you’ve created.
Side note, there’s a distinction between the XML sitemaps we’re covering here, which are optimized for search engines and HTML sitemaps, which are made for human navigation. It’s worth understanding the XML vs HTML sitemap difference and having both. That’s just the type of the iceberg, there are many different types of sitemaps out there that suit various purposes.
To create a sitemap is one thing, to optimize it to give it the best shot at being crawled and indexed is another.
🎬 Learn what Slickplan can do!
We filmed a short video to show you exactly how to use Slickplan
Creating an effective sitemap
A sitemap is a reflection of your site’s architecture and a way to easily convey it to a search engine. Using a visual sitemap generator helps you illustrate the connections between pages and define the hierarchy.
Optimizing a sitemap for search means including the right pages, staying organized and ensuring you fall under the size limits.
Choosing the right URLs
Your sitemap should include the important pages that are of value for users as well as utility-type pages. The idea is to incorporate anything you’d want a search engine to index so they can show it to users in the SERPs.
Make sure all links are canonical URLs — these are the preferred URLs for displaying in search results, which helps avoid content duplication.
Remember, your sitemap should serve as a beacon for useful, indexable pages that contribute to the user experience and your site’s relevance.
Organizing your sitemap
Hierarchy, hierarchy, hierarchy.
An organized XML sitemap is a more easily indexed sitemap.
When you design a sitemap, consider using a tree hierarchy, with the homepage at the top, followed by content categories and individual pages beneath.
Clear main categories, subcategories and consistent navigation form a crucial part of a logical sitemap structure.
SlickTip: If you’re having trouble getting your site organized, tools like visual sitemaps user flow diagrams and an AI sitemap generator can help you get clarity when mapping your website architecture.
Separating sitemaps by content, categories or page types enhances management efficiency.
Managing large websites with multiple sitemaps
XML sitemaps have their limitations.
Those limits, to be exact, are 50,000 URLs and a maximum uncompressed file size of 50MB.
To overcome those limitations, you’ll need to break your sitemap into smaller files and submit them to your search engine of choice.
Introducing the sitemap index file.
Sitemap index files
A sitemap index is a separate file that references multiple XML sitemaps, ensuring all XML files and sitemap files of various types can be crawled and indexed.
Consider them the main entry point to reference multiple sitemap files, simplifying management and submission to search engines.
A single sitemap index file can include up to 50,000 'loc' tags, each of which points to a distinct sitemap, facilitating the inclusion of numerous pages.
For expansive websites, a sitemap index file can reference other sitemap index files, creating a multi-level structure that maintains manageable file sizes and optimizes the indexing process.
Splitting sitemaps
The goal is to give web crawlers the most organized roadmap possible, sometimes that means splitting your sitemap into different parts.
A large sitemap can be split into smaller files for better management.
When making the separation, it’s crucial that all the sitemaps are located in the same root directory or a subdirectory relative to the location of the sitemap index file.
Large sites, new websites with limited external links and sites with considerable media content or news articles can significantly benefit from using more than one sitemap to ensure thorough indexing of their web pages.
Sitemap validation and submission
Before you submit sitemaps to search engines, you’ll want to validate them and check for any potential formatting errors.
You can pop your sitemap into our free XMLsitemap validator and fix any issues that may arise.
Once validated, you can submit your sitemap, or sitemaps, to Google, Bing and the like for indexing of your URLs.
Once submitted, you’ll have access to helpful status information, processing errors and additional data.
Troubleshooting common sitemap issues
Navigating sitemap errors may seem daunting, but Google Search Console serves as your trusty sidekick, providing insights into what to do when faced with errors and warnings.
Handling errors and warnings
Review the details in the Sitemaps report to resolve errors by addressing the specific issues highlighted by Google.
Ensuring that each URL in your sitemap is fully specified, including the protocol and any necessary URL escape codes, can resolve submission errors.
An empty sitemap error can be corrected by confirming that the sitemap includes URLs that are accurately tagged according to sitemap guidelines and protocol.
For HTTP errors, verify the sitemap URL for accuracy and consider revisiting the URL structure if persistent issues occur before resubmission.
Incorrect namespaces in your sitemap can be resolved by revising the specified namespaces, the "xmlns" code mentioned above, to be in line with the standard format before resubmitting.
Summary
From purpose to organization to submission, we covered a lot of ground. Creating an effective XML sitemap is about more than just getting a list of URLs together though. The XML tags used in sitemap protocol are how you translate your pages and site structure into something a search engine can understand, navigate and index effectively.
Design user-friendly sites with Slickplan
Use our easy drag-and-drop interface to ensure people can get where they want to go.
Frequently asked questions
What's the purpose of a sitemap?
The purpose of a sitemap is to facilitate the crawling and indexing process for search engines by providing detailed information about all the URLs and files of a site. However, it doesn't guarantee the indexing of all items listed within it.
What is the format of a sitemap?
There are many sitemap formats, from XML to HTML to a text file and beyond, the XML format is limited to 50,000 URLs or 50MB per individual sitemap. It's important to use UTF-8 encoding and escape codes for special characters as well.
What is the benefit of multiple sitemaps for large websites?
Using multiple sitemaps for large websites ensures thorough indexing and is especially beneficial for new websites with limited external links or sites with lots of media content or news articles.
How to handle errors and warnings in sitemaps?
To handle errors and warnings in sitemaps, address the specific issues highlighted by Google, ensuring that each URL is fully specified and accurately tagged according to sitemap guidelines.
What is the importance of keeping the XML sitemap up-to-date?
Keeping your XML Sitemap up-to-date gives search engines the most current view of your website and helps prevent indexing issues by removing old or non-existent pages. See our update sitemap guide for details.
Facebook
Youtube
Twitter
LinkedIn
Vimeo
Codepen
Dribbble
Pinterest
