Last November, Google announced they were working on “mobile first” indexing. In summary, Google’s index has historically been based on the desktop version of a page (with added signals about the mobile experience used in ranking the page for mobile queries). However, as “most people” (according to Google) are now searching Google on mobile devices, Google began a plan for indexing pages based on the mobile version (with added signals about the desktop experience for ranking the page for desktop queries).

Now, Google has published new guidance and timelines on getting your site ready for their switch to mobile-first indexing. In addition, they’ve said that they’re switching the indexing method on a site-by-site basis (not for whole web at once) and have provided details on how to know if your site has been moved from desktop indexing to mobile indexing.

First, how to use Keylime Toolbox to analyze your site’s server logs to track when Google has moved your site from desktop indexing to mobile indexing. And then an explanation of how mobile-first indexing is different, how your site could be impacted, and how to prepare.

March 2018 update: Google ha announced they’ve begun moving sites to mobile-indexing. Also note that Google has clarified that they haven’t created a new index and therefore isn’t now serving results from two different indexes: a desktop-first index and a mobile-first index. Rather, Google is shifting how it’s gathering data for a site in a single, combined index.

Monitoring When Your Site Is Moved to Google’s Mobile-First Indexing

Google isn’t moving the entire web from desktop-first indexing to mobile-first indexing all at once. Rather, they are moving sites as they are ready (more on this below).

Tracking Crawling

Google’s John Mueller noted that for desktop-first indexing, 80% of Googlebot’s crawl is done by the desktop crawler and 20% is done by the mobile crawler. Once Google has moved a site to mobile-first indexing, most crawling will be done by Googlebot’s smartphone user agents instead. Google’s latest blog post reiterates this: ” Webmasters will see significantly increased crawling by Smartphone Googlebot.”

How can you monitor when Google starts crawling your site primarily with their smartphone user agents? By analyzing your site’s server logs for Google user agents. (Although note the caveats below if your site includes AMP pages.)

Google’s desktop user agent generally looks like this:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

And Google’s mobile user agent generally looks like this:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

For instance, the log file entry in total would look something like this (for a Google smartphone request):

66.249.79.81 – – [28/Nov/2017:17:01:13 +0000] “GET /example/page/ HTTP/1.1” 200 ”-“ “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

Keylime Toolbox can track this for you automatically. Our Crawl Analytics tool ($49/month per domain) tracks the number of URLs crawled by each bot (including Googlebot desktop and Googlebot smartphone user agents) and provides a daily Excel file with each URL crawled.

Below is an example of a site that Google is  crawling significantly more with its smartphone user agent than its desktop user agent:

One day detail in available Excel file:

One day detail in web-based report:

Trended data in web-based reports:

In addition to tracking overall trends, you can view detailed data about the crawl. For this example, note that about 10% of the responses that Googlebot desktop user agents is getting are “error” response codes, whereas that’s not the case for the Google smartphone user agent. Looking in more detail at the specific status codes returned, you can see that those “error” responses are 302s, so are likely due to the desktop user agent being redirected when accessing mobile pages.

The daily Excel file shows the exact URLs crawled and the corresponding status codes. In this case, you’d want to verify that the 302 redirects are in fact requests for mobile URLs being redirected to desktop pages and would want to check to see why the smartphone user agent is not showing similar numbers (as it should be redirected from the desktop URLs to the mobile URLs). In this example, the site includes AMP pages, and desktop user agents are being redirected to the desktop equivalents, so the higher number of redirects for only desktop user agents is as expected.

The example below shows a site that does not include AMP pages and you can see that the Google desktop crawl is much higher than the Google smartphone crawl:

AMP Considerations

Note that if your site includes AMP pages, the logs will not show 80% crawling by the Google desktop user agent even the site has not moved to mobile-first indexing. This is because for many configurations, Google requests the AMP URL (with the smartphone user agent) each time a user views an AMP page.

If your site includes AMP pages:

1. Look at a one day server log sample and filter out the requests to AMP URLs. With Keylime Toolbox, you can do this by downloading a log details Excel file, clicking the Google Web  Smartphone 200 tab, and filtering out the URL pattern that matches AMP pages.
   
2. Count the number of rows remaining once the AMP URLs are filtered out. (In this example, the count for filtered URLs is 382 vs. the overall Google smartphone user agent count of 28,654.)
3. Compare the filtered count with the count of URLs crawled by the Google desktop user agent. If the filtered number is still significantly higher than the Google desktop user agent count, then your site likely has been moved to mobile-first indexing. (In this example, the Google desktop user agent has crawled 9,51 unique non-AMP URLs vs. crawling 382 non-AMP URLs with the Google smartphone user agent, so the site has not been moved to mobile-first indexing.)
4. If the filtered count from the Google smartphone user agent is lower than the Google desktop user agent count, watch the summary trends. Once the Google desktop user agent count begins to decline and the Google smartphone user agent begins to increase (significantly), your site has likely been moved to mobile-first indexing (although you should repeat steps 1 and 2 to confirm).

Other Benefits of Server Log Data

Of course, server log data is useful for all kinds of reasons beyond just knowing when your site has been moved to mobile-first indexing. It uncovers technical issues search engines are having crawling the site, surfaces issues with canonicalization and parameters, highlights what parts of the site search engines are crawling most (and which they aren’t crawling at all), helps you track URL migrations (including HTTP to HTTPS), and helps you evaluate crawl efficiency (among other things).

If you’d like to check out Keylime Toolbox Crawl Analytics, email us at support@keylimetoolbox.com and we’ll work with you to set up a 14 day free trial. We’ll walk you through setting up an automated script to upload your server’s access logs to Amazon’s S3 so Keylime Toolbox can automatically process the crawl data daily and provide reports and details.

Tracking Ranking and Traffic

You can monitor any ranking changes with Google organic search by filtering the Google Search Console Search Analytics report by Device > Mobile. You can monitor traffic changes by filtering your web analytics data by mobile users. (You’ll want to do the same to monitor how the switch has impacted your site’s desktop search ranking and traffic.)

With Keylime Toolbox Query Analytics, you can do more granular monitoring. You can integrate both Google Search Console and Google Analytics data and can aggregate data from both mobile and desktop properties (if they’re separate) and then track traffic, ranking, and other SEO metrics for Google organic mobile search by category of query (such as branded and unbranded), can compare mobile and desktop ranking trends, and more. If you don’t already use Keylime Toolbox Query Analytics, you can sign up for a free trial or can email us at support@keylimetoolbox.com for more information.

In the example below, you can see that for 2017, Google organic desktop traffic is significantly higher than mobile traffic:

And average ranking is generally higher for mobile queries:

Note that if you see changes in Google’s mobile search results, that may not necessarily mean that your site has been moved to mobile-first indexing. Google has been making lots of other changes to their mobile search results. If possible, use the data from your server’s log files to determine if your site has been moved (as outlined above).

Once you see the site has been moved to mobile-first indexing, you can monitor ranking and traffic for both mobile and desktop search to see if the move has made an impact.

Note that Google has said the goal is for this change not to impact rankings much. And their latest post reiterates that “content gathered by mobile-first indexing has no ranking advantage over mobile content that’s not yet gathered this way or desktop content.”

What Is Google’s Mobile-First Indexing?

Google’s index consists of a URL and associated data. Google crawls with both a desktop and mobile user agent, but uses the desktop version to accumulate most data about the URL. Then, Google adds mobile-specific data (for use in search results for mobile users).

With mobile-first indexing, Google will still crawl with both a mobile and desktop user agent, but will use the mobile version to accumulate most data about the URL and will add desktop-specific data for use in search results for desktop users.

After the switch, Google will still continue to rank desktop pages for desktop searchers and mobile pages for mobile searchers.

Google is making the switch slowly and cautiously and don’t have a timeline for when it will be complete. In an initial blog post, they note that they are evaluating each site for mobile-first indexing readiness and are only transitioning them when they are ready. They note that they have transitioned a “handful” of sites and are monitoring those closely. In the more recent blog post, they indicate they have most past the handful of sites they were using for testing and are rolling out the change more broadly, but still only for sites that follow mobile-first indexing best practices.

How to Prepare for Google’s Mobile-First Index

In general, make sure your site provides a useful and fairly equivalent experience for both desktop and mobile users and make sure that the technical infrastructure is implemented correctly. See Google’s mobile-first indexing best practices for additional detail.

All Pages (Responsive/Adaptive Design or Separate Mobile and Desktop URLs)

• Content – are desktop and mobile users served the same content? Google recommends making sure that “the mobile version of the site has the important, high quality content” including text, images and ALT attributes, and videos. Their best practices say this particularly about sites that serve separate mobile and desktop URLs: “If your mobile site has less content than your desktop site, you should consider updating your mobile site so that its primary content is equivalent with your desktop site. This includes text, images (with alt-attributes), and videos – in the usual crawlable and indexable formats.” (This latter recommendation seems to be stronger than the initial guidance in making sure that all content is available on the mobile version.)

In some cases, mobile users see less content or see some content after user interaction (like clicking a “view more” button) and in the past, Google has said that content is valued less. However, Google’s Gary Illyes has said that “in the mobile-first world content hidden for ux should have full weight” and Google’s John Mueller has said that Google “will still treat as normal content on the page even if it is hidden on the initial view”.

Use the Google Search Console Fetch and Render tool using the Mobile: Smartphone option to make sure Google can retrieve and generate the content you expect and to make sure your aren’t blocking resources needed to build the page.

Smartphone Render:

Desktop Render:

• Links – is navigation the same for all users or do mobile users see fewer links? As noted above, the links can be behind a user interaction, but they should be viewable by mobile users. Internal links facilitate PageRank flow through the site.
• Structured markup – Both the desktop and mobile experience should include structured markup. Check this using Google’s Structured Data Testing Tool.
• Meta data – make sure titles, descriptions, and other markup are similar between the mobile and desktop experience.
• Mobile usability – check the Google Search Console mobile usability errors report for any reported issues.

Responsive (or Adaptive) Design

If your site uses responsive design and serves one URL to both desktop and mobile users, you may be mostly prepared already. As noted above, make sure that meta data, structured markup, and important content are the same for both experiences and make sure no issues exist with the mobile user experience.

Separate Mobile and Desktop URLs

If separate pages exist for mobile and desktop users, most of the recommendations are the same as they were before mobile-first indexing. But now is probably a good time to review those recommendations to make sure everything is in place correctly.

• Robots exclusion protocol – ensure that the mobile pages don’t include a noindex attribute and aren’t blocked with robots.txt. You can check for a noindex attribute by crawling the site with a tool such as Screaming Frog. You can check to make sure the pages aren’t blocked using the Google Search Console robots.txt Tester.

If you use Keylime Toolbox, you can also review the list of Google-reported crawl errors for the domain or subdomain that hosts the mobile pages. You can find this report by going to Downloads > Individual Sites/Subsites > [domain or subdomain] and then downloading the latest Web Crawl Errors file. Keylime Toolbox generates this report from Google’s API and it contains additional data (such as the list of URLs Google tried to crawl that are blocked by robots.txt) not available in the Google Search Console user interface reports.

• Canonical attributes – you don’t need to change these. Google has historically recommended that mobile pages have canonical attributes to the desktop equivalent URLs. They say that you don’t need to change for the mobile-first index as they’ll “continue to use these links as guides to server the appropriate results to a user searching on desktop or mobile”. Basically, they’ll continue to use the canonical attribute to map the mobile URL to the desktop URL.
• rel=alternate markup – you don’t need to change the rel=alternate media or rel=alternate hreflang markup (as long as they are already set up correctly). Basically:
  • all international mobile pages should map to each other and all international desktop pages should map to each other,
  • all mobile pages should map to the equivalent desktop pages using canonical attributes to the desktop pages,
  • all desktop pages should map to the equivalent mobile pages using the rel=alternate media attributes.
• Redirects – For the best user experience (and to keep signals to Google consistent), implement redirects based on the user’s device. When a mobile user requests a desktop URL, (ideally 302) redirect to the mobile URL and when a desktop user requests a mobile URL, redirect to the desktop URL.
• Google Search Console – If your mobile pages are on a different subdomain from the desktop pages, make sure you’ve verified that mobile subdomain in Google Search Console so you can monitor errors (along with other data. If you use Keylime Toolbox, you can either aggregate data from the mobile and desktop URLs in this case, or can view data for them separately in different reporting groups.
• Crawl capacity – Google recommends making sure “the servers hosting the site have enough capacity to handling potentially increased crawl rate.” Check out our guide on Google’s crawl budget and crawl efficiency for recommendations on making the crawl more efficient to make room for additional crawling.

No Mobile URLs

If your site doesn’t provide a customized experience for mobile users, Google will continue to crawl and index the desktop URLs. While it’s valuable for lots of reasons to provide a usable mobile experience, Google recommends that you wait until your mobile version is ready before launching it.

You might worry that if your site doesn’t have a mobile version, it will be dropped from indexing once Google switches to mobile-first, but that won’t happen. Some sites will always be desktop-only and Google will continue to crawl and index them. If you launch a mobile experience that’s broken or doesn’t work well, Google will crawl that and may not rank it well. Google has said: “If you only have a desktop site, we’ll continue to index your desktop site just fine, even if we’re using a mobile user agent to view your site.”

References

• Rolling Out Mobile-First Indexing (Google Webmaster Central Blog, March 2018)
• Introducing Mobile-First Indexing (Google Webmaster Central Blog, November 2016)
• Getting Your Site Ready for Mobile-First Indexing (Google Webmaster Central Blog, December 2017)
• Google’s Mobile SEO Overview (Google Developer Documentation)
• Google’s Best Practices for Mobile-First Indexing (Google Developer Documentation)
• Google Structured Data Testing Tool
• Google Mobile Usability Report
• Google Fetch and Render Tool