SEO 101: When Your Site Isn’t Indexed At All

From the archives…

Yesterday, someone asked me to take a look at their blog. It had been indexed in search engines fairly well, but had been migrated to WordPress a couple of weeks ago and since then, everything had taken a dive. They wondered how long it took Google to follow redirects.

I started checking things out. Were all the old URLs redirecting one-to-one to the new URLs with 301s? Yep. Do the 301s seem to be redirecting correctly? Yep. How does the robots.txt file look? It’s allowing everything. Looks good so far.

Then, I checked out the site in Google Search Console. Which is a pretty handy program. You should all really looking into it! I noticed that the query stats weren’t reporting any data. This could mean that the site hasn’t come up for any searches lately. Not a good sign. Then I checked the crawl rate information. Indeed, the crawl was slowing down.

Huh. My first guess was that it was taking a while for Google to pick up the redirects because it wasn’t crawling the site very quickly. So, I got set to figure out why that might be. But then I thought I’d just check one last thing.

And sure enough. There it was. In the source code of every page.

<meta name="robots" content="noindex, nofollow">

The search engines were following the redirects and getting a locked door.

What likely happened is that the designer put up the meta tag during the development phase and forgot to take it off once the site went live.

We could have spent days investigating the redirects and how they were being followed and indexed and speculated about possible penalties or sent out perky yet persuasive link exchange requests in a far-flung effort to increase PageRank and kick start crawling, but sometimes, the simplest explanation is the right one.

The first rule of indexing: make sure you’re letting the site be indexed.

As a sidenote, through Google Search Console, we found that some of the old pages were giving 403 errors, which means that when they shut down the old blog, they password protected some of the pages. Obviously, search engine bots won’t be able to follow redirects to the new pages if they can’t crawl the old ones, so watch for shutting things down too quickly. We also found a list of old pages that were returning 404s, which was a handy way to see what redirects were missed.

I think that meta tag thing might mostly do the trick though. Just a hunch.

20 thoughts on “SEO 101: When Your Site Isn’t Indexed At All

  • Pingback: Why One Of My Sites Took A Dump In Google

  • JLH

    Blogger had a bug/feature not too long ago where it installed that same meta tag on a bunch of blogs.

    Using firefox with the Search Status add-on makes pages like that light up PINK if you’ve got the highlight nofollow links checked. The diagnosis took seconds then.

    Reply
  • Brendan Picha

    Reminds me of this one time when I was working for a search marketing firm in Boston. Some guys couldn’t figure out why this desktop wasn’t working. They took it apart and were working hard to get to the source of the problem. After everything…it was simply a blown power box.

    Reply
  • aaron

    Vanessa- I am not a fan of the “social” internet (though I got a small fleet of cats including a black one identical looking to yours I call “Chuckie”) BUT this post interests me… not exactly that you are telling me anything I do not already know but it gives me the opportunity to try to bounce a new question off you. =P

    I recently changed the permalink structure in my wordpress blog, all posts used to end in .html now end in /. Logically one would assume that Google requires a 301 redirect from all the old URLs to the new but I have not done any yet.

    Google seems to slowly be reindexing the pages that end in / without a 301 redirect?

    Am I risking having my site all messed up or will Google eventually get ALL the pages indexed correctly?

    Also, do you see smarter algorithms that will go beyond and not require redirects in the future? I do! πŸ˜‰

    Reply
  • Vanessa

    aaron, 301 redirects aren’t required. The new URLs will likely eventually get indexed. Googlebot will find them over time (for instance, when you do a new post, your blogging software probably sends out a ping, Googlebot grabs the post based on that, then can follow links from there back to older posts).

    A 301 redirect will probably make indexing faster, since Googlebot generally returns to pages it already knows about to see if anything has changed and then can quickly see the redirects.

    Do the old pages 404 or do they still exist? If they still exist you may have an issue where the old pages show up in search results rather than the new ones from time to time.

    As for the future? Heck yeah, algorithms will continue to get smarter!

    Reply
  • Aaron Pratt

    “Do the old pages 404 or do they still exist?”

    They 404, ain’t WordPress extremely cool? πŸ™‚

    Thanks again, this question went unanswered in Google Groups BTW. ;-(

    Reply
  • Yuri

    Yeah, but since links point to old post URLs, it’d better have 301s, right? Or will Googlebot understand that the new pages are infact old pages and take the old links into account?

    Oh wait. You don’t work for Google any more. Just curious, though πŸ˜‰

    Reply
  • dockarl

    Good catch! Isn’t the first, definitely not the last!

    Vanessa – as a wordpress user you might be interested in this (http://www.utheguru.com/seo_wordpress-wordpress-seo-plugin).

    I’m not ‘plugging my plugin’ but there is a hell of alot of buzz at the moment around the web about ‘shaping pagerank’, and more particularly using rel=nofollow internally to ‘herd’ the bots..

    I think some folks go a bit overboard when they talk about using rel=nofollow to stop pages ‘bleeding’ pagerank to other pages within the site – but one thing that does interest me from a purely academic perspective is whether or not using rel=nofollow in a link to a page is essentially the same from an seo perspective as using meta noindex,nofollow.

    I’ll put it another way – we all know rel=nofollow is meant to stop the transfer of PR from source to target. We all assume this applies irregardless of whether the target is external or internal.

    Now assume that a page simply has meta noindex, nofollow – do internal links to that page (without a link condom) still ‘bleed’ PR to that page?

    we’ve been chatting about this and a bit of other stuff over in this thread.. it would be cool to have your insight πŸ™‚

    Matt

    Reply
  • dockarl

    Oh.. oops – I muffed up the link to my wordpress plugin (of more interest than my rambling at the beginning of that post is the user discussion afterwards, particularly Milan’s comments) – here is the link

    On a tangent – it’s cool to see you’re no longer requiring registration to post comments on this blog πŸ™‚ Plus it would be cool to hear a bit more about zillow..

    M

    Reply
  • Vanessa

    Yuri, it’s definitely better to 301. With the structure that Aaron describes, search engines will eventually deindex the old URLs and index the new ones but it will take a bit of time and the link credit to the old URLs won’t transfer to the new ones. With 301s, reindexing will be faster and the link credit will transfer.

    Matt, I’ll take a closer look at the plugin and thread tomorrow and let you know my thoughts. I would look now but it’s 1am and as noted in a comment in another thread, I just got back from the bar. πŸ™‚

    Reply
  • Maurice

    πŸ™‚

    Remind me of the time when one of my clients managed to block *.php via robots.txt – they where trying to get cute with mode rewrite.

    The other thing to watch with wordpress is make sure the ping servers are setup – I normally turn of all updates while we are developing just so test posts don’t get pinged and we have hada couple of sites where someone forgot to turn the ping servers back on – oops!

    Reply
  • dockarl

    Aaah.. Aaron – just saw your post – I think I’m right in saying that even though your new URL’s will redirect just fine, if the old ones are throwing 404’s you’re going to lose the pagerank and anchor text value that those old pages had accumulated, which isn’t what you want (especially if it is an established blog like seobuzzbox).

    Under these circumstances I’d suggest reversing your permalink changes, then installing Dean’s Permalink Redirect Plugin for WordPress, then reimplementing the permalink changes.

    That should ensure that the old pages don’t 404 anymore, and that your new url’s get the full benefit of accumulated PR.

    M

    Reply
  • dockarl

    Thanks Vanessa, that would be appreciated πŸ™‚

    Just been at a bar – Vanessa? NEVER – I don’t believe it for a second πŸ˜‰

    Reply
  • aaron

    Dax made a great redirect plugin for WordPress, I just went and changed ALL the URLs and redirected them, what a time saver.
    http://www.webmaster404.com/redirect-all-wordpress-plugin/

    Vanessa – Quick question, Google has been trying to figure out my URLs for a week or two now (without having any 301 redirects in place). Are the 301s just initiated too late for receiving “link credit”?

    I know, this stuff seems obvious to Google employees and spammers but for the average webmaster, it is a head scratcher!

    I could have just forced most of my pages into supplementals because I didn’t redirect, doh!!!

    Thanks

    Reply
  • Vanessa

    It’s not too late to do redirects. Even if Googlebot knows about the new URLs, the redirects will still help with link credit.

    It looks like Dax’s plugin redirects all 404s to the home page, which is cool, but if you have old posts that used to have one URL structure and now have another, you probably want to do a one-to-one redirection from the old to the new.

    The plugin dockarl linked to may do that — I’m not sure.

    Reply
  • dockarl

    Yep Vanessa / Aaron – Dean’s plugin 301 redirects on a one to one basis. It’s worth having a look at the code – it’s quite novel how he gone about it.

    M

    Reply
  • aaron

    Sorry, I linked the wrong plugin, it is one called “objection redirection” that was also made by Dax and can be found on Greg Boser’s blog here:

    http://tools.webguerrilla.com/wordpress-redirection-plugin/

    I like the part in the plug-in where is says, “redirect their ass”, check it out, it is very useful for 1 to 1 redirects. 195 redirects took me just 20 minutes to do, I cut and pasted the “pages not found” URLs from webmaster tools.

    Yes, “redirect all” also has it’s uses, I used it to scramble my SEO blog when I deleted old posts. πŸ˜‰

    Another while Vanessa is hot – I turned my “rain barrel” site into a wordpress blog and am now not sure it was a good idea, I seems to bounce around for my “keywords” much more than when it was a static site which stinks during gardening season. I notice one page sites that sell similar products doing much better and are much more stable, true?

    I would love to have you do a little assesement of my site Vanessa, there are questions that can never get answered in noisy places like Google groups or SEO forums that are filled with trolls looking for “clients”.

    I will email you the url if you have the time, sometime?

    Reply
  • dockarl

    Dude – that 20 minutes sounds ok – but seriously – dean’s just does it automatically.. You must have too much time on your hands πŸ˜‰

    Reply
  • aaron

    Ok ok, I will check “Dean’s” out tonight when the family is all tucked in their beds while visions of sugar plums dance in their heads…

    πŸ™‚

    Reply
  • vishal

    one of my blog is indexed in google but the other one is not.
    this is indexed techsupportforpc.blogspot.com but
    thegreatsonunigam.blogspot.com is not

    Reply

Leave a comment to Maurice Cancel reply

Your email address will not be published. Required fields are marked *