Anyone can act like a bot just by using the Googlebot useragent in a request. Sometimes crawlers do that to see what other bots might see. Sometimes it’s to circumvent robots.txt directives that apply to them, but not to Googlebot. Sometimes people hope to get a glimpse at cloaking. Whatever the reason, these kinds of requests can be annoying since they make log file analysis much harder. Motivation for this excursion:

So you got your staging site indexed? Happens to everyone. Here’s a rough guide on fixing it, and suggestions for preventing it. (thought I’d write this up somewhere) The fastest way to get the staging site removed from search is remove it via Search Console. For that, you need to verify ownership via Search Console [1] (ironically, this means you’ll likely have to make it accessible to search engines again, or figure out DNS verification, which isn’t that common but also not that hard). From there, you can do a site-removal request [2], which will take the whole hostname out of Google’s search for ca 90 days. During this time, you can figure out and implement your general plan to block the staging site from search.

The “trailing-slash or not” question comes up from time to time, so I thought I’d write something short up. tl;dr: the slash after a hostname or domain name is irrelevant, you can use it or not when referring to the URL, it ends up being the same thing. However, a slash anywhere else is a significant part of the URL and will change the URL if it’s there or not. This is not SEO-specific, but just how websites work :).

“When people search for our event, they find last year’s listing; help!” Whether it’s an event (FooBarConf 2017), a recurring report (FooBar Earnings Q1 2017), an updated product (FooBarPhone 23) or anything else that has a current version and previous versions, this is a really simple way to help make sure that search is able to easily find the current version. Place the current version on a generic, non-versioned URL (/foobarconf) Copy last year’s version onto a versioned URL (/foobarconf/2016) Browse Twitter or Google+ (Technically the last step isn’t necessary, but if someone asks, you can say you’re just following instructions.) Doing it this way will help the generic URL to gain value over the years, while still allowing the older versions to be accessible if someone explicitly looks for them. Another advantage of this setup is that it’s trivial for folks to find and go to the current version. FWIW we get this wrong at Google, with our own content, from time to time too – it looks simple and obvious, and like many such things, you only notice it when things don’t work as expected.

How to make JavaScript-based sites work well with search is something I’ve been getting asked about a bit, so in the spirit of having open discussions, I set up a public working-group to discuss how things are working out :) Feel free to join and/or send folks there! https://groups.google.com/forum/#!forum/js-sites-wg (archive.org) My goal is to figure out how things are working out for sites at the moment, what tricks they’re doing to “fix” search, and what we need to change or document on our side. Some sites do crazy things, I think! We only see the end result for search, so it’s hard to know.

HTML validation and Google’s web-search … validation isn’t necessary for crawling, indexing, or ranking, as we’ve said [1] many times in the past. However, if you have problems with meta-tags, structured data, or link-elements (for example, if your hreflang markup isn’t being picked up properly), then using a HTML validator can sometimes point you at problems in your markup. In a recent example, a site was accidentally including a banner in the HEAD of a page. That might look ok in a browser, but it breaks the markup in the HEAD, causing us to potentially ignore the rest of it. A HTML validator can catch that for you. While validators aren’t perfect, they will help you get closer to clean markup, which will make it easier to automatically extract your site’s meta-data.

HTTP/2 & You? I occasionally hear webmasters ask about HTTP/2 and their site + web-search. The good news is that HTTP/2 doesn’t change the core concepts of HTTP, it doesn’t change the URLs, and is transparently supported for users & crawlers if they ask for it. Its primary differences focus on improved performance. Your hoster could add support for HTTP/2 and you might not even notice – who knows, maybe they support it already :). Additionally:

I have a large site and removed lots of irrelevant pages for good. Should I return 404 or 410? What’s better for my “crawl budget”? (more from the depths of my inbox) The 410 (“Gone”) HTTP result code is a clearer sign that these pages are gone for good, and generally Google will drop those pages from the index a tiny bit faster. However, 404 vs 410 doesn’t affect the recrawl rate: we’ll still occasionally check to see if these pages are still gone, especially when we spot a new link to them. 

HTTPS & HSTS: 301, 302, or 307? If the combination of these letters & numbers mean anything to you, you might be curious to know why Chrome shows you a 307 redirect for HSTS pages. In the end, it’s pretty easy. After seeing the HTTPS URL with the HSTS header (for example, with any redirect from the HTTP version), Chrome will act like it’s seeing a 307 redirect the next time you try to access the HTTP page. Your server’s not returning a 307, Chrome is just showing it to you as such to explain that it’s doing the redirect for you. You can confirm that by looking at the size of the response – 0 bytes for the 307 “redirect.” In other words, the 307 isn’t actually a redirect at all, it’s just a placeholder.

I was using my phone more on the weekend, and your mobile-friendly sites blew me away. Way too many of these are just horrible. Subscription interstitials, app interstitials, browser popups asking for my location, impossible to fill out search forms, login interstitials, tiny UI elements, cookie & age interstitials, “you’re in the wrong country, idiot” interstitials, full-screen ads, “add to homescreen” overlays, etc. One - popular & well-known - site had four levels of popups/overlays on a page. Four, Bob. Tip: it’s not a contest to see who can hassle the user in the most obnoxious ways. If people aren’t using your mobile website to convert and buy something, do you think this will help?