"Pure Spam" - Or, a lesson in shared hosting and DNS.
I maintain, or am associated with, a number of websites including some quite high profile and well respected ones. Imagine my shock when one of these disappeared from the Internet (well, Google anyway) entirely overnight.
Logging into Google Webmaster Tools I could see this was the result of a "Manual Action" by Google's team, specifically that my website was considered to be "Pure Spam"
The site appears to use aggressive spam techniques such as automatically generated gibberish, cloaking, scraping content from other websites, and/or other repeated or egregious violations of Google’s quality guidelines.
How could this be? All the content is original, nothing is automatically generated, no cloaking or any other questionable techniques are used.
This was also considered a Site-wide match by Google, meaning that the entire website (including subdomains) were delisted from Google. Even searching for the company by name wouldn't find it.
Use of Shared Hosting
Like many companies, we use virtual machines like those offered by Digital Ocean (there's a coupon here if you want to try them).
With these decided virtual servers you get your own static IP. The problem is, you don't know who's had it before or - when you delete the VM - who gets it next, as they're recycled regularly...
You don't know what the next owner is going to us it for.
Stale DNS
We had a number of subdomains (e.g. <something>.example.com) which pointed to Digital Ocean IP addresses. When the need for these VMs was gone, we deleted them but, crucially, did not delete the DNS entries.
One of these IP addresses was subsequently used by another Digital Ocean customer to create a website that would certainly meet Google's defininition of "Pure Spam". The trouble is, as well as being accessible on whatever domain name they had pointed to that IP address it was also accessible on our own domain.
As a result, because some_temporary_unimportant_host.example.com now presents a webpage containing gibberish, Google tars the whole of example.com with the same brush, and removed the entire website from the search listings.
No Notifications
Unfortunately, whilst the usual subdomains (like www.example.com, news.example.com and help.example.com) are all registered with Google's Webmaster Tools the domain in this case was not – the reason was simple, it was never used for any sort of web service.
As a result, there was never any report received of misuse of this domain. This is particularly frustrating, if Google had sent a notification (the parent domain was registered) then it would have been acted on immediately.
Moral of the story - check your DNS regularly!
If you maintain a large domain with many subdomains, audit your DNS configuration regularly. Make sure there are NO subdomains that point to IP addresses or CNAMEs that are no longer under your control, and ensure any unused domains are removed.
Some random non-web-related experiment like my_new_asterisk.example.com or couchdb_test.example.com that lives on an external hosting service for a short time could jeapordise your entire web presence if that domain is allowed to exist longer than necessary, and the IP address is reused for more nefarious purposes.