Often the internal structure of links of a blog is not properly optimized in terms of content, often creating duplicate content, something that Google is fighting ferociously, reducing the number of visits by googlebot and penalizing the positions in the pages of search if it finds duplicate content.

ShoeMoney said that through a proper optimization of the robots.txt file has traffic increased 1400%, an impressive increase by only properly optimize a text file.

Basically, the optimization is based on filtering all those blog duplicate content, like categories, feeds, files by date, etc, which although initially tend to increase the number of pages indexed, after an analysis by Minusio Google’s this backfires, thus affecting all pages indexed, both the quality and content that generate “noise.”

AskApache not give some examples of robots.txt file optimized for most common platforms that you can use as a base to undergo change.

However, a word of caution: An incorrect statement in the robots.txt file can leave out all the search engines, therefore to use caution.

So, the sites that lost these rankings are scrambling to find some answers as to why their site dropped. While it’s my business to know the intricacies of how this particular update impacted the search algorithm, there are some common ground starting points that if you apply these to all of your sites, you should be able to survive any update intact.

• Proper naming structure
• Name your page titles with your keywords if possible
• Always have a sitemap
  
• Always include a robots.txt file
• If you must use a re-direct, be sure it’s server side, not with a meta refresh tag
• Don’t use hidden text
• Make sure your keyword phrase is included in your H1 tags
• Don’t optimize for more than 2 keywords per page
• Use text links where possible
• In any product image, be sure to use the alt tag
• Use hyphens, not underscores when you name a page file
• Make sure your site has an error handling page
• Create a Google Sitemap and submit it to them (This is in addition to a typical sitemap)
• Offload all your js and css code
• Don’t forget about meta tags

Be sure to follow these simple guidelines and you won’t need to worry too much anytime Google has an update – your site will not be impacted by any filters that are checking for spamming, hidden text, or anything that resembles cloaking.