1. Search Engine Crawl Error Pages

It is absolutely crucial to monitor search engine crawl error reports to keep track of how your website and pages are performing. Monitoring error reports will help webmasters determine when and where Googlebot or another search engine crawler is having trouble indexing your website content – and will help you find a solution to the indexing problem.

2. sitemap and robots.txt files

These files are supported by all major search engines and are incredibly useful tools for ensuring that SE crawlers index your important site content while avoiding those sections/files that you deem to be either unimportant or cause problems in the crawl process. I’ve seen the proper use of these files make all the difference between a total crawl failure of a site and a full index of content pages. And this makes them crucial from an SEO standpoint. You should create and/or update robots.txt and sitemap files every time changes occur.

3. Check Googlebot activity reports

These free reports(most hosts provide this) lets you to monitor how long it’s taking Googlebot and any other search engine crawlers to access your web pages. This information can be very important if you are on a slow network or experiencing web server problems. If it is taking search engine crawlers a long time to index your pages it may be the case that there are times when they “time out” and stop trying. Additionally, if the crawlers are unable to call your pages up quickly you can be certain users are experiencing the same lag in load times, and we all know how impatient Internet users are and can be.

4. Your site in browsers with image and JavaScript support disabled

Well, the best ways I found to determine what your site looks like to any search engine crawler is to open your pages in a browser without image and JavaScript support. Mozilla’s Firefox browser has a plug-in available called the “Web Developer Toolbar” that adds this functionality and a lot more to the popular standards-compliant browser. If after turning off image and JavaScript support you aren’t able to make sense of your pages at all, it is a very good sign that your website is not optimized well for search engines. I know that  images and JavaScript can add a lot to the user experience they should always be considered as a “luxury”, or simply put an improvement upon an solid textual content base on your pages. So don’t go overboard with images and java. You have to find a good balance between the two.