Recent blog posts
- Is A Yahoo Directory Link Worth It?
- Fast and Effective PHP Session Setup Example to Minimize Logouts and Problems
- Installing APC from beta/alpha development onto centOS 5.4 64 bit dual core
- Using Jquery, JSON, and AJAX in Drupal
- osDate and mysql Performance Optimization Tips
- osDate Permission Denied Behavior
- Landing Page and Referral Tracking For PHP/mysql osDate Site
- Installing Flash Video Chat With Your Own Chat Server or p2p
- Setting up DKIM, SPF, Domainkeys DNS, Regular DNS on CentOS 5.3 at Pacificrack.com
- Installing Bugzilla on centos - perl modules and other issues
Popular content
All time:
- Drupal Customization and Website Solutions By Peter Christopher at Palma SEO
- Creating a Custom Drupal Block With PHP
- How To Setup Drupal RSS Feed With FeedBurner and Adsense
- Setting up DKIM, SPF, Domainkeys DNS, Regular DNS on CentOS 5.3 at Pacificrack.com
- Moving A Single Drupal Install from default Directory to Multi-Site Drupal Installation
- robots.txt, nofollow, noindex, and Search Engine Behavior
- Installing Bugzilla on centos - perl modules and other issues
- Configuring A Few FCKeditor Attributes For Drupal 6
- About Peter Christopher
- Palma SEO Services Offered
- Yahoo Groups Are Dead
- Package Specials
- Page Rank Meditations on a Theme by Matt Cutts
- Change Open Realty URLs to Remove .html Suffix
- Multiple H1 Tags and Drupal
- How to Set Drupal FCKEditor Role Precedence Order for Users And Multiple Profiles
- Networking Vista and XP: Steps I Had to Take
- Fixing osDate captcha
- Installing Flash Video Chat With Your Own Chat Server or p2p
- Guarantee
- osDate and mysql Performance Optimization Tips
- How to Get a Copy of Open Realty Running on Your Local Machine For Development
- Be Careful With Clicksor: Their Javascript Embeds Links Even If You Only Add a Banner
- Installing Vista Service Pack 1 on Gateway cx210x Convertible Laptop With Upgraded Hard Drive, Linux, and Error 0x800F0826
- Take Stanford SEE AI Course For Free; Learn Machine Learning like Google's Founders Learned
- 1998: Amazon Snubs Google
- Check Out The State of Drupal Address by Dries From March 2009
- Thinking about SEO Search Engine Optimization When Implementing Drupal
- Does Using Google Toolbar Give Google Information That Will Give You A Site Penalty?
- To What Extent Does Google Use Visitor Time vs. Links in Page Rank / SERPs?
postscript - follow, noindex
As an experiment, I did something differently when I launched another site (Philippines Living). In this case, I set the default robot meta tag to be "follow,noindex". I expected that nothing would show up in the search results: all the pages would be either robots.txt'd out, or not indexed because of noindex.
I was wrong. Here's a few weeks after launch, when I still had the site set to "follow,noindex" (right before setting to the normal setting of "follow,index"):
So, what happened?
Google crawled all the pages not disallowed by robots.txt, and it saw the links to the robots.txt disallowed URLs, but didn't officially crawl them. Google then did not index any of the crawled pages, because it knew that the pages were "noindex" in the meta tag. But, it did index the links to the robots.txt disallowed URLs! Of course, they have no titles or descriptions, because the googlebot doesn't officially crawl them... but, yes, the only pages google indexed were the robots.txt barred pages. Pretty amusing. I'll make another entry when I see that google has figured out that the pages are indexable and put them in the online index. We'll see how long past January 4th that is.