Middleswarth

After several different configurations, I have not be able to get a robotstxt to function on a site by site basis. Part of the confusion is that multisite, single db, single codebase, with sites/mysite.com/modules/robotstxt installation instructions are a bit varied and I may just not have gotten the correct combination of tricks (serveralias, synlinks,staged installation of modules...). My primary difficulty is understanding how drupal manages the 'variable' table. Whenever I update the robots.txt file via the robotstxt.module UI a single variable 'robotstxt' is updated in the 'variable' table. For those using a prefix stategy 'ex: mysite_com_' which prefixes the table ( mysite_com_viariable), this would not be a problem. [For a long term fix, druapl may need to updates its multisite code, such that a new variable is created and used (ex: mysite_com_robotstxt) for persisting a unique variable for that module.]

My workaround:
Until I can figure out the multisite magic, here is my work around. I installed the robotstxt module inside the main /module area. Then I used the settings.php, located inside the site specific installation (sites/mysite.com/settings.php) file for customizing the sites in which I wish to have a unique robots.txt file.

$conf = array(
'robotstxt' => "User-agent: *
Disallow: /"
);

This is a bit simplified, since it blocks all robots from that site. This works great if you wish to only have bots hit you primary sites and avoid a specific set of secondary sites. It also allows you a single robotstxt UI to update you primary sites. The downside is that this misses the goal of multisite robotstxt UI. I look forward to your install.txt file to clarify how to properly install this module for a multisite structure with a shared db.

http://drupal.org/node/82559