I like the web.[configuration].config replacement technique that is now available in Visual Studio, and for most files, you can now get this functionality with another plugin called SlowCheetah. However, SlowCheetah can't work with plain old .txt files like robots.txt. If you're like me, you really just need to swap out some robots.txt file that says "index my site" with another that says "don't index, you never saw this staging server". In this case, the solution is simple enough -- Pre-Build Event. Do the follow:
1) Right-click on your project and select "Properties".
2) Select "Build Events". Here is where we are going to add a single magic line.
3) Here is where we are going to add a single magic line which copies one of my versions to the root as robots.txt:
copy $(ProjectDir)Content\robots_txt\robots.$(ConfigurationName).txt $(ProjectDir)robots.txt /Y
Note that I am storing my different robots.[ConfigurationName].txt versions in /Content/robots_txt - you can put this anywhere that make sense in your project just change the first argument of copy command.
4) That's it! Change your configuration and then build the solution. You should see the different robots.txt file in your solution.