When having development servers that clients needs to access, but you don't what search engines to index one solution is to protect the server with HTTP authorization implemented using PHP. This post while explain how to do just this on an Aegir hosting solution.

The users browser don't cache static content, if your web-server don't sets the right expire or max-age headers. Drupal sets these headers for static content in the .htaccess file that comes with Drupal, if you have the expire Apache extension enabled.