モバデビのrobots.txtファイルの記述内容です。

mbdb-robots.txt

User-agent: *
Disallow: /wp-admin
Disallow: /wp-includes
Allow: /wp-content/uploads
Allow: /wp-includes/js
Disallow: /*?*
Disallow: /*?
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: */feed/
Disallow: */trackback/

# allow Googlebot to css/js
User-agent: Googlebot
Allow: /*.js$
Allow: /*.css$

# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow: /wp-includes/
Allow: /wp-content/uploads/
Allow: /wp-content/uploads-*
# allow Google adsense bot on entire site
User-agent: Mediapartners-Google*
Allow: /*