The robots.txt file will tell the search engine robots what pages on your blog or website should be crawled and indexed and you really don’t want them to index your admin files because there is no relevant content there so creating a simple robots.txt file can actually improve your website’s crawlability and overall SEO.

Here’s an example of Dupe content from a pligg website :

http://buzz.seojr.com/InternetMarketing/Most-Profitable-Adsense-Keywords-List/

http://buzz.seojr.com/story/title/Most-Profitable-Adsense-Keywords-List/

Both of those URLs  point to the same page and will be indexed by search engines and there goes the flag, so here is the fix.

1. Open your NotePad.

2. Copy the text below and paste it into the file.

# All robots will spider the domain
User-agent: *
Disallow: /templates/
Disallow: /3rdparty/
Disallow: /libs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /internal/
Disallow: /backup/
Disallow: /thickbox/
Disallow: /api/
Disallow: /evb/
Disallow: /avatars/
Disallow: /admin_index.php
Disallow: /admin
Disallow: /story.php
Disallow: /login.php
Disallow: /user.php
Disallow: /js/
Disallow: /img/
Disallow: /story/
Disallow: /story/title/

3. Save the file on your computer as robots.txt

4. Now upload the newly created robots.txt file to you pligg rott directory overwriting the existing file if there is one already.There you have it, no more duplicate content.  Read more on Robots.txt and SEO.