Most websites powered by CMS (Joomla, WordPress, etc.) have files and folders that are not relevant for search engines (like images or admin files) amd Pligg is no exception with story pages having several URL’s that get indexed by the search engines and it would be flagged as duplicate content. The most effective and easiest way to fix this is by adding a robots.txt file to your root folder.

The robots.txt file will tell the search engine robots what pages on your blog or website should be crawled and indexed and you really don’t want them to index your admin files because there is no relevant content there so creating a simple robots.txt file can actually improve your website’s crawlability and overall SEO.

Here’s an example of Dupe content from a pligg website :

http://buzz.seojr.com/InternetMarketing/Most-Profitable-Adsense-Keywords-List/

http://buzz.seojr.com/story/title/Most-Profitable-Adsense-Keywords-List/

Both of those URLs  point to the same page and will be indexed by search engines and there goes the flag, so here is the fix.

1. Open your NotePad.

2. Copy the text below and paste it into the file.

# All robots will spider the domain
User-agent: *
Disallow: /templates/
Disallow: /3rdparty/
Disallow: /libs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /internal/
Disallow: /backup/
Disallow: /thickbox/
Disallow: /api/
Disallow: /evb/
Disallow: /avatars/
Disallow: /admin_index.php
Disallow: /admin
Disallow: /story.php
Disallow: /login.php
Disallow: /user.php
Disallow: /js/
Disallow: /img/
Disallow: /story/
Disallow: /story/title/

3. Save the file on your computer as robots.txt

4. Now upload the newly created robots.txt file to you pligg rott directory overwriting the existing file if there is one already.There you have it, no more duplicate content.  Read more on Robots.txt and SEO.