What is Robots.txt file

Robots.txt ek text file hota hai jo ki apne websiteor Blog rakte hai. Robots.txt file ke help se hum Search Robot ko ye batate hai ki apne websiteor blog ke kis page ko visit/crawl karna hai or kise nai. 

Robots.txt  txt file ko main directory me rakha jata hai.

Basic format of Robots.txt:

User-agent: [user-agent name]

Disallow: [URL string not to be crawled]

Example of Robots.txt:

User-Agent: *

Disallow: /wp-content/uploads/

Disallow: /wp-content/plugins/

Disallow: /wp-content/themes/

Disallow: /readme.html

Disallow: /tools/*

User-agent: Googlebot

Disallow: *?