Question: ROBOTS.TXT FILE
Answer: The robots.txt protocol, also known as Robots Exclusion Protocol, is a file which informs the search engine not to crawl or index the file mentioned. This file is used to restrict the indexing of admin panel and other private information placed on the site. The robots.txt file can be written in any way according to the requirement:- 1) Here * specifies all the robots. The above syntax allows the bots to crawl all the files. User-agent: * Disallow: 2) This sitemap restricts all the robots from indexing the site. User-agent: * Disallow: / 3) The example below restricts the bots from indexing the three directories mentioned. User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /junk/ 4) This example restricts the bots from only indexing only the directory mentioned. User-agent: * Disallow: /directory/file.html 5) We can also select the bot we do not to index our site. User-agent:BadBot #replace 'BadBot' with the actual user-agent of the bot Disallow: / 6) We can also restrict a specific bot from indexing specific files. User-agent:BadBot# replace 'BadBot' with the actual user-agent of the bot Disallow: /junk/ Name of some of the bots of Google are- googlebot ( all services), googlebot-news ( bot for news), baiduspider, bingbot, yandexbot, etc.
āĻŦাংāϞা āĻ
āύāĻŦাāĻĻ ঃ
āĻĒ্āϰāĻļ্āύ: āϰোāĻŦāĻāϏ.āĻিāĻāĻ্āϏāĻি āĻĢাāĻāϞ
āĻāϤ্āϤāϰ: āϰোāĻŦāĻāϏ āĻেāĻ্āϏāĻ āĻĒ্āϰোāĻোāĻāϞ, āϝা āϰোāĻŦāĻāϏ āĻāĻ্āϏāĻ্āϞুāĻļāύ āĻĒ্āϰোāĻোāĻāϞ āύাāĻŽেāĻ āĻĒāϰিāĻিāϤ, āĻāĻāĻি āĻĢাāĻāϞ āϝা āĻ
āύুāϏāύ্āϧাāύ āĻāĻ্āĻিāύāĻে āĻāϞ্āϞিāĻিāϤ āĻĢাāĻāϞāĻি āĻ্āϰāϞ āĻŦা āϏূāĻী āύা āĻāϰাāĻে āĻ
āĻŦāĻšিāϤ āĻāϰে। āĻāĻ āĻĢাāĻāϞāĻি āĻ
্āϝাāĻĄāĻŽিāύ āĻĒ্āϝাāύেāϞ āĻāĻŦং āϏাāĻāĻে āĻĨাāĻা āĻ
āύ্āϝাāύ্āϝ āĻŦ্āϝāĻ্āϤিāĻāϤ āϤāĻĨ্āϝেāϰ āϏূāĻāĻে āϏীāĻŽাāĻŦāĻĻ্āϧ āĻāϰāϤে āĻŦ্āϝāĻŦāĻšৃāϤ āĻšāϝ়। āĻĒ্āϰāϝ়োāĻāύ āĻ
āύুāϏাāϰে āϰোāĻŦāĻāϏāĻিāĻিāĻāϏāĻি āĻĢাāĻāϞ āĻোāύāĻāĻাāĻŦে āϞেāĻা āϝেāϤে āĻĒাāϰে: - ā§§) āĻāĻাāύে * āϏāĻŽāϏ্āϤ āϰোāĻŦāĻ āύিāϰ্āĻĻিāώ্āĻ āĻāϰে। āĻāĻĒāϰেāϰ āϏিāύāĻ্āϝাāĻ্āϏāĻি āĻŦāĻāĻুāϞিāĻে āϏāĻŽāϏ্āϤ āĻĢাāĻāϞ āĻ্āϰāϞ āĻāϰাāϰ āĻ
āύুāĻŽāϤি āĻĻেāϝ়। āĻŦ্āϝāĻŦāĻšাāϰāĻাāϰী-āĻāĻেāύ্āĻ: * āĻ
āϏ্āĻŦীāĻাāϰ āĻāϰুāύ: 2) āĻāĻ āϏাāĻāĻāĻŽ্āϝাāĻĒāĻি āϏāĻŽāϏ্āϤ āϰোāĻŦāĻāĻুāϞিāĻে āϏাāĻāĻেāϰ āϏূāĻিāĻāϰāĻŖ āĻĨেāĻে āϏীāĻŽাāĻŦāĻĻ্āϧ āĻāϰে। āĻŦ্āϝāĻŦāĻšাāϰāĻাāϰী-āĻāĻেāύ্āĻ: * āĻ
āϏ্āĻŦীāĻাāϰ āĻāϰুāύ: / 3) āύীāĻেāϰ āĻāĻĻাāĻšāϰāĻŖāĻি āĻāϞ্āϞিāĻিāϤ āϤিāύāĻি āĻĄিāϰেāĻ্āĻāϰিāĻে āϏূāĻীāĻāϰāĻŖ āĻĨেāĻে āĻŦāĻāĻুāϞিāĻে āϏীāĻŽাāĻŦāĻĻ্āϧ āĻāϰে। āĻŦ্āϝāĻŦāĻšাāϰāĻাāϰী-āĻāĻেāύ্āĻ: * āĻ
āύুāĻŽāϤি āĻŦাāϤিāϞ / / āϏিāĻি-āĻŦিāύ / āĻŦাāϤিāϞ / / āĻিāĻāĻŽāĻĒি / āĻŦাāϤিāϞ: / āĻাāĻ্āĻ / 4) āĻāĻ āĻāĻĻাāĻšāϰāĻŖāĻি āĻেāĻŦāϞ āĻāϞ্āϞিāĻিāϤ āĻĄিāϰেāĻ্āĻāϰিāĻিāĻে āĻেāĻŦāϞ āϏূāĻীāĻāϰāĻŖ āĻĨেāĻে āĻŦāĻāĻে āϏীāĻŽাāĻŦāĻĻ্āϧ āĻāϰে। āĻŦ্āϝāĻŦāĻšাāϰāĻাāϰী-āĻāĻেāύ্āĻ: * āĻ
āύুāĻŽāϤি āĻŦাāϤিāϞ / / āĻĄিāϰেāĻ্āĻāϰি / āĻĢাāϝ়াāϞ āĻāĻāĻāĻিāĻāĻŽāĻāϞ 5) āĻāĻŽāϰা āĻāĻŽাāĻĻেāϰ āĻŦুāĻāĻিāĻ āĻāĻŽাāĻĻেāϰ āϏাāĻāĻāĻে āϏূāĻি āύা āĻāϰāϤে āĻŦেāĻে āύিāϤে āĻĒাāϰি। āĻŦ্āϝāĻŦāĻšাāϰāĻাāϰী-āĻāĻেāύ্āĻ: āĻŦ্āϝাāĻĄেāϰ āĻĒ্āϰāĻৃāϤ āĻŦ্āϝāĻŦāĻšাāϰāĻাāϰী-āĻāĻেāύ্āĻেāϰ āϏাāĻĨে āĻŦ্āϝাāĻĄāĻŦāĻ # āϰিāĻĒ্āϞেāϏ 'āĻŦ্āϝাāĻĄāĻŦāĻ' āĻŦাāϤিāϞ āĻāϰুāύ: /)) āĻāĻŽāϰা āύিāϰ্āĻĻিāώ্āĻ āĻĢাāĻāϞāĻুāϞি āϏূāĻীāĻāϰāĻŖ āĻĨেāĻে āĻāĻāĻি āύিāϰ্āĻĻিāώ্āĻ āĻŦāĻāĻেāĻ āϏীāĻŽাāĻŦāĻĻ্āϧ āĻāϰāϤে āĻĒাāϰি। āĻŦ্āϝāĻŦāĻšাāϰāĻাāϰী-āĻāĻেāύ্āĻ: āĻŦ্āϝাāĻĄāĻŦোāĻ # āĻŦāĻāĻিāϰ āĻĒ্āϰāĻৃāϤ āĻŦ্āϝāĻŦāĻšাāϰāĻাāϰী-āĻāĻেāύ্āĻেāϰ āϏাāĻĨে 'āĻŦ্āϝাāĻĄāĻŦāĻ' āĻĒ্āϰāϤিāϏ্āĻĨাāĻĒāύ āĻāϰুāύ āĻŦাāϤিāϞ āĻāϰুāύ: / āĻাāĻ্āĻ / āĻুāĻāϞেāϰ āĻিāĻু āĻŦāĻেāϰ āύাāĻŽ āĻš'āϞ- āĻুāĻāϞāĻŦāĻ (āϏāĻŽāϏ্āϤ āĻĒāϰিāώেāĻŦা), āĻুāĻāϞāĻŦāĻ-āύিāĻāĻ (āϏংāĻŦাāĻĻেāϰ āĻāύ্āϝ āĻŦāĻ), āĻŦৈāĻĻুāϏ্āĻĒিāĻাāϰ , āĻŦিংāĻŦোāĻ, āĻāϝ়াāύāĻĄেāĻ্āϏāĻŦোāĻ āĻāϤ্āϝাāĻĻি