The "Robot.txt" an Extension for Website Security

viewgreen By viewgreen, 28th Jan 2015 | Follow this author | RSS Feed
Posted in Wikinut>Writing>Tips

What is the robots.txt and what will it to do? Ever imagined in your mind to knowing to the robots.txt? This text file is usually underestimated, however its operate is extremely exceptional. Why incredible? Okay, now check up on this text as a result of notional describe a bit regarding this to you all.

The Robot.txt Working

First of all it would be nice if we get acquainted with small files on this one. The robots.txt is a filter for our website, or can we equate with a security guard in front of the home or office. We know the guard is useful to filter out anyone who might come into the office and also gives a direction to that in which the goal and what does it matter, for example, to the receptionist. The security guard also provides for a prohibition or anyone that should not to place undue, ie do not have to go to the barn, the secret chamber, and others.
As security illustrated above, which robots.txt also provide direction and prohibition to guests or visitors. And this is a search engine robot. As for example here is Google. Google has a robot called Google-bot. In stints Google bots are looking for and track every website listed in Google webmaster. By using Google-bot, then Google will collect all the things on the website, such as a recent article, login pages, specific pages, and many others.

The Google-Bot

Well, this is where the role of a small file called robots.txt. This robots.txt will provide direction and ban Google-bot. For example robots.txt allows Google-bot to access the folder, or prohibit Google-bot access the folder. However, if the robots.txt file is not found on a website, of course, Google-bot would explore the website by itself.
And Then, when Google-bot exploring yourself then influence what? This is obviously dangerous. Because the configuration file, upload form, the upload folder, and other confidential files will be tracked by Google-bot and save it to a database of Google.
Actually this is not a problem. However, problems arise when there are people who deliberately seek out the website as a victim of hacking for fun in Google. They will easily find our website configuration file and easily outsmart even replace it.

The Name and Extension

Obviously, the robots.txt file is very useful for a website. Now we are trying to create a robots.txt file although many providers robots.txt generator sites. The initial stage we make this scenario, the search engine robots owned should not access the folder config and admin.
Here is how to create it:
First create a file with the name and extension txt robots Thus, his name robots.txt then place it in the main folder (if the hosting is placed in public_html / www). Then type the following code (shown in bold).

  1. User-agent: *
  2. This means that the principles are written is intended for all robots of all search engines
  3. Dissallow: / config /
  4. This means that prohibits robots of all the search engines to find the config folder
  5. Dissallow: / admin /
  6. This suggests that prohibits robots from all search engines to browse the folder admin

Then save. Be our robots.txt file. easy so, however its go to is remarkable. That is my brief rationalization during this article regarding the robots.txt file and its functions and how to manufacture robots.txt itself. I hope this helps. A presentation video that is explain what is robot.txt by Claude Pelanne below:

Finally, for more information please go and find it here!.

Thank you very much for your stopping here to read this page. :)

"The Robot.txt" - ©2015| viewgreen for the Wikinut - Unique Contents.
Images Credit^>> & Videos Credit^>>

Become a WIKINUT - it's free to JOIN Here!!!
WRITE, SHARE & EARN!!!

Tags

Computer, Computer Security, Computer Tips, Computers Software, Computing, Internet, Internet Security, Robot, Robotics

Meet the author

author avatar viewgreen
A CS engineer who loves to write about TI (Innovations), Socials, Cultures & Heritage (Local Wisdom), Humors and others that I am eager and interested to. Thank you! :D

Share this page

moderator johnnydod moderated this page.
If you have any complaints about this content, please let us know

Comments

author avatar viewgreen
28th Jan 2015 (#)

Dear John, I thank you for the moderated and published this page. :)

Reply to this comment

author avatar Utah Jay
29th Jan 2015 (#)

This is very well written Viewgreen, but way over my head.

Reply to this comment

author avatar viewgreen
29th Jan 2015 (#)

Thank you sir for stopping to see this page. Have a nice day.

Reply to this comment

author avatar Carol Roach
31st Jan 2015 (#)

yes it is well written but well over my head as well. I don't have a brain that grasps technical stuff

Reply to this comment

author avatar viewgreen
31st Jan 2015 (#)

Absolutely, those are the command language of computer program which using to protecting a personal blog and websites. It's seem like a security data but it is not a software like the others in elsewhere. Thank you for stopping by madam.

Reply to this comment

author avatar Retired
8th Feb 2015 (#)

Excellent article!

Reply to this comment

author avatar viewgreen
8th Feb 2015 (#)

Thank you for the appreciation my friend Carmen. :)

Reply to this comment

author avatar Gclick
12th Feb 2015 (#)

Am hate programme too as like the Math. great job my friend. thanks.

Reply to this comment

author avatar Colekdikit
13th Feb 2015 (#)

it's seem like the html programme language. thanks for sharing this.

Reply to this comment

Add a comment
Username
Can't login?
Password