Glossary of Security Terms: Robots.txtby@mozilla
114 reads

Glossary of Security Terms: Robots.txt

by Mozilla Contributors1mSeptember 9th, 2020
Read on Terminal Reader
Read this story w/o Javascript
tldt arrow

Too Long; Didn't Read

Mozilla (stylized as moz://a) is a free software community founded in 1998 by members of Netscape. It decides whether crawlers are permitted or forbidden access to the web site. For example, the site admin can forbid crawlers to visit a certain folder (and all the files therein contained) or to crawl a specific file, usually to prevent those files being indexed by other search engines. It is a file which is usually placed in the root of any website and decides if crawlers should be allowed access.

Companies Mentioned

Mention Thumbnail
Mention Thumbnail
featured image - Glossary of Security Terms: Robots.txt
Mozilla Contributors HackerNoon profile picture
Mozilla Contributors

Mozilla Contributors

@mozilla

Mozilla (stylized as moz://a) is a free software community founded in 1998 by members of Netscape.

About @mozilla
LEARN MORE ABOUT @MOZILLA'S
EXPERTISE AND PLACE ON THE INTERNET.

Share Your Thoughts

About Author

Mozilla Contributors HackerNoon profile picture
Mozilla Contributors@mozilla
Mozilla (stylized as moz://a) is a free software community founded in 1998 by members of Netscape.

TOPICS

THIS ARTICLE WAS FEATURED IN...

Permanent on Arweave
Read on Terminal Reader
Read this story in a terminal
 Terminal
Read this story w/o Javascript
Read this story w/o Javascript
 Lite
L O A D I N G
. . . comments & more!