Robots.txt is a file which is usually placed in the root of any website. It decides whether are permitted or forbidden access to the web site. crawlers For example, the site admin can forbid crawlers to visit a certain folder (and all the files therein contained) or to crawl a specific file, usually to prevent those files being indexed by other search engines. Learn more General knowledge on Wikipedia Robots.txt https://developers.google.com/search/reference/robots_txt Standard specification draft: https://tools.ietf.org/html/draft-rep-wg-topic https://www.robotstxt.org/ View Previous Terms: Block cipher mode of operation Certificate authority Challenge-response authentication Cipher Cipher suite Ciphertext CORS CORS-safelisted request header CORS-safelisted response header Cross-site scripting Cryptanalysis Cryptographic hash function Cryptography CSP CSRF Decryption Digital certificate DTLS (Datagram Transport Layer Security) Encryption Forbidden header name Forbidden response header name Hash HMAC HPKP HSTS HTTPS Key MitM OWASP Preflight request Public-key cryptography Reporting directive Same-origin policy Session Hijacking SQL Injection Symmetric-key cryptography TOFU Transport Layer Security (TLS) Credits Source: https://developer.mozilla.org/en-US/docs/Glossary/Robots.txt Published under license Open CC Attribution ShareAlike 3.0

Google

Mozilla

Glossary of Security Terms: Robots.txt

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Quick Introduction to the Resize Observer API

10 VSCode Extensions To Make Your Life Easier

10 Ways Stand Out as a Java Developer and Land that Dream Job

10 Tips For Junior Developers To Succeed in Code

10 Things Every Beginner Should Know When Learning Javascript And React

A Quick Introduction to the Resize Observer API

10 VSCode Extensions To Make Your Life Easier

10 Ways Stand Out as a Java Developer and Land that Dream Job

10 Tips For Junior Developers To Succeed in Code

10 Things Every Beginner Should Know When Learning Javascript And React

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps