Glossary of Security Terms: Robots.txt

September 9th 2020

@mozillaMozilla Contributors

Mozilla (stylized as moz://a) is a free software community founded in 1998 by members of Netscape.

Robots.txt is a file which is usually placed in the root of any website. It decides whether crawlers are permitted or forbidden access to the web site.

For example, the site admin can forbid crawlers to visit a certain folder (and all the files therein contained) or to crawl a specific file, usually to prevent those files being indexed by other search engines.

Learn more

General knowledge

• Robots.txt on Wikipedia
• https://developers.google.com/search/reference/robots_txt
• Standard specification draft: https://tools.ietf.org/html/draft-rep-wg-topic
• https://www.robotstxt.org/

View Previous Terms:

• Block cipher mode of operation
• Certificate authority
• Challenge-response authentication
• Cipher
• Cipher suite
• Ciphertext
• CORS
• CORS-safelisted request header
• CORS-safelisted response header
• Cross-site scripting
• Cryptanalysis
• Cryptographic hash function
• Cryptography
• CSP
• CSRF
• Decryption
• Digital certificate
• DTLS (Datagram Transport Layer Security)
• Encryption
• Forbidden header name
• Forbidden response header name
• Hash
• HMAC
• HPKP
• HSTS
• HTTPS
• Key
• MitM
• OWASP
• Preflight request
• Public-key cryptography
• Reporting directive
• Same-origin policy
• Session Hijacking
• SQL Injection
• Symmetric-key cryptography
• TOFU
• Transport Layer Security (TLS)

Credits

• Source: https://developer.mozilla.org/en-US/docs/Glossary/Robots.txt
• Published under Open CC Attribution ShareAlike 3.0 license

Related

Bybit - Win $500 Bonus!

Visit Bybit http://bit.ly/39k3Y05

promoted

JavaScript Concurrency Models: The Event Loop

1 reaction

#javascript

@mozillaMozilla Contributors

5min

01/14/21

Synthetic Data’s Role in the Future of AI

pre-emoji story

#synthetic-data

@sadie-williamsonSadie Williamson

01/21/21