Search Engine Optimization
269 views
Robots.txt
Quick Definition
Text file instructing crawlers which pages to crawl
Full Definition
A text file instructing web crawlers which pages or sections of a site to crawl or skip.
Examples
crawl budget management, blocking pages, search engine directives
Related Terms
technical-seo
xml-sitemap
crawlability
More Search Engine Optimization Terms
Domain Authority
Metric predicting likelihood of ranking in search results
Mobile-First Indexing
Google indexing primarily using mobile version of content
Search Engine Optimization
Optimizing websites to rank higher in search results
Anchor Text
Clickable text in a hyperlink providing context
Content Marketing
Creating valuable content to attract target audiences
Crawl Budget
Number of pages search engines crawl on a site