What is Robot tags
Meta robots tag or Robots.txt is a tag that tells search engines what to follow and what not to follow. It is a piece of code in the <head> section of your webpage. It's a simple code that gives you the power to decide about what pages you want to hide from search engine crawlers and what pages you want them to index and look at.
Robots tag optimization in SEO
Meta robots tags or Robots.txt file are commonly used by SEO marketers. It allows you to tell user-agents (think Googlebot) to crawl specific areas. This meta robots tag tells Google's crawler, Googlebot, to not index the page in the search engines and to not follow any backlinks.
meta name="robots" content="index, follow">
Describe metadata within an HTML document:
<head>
<meta charset="UTF-8">
<meta name="description" content="Free Web tutorials">
<meta name="keywords" content="HTML, CSS, JavaScript">
<meta name="author" content="John Doe">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="refresh" content="30">
</head>
<meta charset="UTF-8">
<meta name="description" content="Free Web tutorials">
<meta name="keywords" content="HTML, CSS, JavaScript">
<meta name="author" content="John Doe">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="refresh" content="30">
</head>
Example : type on URL "<http://(website name)/robots.txt" then result will be show like :
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
For more details about Robots.txt file click on link : Robots.txt file
0 Comments