META Tags can be used for excluding content from search engine crawlers. It is also used when you cannot upload a robots.txt file. Its purpose is to keep content out of search engine indexes. They should be added between the HEAD section of your page(s) in question: (no)index determines whether the crawler should index this page. [...]
Search engines collate an image of the internet by searching it, using a special program called a spider. This spider, sometimes called bot, retrieves a map of the internet and all its web pages and files. This map is then used as data to compile results for queries we type into the search engines like [...]