You may not want certain pages of your site crawled because they might not be useful to users if found in a search engine's search results. If you do want to prevent search engines from crawling your pages, Google Search Console has a friendly robots.txt generator to help you create this file. Note that if your site uses subdomains and you wish to have certain pages not crawled on a particular subdomain, you'll have to create a separate robots.txt file for that subdomain. For more information on robots.txt, we suggest this Webmaster Help Center guide on using robots.txt files13.
Search engine optimization is a key part of online marketing because search is one of the primary ways that users navigate the web. In 2014, over 2.5 trillion searches were conducted worldwide across search engines such as Google, Bing, Yahoo, Baidu, and Yandex. For most websites, traffic that comes from search engines (known as "natural" or "organic" traffic) accounts for a large portion of their total traffic.