Generally speaking, it can be used robots.txt Files to block pages that you don't want to be crawled by search engines, but what are these "pages you don't want to be crawled" like? Here are a few simple examples.
(1)多版本URL情况下，非主显URL 的其他版本。比如网站链接伪静态后不希望搜索引擎抓取动态版本了，这时可以使用robots.txt 遮挡掉站内所有动态链接。
(1) In the case of multi version URL, other versions of non main display URL. For example, after the website link is pseudo static, you don't want the search engine to grab the dynamic version, so you can use it robots.txt Block all dynamic links in the station.
(2) 如果网站内有大量的交叉组合查询所生成的页面，肯定有大量页面是没有内容的，对于没有内容的页面可以单独设置个URL 特征，然后使用robots.txt 进行遮挡，以防被搜索引擎认为网站制造垃圾页面。
(2) If there are a large number of pages generated by cross combination query in the website, there must be a large number of pages without content. For pages without content, you can set a URL feature separately, and then use the robots.txt In case the search engine thinks that the website makes spam page.
(3) It is well known that if the website is changed or a large number of pages are suddenly deleted for some reason. The sudden emergence of a large number of dead links on the site is not conducive to the performance of the site in the search engine. Although you can directly submit dead links to Baidu, it's better to block Baidu's capture of dead links directly. In theory, Baidu will not suddenly find that there are too many dead links on the website, or both at the same time. Of course, the webmaster himself to clean up the dead links in the station.
(4) 如果网站有类似UGC 的功能，且为了提高用户提供内容，并没有禁止用户在内容中夹杂链接，此时为了不让这些链接浪费网站权重或牵连网站，可以把这些链接做成站内的跳转链接，然后使用robots.txt进行遮挡。现在有不少已经这样操作了。
(4) If the website has the function similar to UGC, and in order to improve the content provided by users, users are not prohibited from mixing links in the content. At this time, in order not to waste the weight of the website or implicate the website, these links can be made into jump links in the station, and then used robots.txt Occlusion. Now many of them have already done so.
(5) Regular ones that don't want to be indexed by search engines.
This article is provided by e-marketing. Our website is: www.jnexb.com We will provide you with better service with wholehearted enthusiasm. Welcome to visit!