The past, present, and future of robots.txt

<p><span style="font-weight: 400;">In this episode of Search Off the Record, Gary, Lizzi, and Martin from the Google Search team are joined by special guest David Price, a Product Counsel for Google. They discuss human initiated crawl requests vs. bot initiated crawl requests, the robots.txt file, and more!</span></p> <p> </p> <p><span style="font-weight: 400;">Episode transcript → </span></p> <p> </p> <p><span style="font-weight: 400;">Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.</span></p>