Blocking AI Crawlers from Your Site | Help Center

Blocking AI Crawlers from Your Site

3 min read

In this article

Using the robots.txt file to block AI crawlers
Using Robots Meta Tags to Block AI Crawlers
FAQs

Control who can access and use your content by blocking AI crawlers from your site. This is particularly useful for site owners who want to prevent their content from being used to train large language models (LLMs) without permission. For example, a site owner with unique research articles might not want their content scraped and used by AI without consent.

Using the robots.txt file to block AI crawlers

Your site's robots.txt file contains instructions to allow or disallow certain requests from search engines. The “allow” command tells crawlers the links that they can follow, while the “disallow” command tells crawlers the links that they can't follow.

To set the robots.txt file to block AI crawlers:

Go to Search Engine Optimization (SEO) in your site's dashboard.
Select Robots.txt Editor under Tools and settings.
Click View File.
Click the field under Your robots.txt file.
Add your robots.txt file info by writing the following directives under This is your current file:
- User-agent: CCBot
  Disallow: /
- User-agent: GPTBot
  Disallow: /
- User-agent: ChatGPT-User
  Disallow: /
- User-agent: BingAI
  Disallow: /
Click Save Changes.
Click Save.

Tip:

This list includes some known AI crawlers, but you can always add more as needed.

Using Robots Meta Tags to Block AI Crawlers

Meta tags are pieces of HTML code that contain page information that web browsers and search engines can read. Configuring your site's robots meta tags allows you to have page-specific control over how the HTML is crawled and indexed in search results.

To use meta tags to block AI crawlers:

Wix Editor

Studio Editor

Go to your editor.
Click Pages & Menu on the left side of the editor.
Click the More Actions icon next to the relevant page.
Click SEO basics.
Click the Advanced SEO tab.
Click Robots meta tag.
Select the nosnippet checkbox to prevent your page's content from being used as a direct input for AI overviews.