Help Center
  1. Help Center
  2. Promoting your site
  3. SEO (get found on Google)
  4. SEO Settings
  5. Blocking AI Crawlers from Your Site

Blocking AI Crawlers from Your Site

3 min read
In this article
  • Blocking AI Crawlers Using robots.txt
  • Using Robots Meta Tags to Block AI Crawlers
  • FAQs
Blocking AI crawlers from your site helps control who can access and use your content. This is particularly useful for site owners who want to prevent their content from being used to train large language models (LLMs) without permission. For example, a site owner with unique research articles might not want their content scraped and used by AI without consent.

Blocking AI Crawlers Using robots.txt

The robots.txt file contains instructions to allow or disallow certain requests from search engines. The “allow” command tells crawlers the links that they can follow, while the “disallow” command tells crawlers the links that they can't follow. 

To block AI crawlers using robots.txt:

  1. Go to your SEO dashboard.
  2. Select Go to Robots.txt Editor under Tools and settings.
  3. Click View File. 
  4. Click on Edit robots.txt.
  5. Add your robots.txt file info by writing the following directives under This is your current file:
    • User-agent: CCBot
      Disallow: /
    • User-agent: GPTBot
      Disallow: /
    • User-agent: ChatGPT-User
      Disallow: /
    • User-agent: BingAI
      Disallow: /
  6. Click Save Changes. 
  7. Click Save. 
Editing your robots.txt file
Tip:
This list includes some known AI crawlers, but you can always add more as needed.

Using Robots Meta Tags to Block AI Crawlers

Meta tags are pieces of HTML code that contain page information that web browsers and search engines can read. Configuring your site's robots meta tags allows you to have page-specific control over how HTML is crawled and indexed in search results.

To block AI crawlers using robots meta tags:

Wix Editor
Studio Editor
  1. Go to your editor.
  2. Click Pages & Menu  on the left side of the editor.
  3. Click the More Actions icon  next to the relevant page.
  4. Click SEO basics.
  5. Click the Advanced SEO tab.
  6. Click Robots meta tag.
  7. Select the nosnippet option to prevent your page's content from being used as a direct input for AI overviews.
Robots meta tag options shown in the Advanced SEO Settings tab in the editor, with nosnippet selected.
Learn more about robots meta tags from Google.

FAQs

Click a question below to learn more about blocking AI crawlers from your site.

Did this help?

|
Hire a ProfessionalGet matched with a Wix Partner who can help you reach your site goals.
Start Now
In this article
  • Blocking AI Crawlers Using robots.txt
  • Using Robots Meta Tags to Block AI Crawlers
  • FAQs

Did this help?

|