What Is GPTBot? OpenAIβs Crawler Explained for Website Owners
What OpenAIβs crawlers do, how they differ, and how to allow or block each one.
If you check your server logs, you may see visits from GPTBot. It is a web crawler run by OpenAI, the company behind ChatGPT. Website owners often ask what it does, whether it is safe and whether they should block it.
OpenAI's crawlers
OpenAI documents several user agents with different purposes:
- GPTBot β collects publicly available content that may be used to improve OpenAI's models.
- OAI-SearchBot β used to surface websites in ChatGPT's search features.
- ChatGPT-User β visits a page when a user's request in ChatGPT needs it.
Because they have separate names, you can make separate decisions. Check OpenAI's own documentation for the current list, as these details can change.
What GPTBot does and does not do
- It crawls publicly accessible pages.
- It respects robots.txt rules addressed to its user agent.
- OpenAI says it filters sources such as paywalled content and pages known to collect personal information.
- It is separate from Googlebot β blocking it does not affect your Google rankings.
How to allow or block it
Use robots.txt rules addressed to the user agent. To block GPTBot site-wide, add a group for User-agent: GPTBot with Disallow: /. To block only a folder, disallow that path instead. You can keep OAI-SearchBot allowed if you still want to appear in ChatGPT search results. The robots.txt generator can create these groups correctly.
Should you block GPTBot?
It depends on your goals. Publishers whose income depends on original content often block training crawlers. Businesses that want maximum visibility in AI tools often allow them. Read should I block AI crawlers for a full decision guide.
Check what your site allows now
Your robots.txt might already block GPTBot through a rule added by a plugin, host or security service. Enter your domain in the AI crawler checker to see which AI bots, including GPTBot and OAI-SearchBot, your robots.txt allows. You can also add an llms.txt file to point AI tools to your best pages.
A worked example
A software company noticed its pages were not being cited in ChatGPT answers. The AI crawler checker showed that a security plugin had added a robots.txt rule blocking every bot except Googlebot and Bingbot. The team rewrote the file to allow OAI-SearchBot and ChatGPT-User, and blocked GPTBot only on its premium documentation folder. It now controls each use separately.
How to see GPTBot in your logs
Ask your host for access logs or open them in your control panel, then search for "GPTBot". Each line shows the page requested, the time and the response code. This helps you see:
- Which sections of your site it visits most.
- Whether it receives errors or challenge pages from a firewall.
- Whether your robots.txt changes took effect.
Any bot can claim to be GPTBot in its user agent. If you see heavy or suspicious traffic, compare the IP addresses with the ranges OpenAI publishes before deciding what to block.
Related guides
More free tools: browse all our free AI search tools, no sign-up needed.
Frequently asked questions
Does blocking GPTBot hurt Google rankings?
No. GPTBot and Googlebot are separate crawlers.
Does blocking GPTBot remove my site from ChatGPT?
Not necessarily. ChatGPT's search uses OAI-SearchBot, which you control separately.
Can GPTBot see pages behind a login?
No. Crawlers only access publicly available pages.
How do I know GPTBot visited my site?
Look for its user agent in your server access logs.