This did not work for me, I had the below robots.txt
User-agent: *
Disallow: /version-test/
User-agent: GPTBot
Disallow: /user_profile/
Allow: /articles/
User-agent: OAI-SearchBot
Disallow: /user_profile/
Allow: /articles/
User-agent: OAI-SearchBot/1.0
Disallow: /user_profile/
Allow: /articles/
User-agent: ClaudeBot
Disallow: /user_profile/
Allow: /articles/
User-agent: PerplexityBot
Disallow: /user_profile/
Allow: /articles/
User-agent: CCBot
Disallow: /user_profile/
Allow: /articles/
I’ll try to use indexnow to see if that helps.