Results 1 to 5 of 5

Thread: What Is The Limit of A Robot.txt File.?

  1. #1

  2. #2
    Registered
    Join Date
    Feb 2019
    Posts
    288
    there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

  3. #3
    Registered
    Join Date
    Aug 2018
    Posts
    592
    Google has a limit of only being able to process up to 500 KB present in the robots.txt file.

  4. #4

  5. #5
    Registered RH-Calvin's Avatar
    Join Date
    Mar 2017
    Location
    Forum
    Posts
    1,667
    A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
    Cheap VPS Hosting | VPS Starting from $12 PER Year
    Cheap Dedicated Server | Unmetered Bandwidth | Free Setup and IPMI

Similar Threads

  1. How To Use of Robot.txt File In Webmaster..?
    By pihu147741 in forum General Chat
    Replies: 1
    Last Post: 02-06-2019, 10:13 PM
  2. what is character limit
    By Nexevo Technologi in forum Search Engine Optimization
    Replies: 0
    Last Post: 11-23-2015, 08:22 AM
  3. USE of Robot.txt file in SEO
    By Bestseoservice in forum Search Engine Optimization
    Replies: 0
    Last Post: 08-19-2014, 04:00 AM
  4. Looking for web host that does not limit email sending
    By umen in forum Web Hosting & Servers
    Replies: 5
    Last Post: 01-21-2009, 09:41 PM
  5. Gmail outgoing mail limit
    By izwar in forum General Promotion
    Replies: 2
    Last Post: 03-18-2007, 10:41 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •