# The Format
# ==========
# The format and semantics of the "/robots.txt" file are as follows: 
# The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). Each record contains lines of the form "<field>:<optionalspace><value><optionalspace>". The field name is case insensitive.
# 
# Comments can be included in file using UNIX bourne shell conventions: 
# the '#' character is used to indicate that preceding space (if any) 
# and the remainder of the line up to the line termination is discarded. 
# Lines containing only a comment are discarded completely, and therefore
# do not indicate a record boundary.
# 
# The record starts with one or more User-agent lines, followed 
# by one or more Disallow lines, as detailed below. Unrecognised 
# headers are ignored.
# 
# User-agent 
# The value of this field is the name of the robot the record is 
# describing access policy for.
#  
# If more than one User-agent field is present the record describes 
# an identical access policy for more than one robot. At least one 
# field needs to be present per record.
# 
# The robot should be liberal in interpreting this field. A case 
# insensitive substring match of the name without version information 
# is recommended.
# 
# If the value is '*', the record describes the default access policy 
# for any robot that has not matched any of the other records. It is 
# not allowed to have multiple such records in the "/robots.txt" file.
# 
# Disallow 
# The value of this field specifies a partial URL that is not to be 
# visited. This can be a full path, or a partial path; any URL that 
# starts with this value will not be retrieved. For example, 
# Disallow: /help disallows both /help.html and /help/index.html, 
# whereas Disallow: /help/ would disallow /help/index.html but 
# allow /help.html. 
# 
# Any empty value, indicates that all URLs can be retrieved. At 
# least one Disallow field needs to be present in a record.
# 
# The presence of an empty "/robots.txt" file has no explicit 
# associated semantics, it will be treated as if it was not present,
# i.e. all robots will consider themselves welcome.
# 
# User-agent: WebZIP Disallow: /
# User-agent: WebTrends Disallow: /

User-agent: *
Allow: /