robots.txt file: Unix linebreaks

SUSIE BREIER susie at alcor.concordia.ca
Thu Dec 2 12:41:31 EST 1999


I have been working with our university's network administrator on getting
our library site properly indexed in a newly installed  university-wide
search engine (http://dig). Some of the folders that I disallowed in my
robots.txt file were nevertheless included in the index. Apparently this
is because the line endings are of the Mac OS variety <CR>, when the robot
accepts only the Unix variety <LF>. I used my editor's (BBEdit lite) "Save
as: Linebreaks: Unix" option, but according to the administrator this did
not do the trick and the line endings are now of the <CR>+<LF>. Variety.

So, finally my question, to anyone out there who might have dealt with
this before: how can I ensure that the line endings are (and remain) Unix
<LF>? I have been sent a file with the correct line endings by the
administrator, but am afraid that if I save the file with BBedit again (or
any other Mac or windows wordprocessor for that matter) in order to put
it on our server, the linebreaks will revert back to MacOS.

TIA for any hints out there!

Susie Breier
Assistant Systems Librarian
Concordia University Libraries
1400 de Maisonneuve W, LB 303-3
Montreal, Quebec




More information about the Web4lib mailing list