[WEB4LIB] robots.txt file: Unix linebreaks

[Avi Rappoport]

This is a really good point.  Apparently, several robots depend on 
CR/LF although it's explicitly not required.  So annoying!

In any case, BBEdit has an option in the Save As dialog to save as a 
PC file, which will stick in both CR and LF.  I'm not sure if it 
remembers this every time, which could be a hassle.

Best of luck,

Avi

At 9:43 AM -0800 12/2/1999, SUSIE BREIER wrote:
>I have been working with our university's network administrator on getting
>our library site properly indexed in a newly installed  university-wide
>search engine (http://dig). Some of the folders that I disallowed in my
>robots.txt file were nevertheless included in the index. Apparently this
>is because the line endings are of the Mac OS variety <CR>, when the robot
>accepts only the Unix variety <LF>. I used my editor's (BBEdit lite) "Save
>as: Linebreaks: Unix" option, but according to the administrator this did
>not do the trick and the line endings are now of the <CR>+<LF>. Variety.
>
>So, finally my question, to anyone out there who might have dealt with
>this before: how can I ensure that the line endings are (and remain) Unix
><LF>? I have been sent a file with the correct line endings by the
>administrator, but am afraid that if I save the file with BBedit again (or
>any other Mac or windows wordprocessor for that matter) in order to put
>it on our server, the linebreaks will revert back to MacOS.
>
>TIA for any hints out there!
>
>Susie Breier
>Assistant Systems Librarian
>Concordia University Libraries
>1400 de Maisonneuve W, LB 303-3
>Montreal, Quebec

________________________________________________________________
Avi Rappoport, Search Tools Maven: <mailto:avirr at lanminds.com>
Guide to Site Indexing and Local Search Engines: <http://www.searchtools.com>