Fwd: Re: Fwd: [WEB4LIB] "spiderability" problem

[Nick Arnett]

 From the robot wizards... a bit more detail.

>Date:         Thu, 6 May 1999 12:58:34 -0500
>Reply-To:     "Michael A. Grady" <m-grady at uiuc.edu>
>Sender:       robots at MCCMEDIA.COM
>From:         "Michael A. Grady" <m-grady at UIUC.EDU>
>Subject:      Re: Fwd: [WEB4LIB] "spiderability" problem
>Comments: To: robots at MCCMEDIA.COM
>To:           robots at MCCMEDIA.COM
>
>Well, if you 'telnet aabc.bc.ca 80', and then, once the connection is
>established, you enter 'GET / HTTP/1.0', you'll note several problems
>with what their web server returns:
>
>1) the server should WAIT to return a page until I enter a null line,
>    so that I have the option of entering additional info such as
>    HOST: etc., but it does not -- it returns the page immediately
>
>    However, I'm sure the robot doesn't care too much about that.
>
>2) the ONLY header the server returns is 'HTTP/1.0 200 OK'. It does
>    not return a 'Content-Type:', 'Content-Length:', etc.
>
>    I could very well see a robot complaining about not getting one
>    or more of these headers returned.