[WEB4LIB] "spiderability" problem

[Thomas Dowling]

Your server is sending back only one HTTP header:

  HTTP/1.0 200 OK

A more typical set of headers would be:

  HTTP/1.1 200 OK
  Date: Thu, 06 May 1999 17:16:57 GMT
  Server: Apache/1.3.1 (Unix)
  Connection: close
  Content-Type: text/html

I can't find a handy list of what headers are required, but I strongly
suspect your server isn't providing everything it's supposed to under HTTP
1.0, and it obviously isn't providing information that the Lycos spider
requires.  I don't know what server software you're running, but I suggest
you run a mainstream server that provides a standard set of response
header, and that won't have this problem.

Thomas Dowling
OhioLINK - Ohio Library and Information Network
tdowling at ohiolink.edu




----- Original Message -----
From: Erwin Wodarczak <ewodar at interchange.ubc.ca>
To: Multiple recipients of list <web4lib at sunsite.Berkeley.EDU>
Sent: Thursday, May 06, 1999 12:35 PM
Subject: [WEB4LIB] "spiderability" problem


> I chair a committee which oversees the Web site for our professional
> organization.  When we tried to register our URL with Lycos, we received
> this response:
>
> >We were unable to spider your page.
> >url: http://aabc.bc.ca/
> >error: HTTP Client: missing header
> >