[FX.php List] Security Concerns with FileMaker Website

[Gjermund Gusland Thorsen]

Anyone looked here?
http://diveintomark.org/archives/2003/02/26/how_to_block_spambots_ban_spybots_and_tell_unwanted_robots_to_go_to_hell

ggt667

On 1/25/07, Gjermund Gusland Thorsen <ggt667 at gmail.com> wrote:
> Alternative 3 works in theory but still leaves some bots crawling,
> as the worst bots tells your webserver that it's the most popular browser.
>
> ggt667
>
> On 1/24/07, Jason LEWIS <jasonlewis at weber.edu> wrote:
> > Ed,
> >
> > There are a few things you can do:
> >
> > 1. Sue to get them to stop (I don't like this option because it makes
> > enemies and takes a VERY long time.)
> > 2. use a robots.txt (This option is good for bots that actually honor
> > this.  Google honors the robots.txt.)
> > 3. Detect the type of browser before you give them anything and ship
> > out bad information for unwanted browser types.  (This is only good if
> > the bot owner does not imitate browser variables.)
> >
> > As for my options, I prefer #3 as I can give the bots something that
> > they are looking for, bad information.  In perl, I use
> > $ENV{HTTP_USER_AGENT}, but I am not sure how to call this in php.
> > Anyone else know?  A quick search returned nothing on this.  Could this
> > involve $HTTP_ENV_VARS?
> >
> > Jason
> >
> > >>> elford at cs.bu.edu 01/23/2007 10:18 PM >>>
> > Hello everyone,
> > In the past hour, I've done some analysis of various logs and emails,
> >
> > and I've come to a chilling realization that I've never had before
> > about bots harvesting information from websites -- I knew it
> > happened, but I never knew the scope of the problem until tonight --
> > and this is a low traffic website!
> >
> > So, I have a website which contains a public listing of email
> > addresses and websites from a FileMaker database.  I want to stop
> > unknown bots from crawling the site.  All of the data comes out of
> > FileMaker, nicely formatted as links for the end user's clicking
> > convenience.  I have a solution to fix email addresses from being
> > harvested, but I was wondering if anyone knows of a way to prevent
> > website addresses from being harvested, but still clickable as a
> > hyperlink.
> >
> > I thought maybe a PHP redirect link, like redirect.php?id=16 where
> > redirect puts a user at the website listed in record 16, but once the
> >
> > PHP is all said and done, we're still at the linked website, so that
> > doesn't really prevent anything from being harvested.
> >
> > Is there a way to maybe detect is a link was actually clicked by a
> > person, and not just passed through by an automated bot?  PHP is
> > preferable for such a solution -- JavaScript is too easy to turn
> > off.  Or, is there a way to specify that only bots from places like
> > Google, Live, and Yahoo are allowed to crawl the site?
> >
> > Hopefully my predicament is clear.  I need to solve this ASAP...
> >
> > --Ed
> > ---------------------
> > http://www.edwardford.net
> >
> >
> > _______________________________________________
> > FX.php_List mailing list
> > FX.php_List at mail.iviking.org
> > http://www.iviking.org/mailman/listinfo/fx.php_list
> >
>