Top > Network Applications > Tools > Larbin

Larbin - Web crawler

Larbin is an HTTP Web crawler that can fetch more than 5 million pages a day on a standard PC (pentium II 300, 128 Mo SDRAM and a 10 Mbit ethernet card, with a good network). Larbin uses standard libraries, plus adns. The program is multithreaded but prefers using select instead of a lot of threads (for efficiency purposes). The advantage of Larbin over wget or ht://dig is that it is much faster (because it opens a lot of connexions at a time) and very easy to customize).

Common uses include: a crawler for a standard search engine, a crawler for a specialized search engine (xml, images, mp3...), and to provide statistics about servers or page contents).

Obtaining

Web pagehttp://larbin.sourceforge.net/index-eng.html
Source tarballhttp://prdownloads.sourceforge.net/larbin/larbin-2.6.3.tar.gz?download
Version 2.6.3 (stable) released on 2002-07-16
Licensed under The GNU General Public License, Version 2.
This is not a GNU package.

Documentation

User guide included
Support contacts

Help List<sebastien@ailleret.com>
Developer List<sebastien@ailleret.com>
Bug List<sebastien@ailleret.com>

Project contacts

Maintainers
Developers

Related information

Interfacesweb
Source languagesC++
Related programsWget, ht://Dig

Entry information

License verified byJanet Casey <jcasey@gnu.org> on 2001-04-04
Entry compiled byJanet Casey <jcasey@gnu.org>

Categories



The copyright licensing notice below applies to this text. The software described in this text has its own copyright notice and license, which can usually be found in the distribution itself.

Copyright © 2000, 2001, 2002, 2003 Free Software Foundation, Inc.

Permission is granted to copy, distribute, and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of this license is included in the file COPYING.DOC.

Please report any problems in this page to bug-directory@gnu.org, or find out how you can help fix them.

The FSF provides this directory as a service to the free software community. Please consider donating to the FSF to help support this project.