Abstract—In last decades, the number of web pages on the Internet has been exposed a rapid increase intrinsically, and the information on the Internet has reached a very large size. Search engines have been developed to access this large-scale information efficiently. Web crawlers play a very important role in search engines. In this paper, an efficient multi-threaded web crawler is proposed, and empirically analyzed in terms of crawling speed and coverage.
Index Terms—Coverage, HashMap, web crawler.
Y. Kansu is with the Command Control and Combat Systems, Havelsan, Ankara, Turkey (e-mail: ykansu@ havelsan.com.tr).
B. Mutlu, A. Utku and M. Ali Akcayol are with the Department of Computer Engineering, Gazi University, Ankara, 06570 Turkey (e-mail: begummutlu@gazi.edu.tr, anilutku@gazi.edu.tr, akcayol@gazi.edu.tr).
[PDF]
Cite:Yasin Kansu, Begum Mutlu, Anıl Utku, and M. Ali Akcayol , "An Efficient Multi-Threaded Web Crawler Using HashMaps," Journal of Advances in Computer Networks vol. 5, no. 2, pp. 65-69, 2017.