An Optimized Hyperlink Allocation Algorithm for High-Volume Web Mining

International Journal of Computer Science and Technology Vol. 4, No. 9, September 2002 | pp. 65–72

DOI: 10.46882/2002/IJCST/000045

Research Article

Title: An Optimized Hyperlink Allocation Algorithm for High-Volume Web Mining

Names of Authors: Sarah Jenkins¹, Vikram Rao²

Authors’ Affiliations: ¹Department of Computer Science, Columbia University, New York, USA; ²School of Computer Science and Automation, Indian Institute of Science, Bangalore, India.

Abstract:
Distributed web-crawlers and indexing engines struggle with intense network overhead and unbalanced resource distribution when downloading massive structural maps of the internet. The primary operational bottleneck involves assigning interconnected hyperlink schemas across independent physical nodes without generating heavy cross-partition lookups that degrade server processing cycles. This paper outlines an optimized graph-partitioning scheme, Balanced Hyperlink Cut (BHC), specifically engineered to handle power-law web graphs containing highly referenced domain nodes. The algorithm tracks domain boundary relationships and evenly balances physical storage weights while minimizing the inter-node page replication factor. We deployed the BHC framework across a distributed cloud harvesting cluster and evaluated indexing timelines using a raw link graph containing 80 million target URLs and 1.2 billion hyperlink vectors. The experimental findings indicate that BHC cuts inter-node message routing overhead by 43.6% and speeds up structural index creation by a factor of 3.2 relative to classical hash-based URL allocation approaches. Storage capacity variance stayed below 1.8%, validating data allocation stability.

Keywords: Web Crawling, Graph Partitioning, Distributed Systems, Hyperlink Analysis, Storage Scaling, Index Optimization.

Manuscript Timeline: Received: May 10, 2002; Revised: June 15, 2002; Accepted: July 09, 2002; Published: September 24, 2002.

Citation: Jenkins, S., & Rao, V. (2002). An Optimized Hyperlink Allocation Algorithm for High-Volume Web Mining. International Journal of Computer Science and Technology, 4(9), 65–72.

Subscribe to read the full article: https://internationalscholarsjournals.org/subscribe-to-read