A Scalable Multi-Node Text Tokenization Pipeline for Fast Index Generation

International Journal of Computer Science and Technology Vol. 1, No. 5, May 1999 | pp. 35–42

DOI: 10.46882/1999/IJCST/000005

Research Article

Title: A Scalable Multi-Node Text Tokenization Pipeline for Fast Index Generation

Names of Authors: Robert Evans¹, Yuki Tanaka²

Authors’ Affiliations: ¹Department of Software Engineering, University of Toronto, Canada; ²Graduate School of Information Science, Tokyo Institute of Technology, Japan.

Abstract:
The expansion of unstructured information across modern repository platforms necessitates the implementation of rapid index mechanisms capable of transforming raw documents into structured elements. Executing lexical analysis over mass volumes requires resilient data pipelines and low-latency storage strategies. This paper introduces an integrated, distributed document indexing architecture utilizing a synchronized cluster layout and an optimized Naive Bayes text parsing system. The processing pipeline scales horizontally to ingest up to 15,000 document records per second. We verified the operational stability of the model using an experimental database consisting of 5 million raw unstructured text records. The architecture recorded a processing accuracy of 89.4% while maintaining sub-second text tokenization latencies from ingestion to physical storage. We implemented a specialized morphology filter that accommodates short forms, jargon, and syntax variations, lifting the system F1-score by 7.3%. Comparative benchmarks verify that this multi-node framework outperforms conventional single-instance disk database indices by a factor of 12, offering scalable infrastructure for massive document architectures.

Keywords: Document Parsing, Information Retrieval, Distributed Databases, Index Optimization, Text Ingestion, Scalable Frameworks.

Manuscript Timeline: Received: February 18, 1999; Revised: March 29, 1999; Accepted: April 15, 1999; Published: May 25, 1999.

Citation: Evans, R., & Tanaka, Y. (1999). A Scalable Multi-Node Text Tokenization Pipeline for Fast Index Generation. International Journal of Computer Science and Technology, 1(5), 35–42.

Subscribe to read the full article: https://internationalscholarsjournals.org/subscribe-to-read