Data lake management using topic modeling techniques

Mohamed CHERRADI

doi:10.56294/dm2024282

Original

Published: 2024-04-15

DOI: https://doi.org/10.56294/dm2024282

Data lake management using topic modeling techniques

Abstract

With the rapid rise of information technology, the amount of unstructured data from the data lake is rapidly growing and has become a great challenge in analyzing, organizing and automatically classifying in order to derive the meaningful information for a data-driven business. The scientific document has unlabeled text, so it's difficult to properly link it to a topic model. However, crafting a topic perception for a heterogeneous dataset within the domain of big data lakes presents a complex issue. The manual classification of text documents requires significant financial and human resources. Yet, employing topic modeling techniques could streamline this process, enhancing our understanding of word meanings and potentially reducing the resource burden. This paper presents a comparative study on metadata-based classification of scientific documents dataset, applying the two well-known machine learning-based topic modelling approaches, Latent Dirichlet Analysis (LDA) and Latent Semantic Allocation (LSA). To assess the effectiveness of our proposals, we conducted a thorough examination primarily centred on crucial assessment metrics, including coherence scores, perplexity, and log-likelihood. This evaluation was carried out on a scientific publications corpus, according to information from the title, abstract, keywords, authors, affiliation, and other metadata aspects. Results of these experiments highlight the superior performance of LDA over LSA, evidenced by a remarkable coherence value of (0.884) in contrast to LSA's (0.768).

Keywords:

Data Lake,

Big Data,

Machine Learning,

Topic Modeling,

LDA,

LSA,

How to Cite

CHERRADI M. Data lake management using topic modeling techniques. Data and Metadata [Internet]. 2024 Apr. 15 [cited 2024 Jul. 27];3:282. Available from: https://dm.saludcyt.ar/index.php/dm/article/view/282

Copyright Notice

The article is distributed under the Creative Commons Attribution 4.0 License. Unless otherwise stated, associated published material is distributed under the same licence.

Article metrics

Google scholar: See link

Metrics

Metrics Loading ...

Vol. 3 (2024)

See full issue

Revistas / Journals

Issue

About

Submissions

Data lake management using topic modeling techniques

Abstract

Keywords:

How to Cite

Copyright Notice

Article metrics

Metrics

Vol. 3 (2024)

Disclaimer