Citation:

Prem Tamanam, 2023. "AI-Powered Data Lakes and Warehouses: The Synergy that is Changing Data Science Forever", ESP Journal of Engineering & Technology Advancements, 3(4): 91-105.

Abstract:

With data exponential in nature, data management strategies have evolved in modern enterprises to embrace the exponential growth of data and AI acts as a transformative enabler for this. This paper explores how AI-driven data lakes and data warehouses converge to improve data science practices. Compared to data lakes, which can give scalable and inexpensive storage to unstructured and semi-structured data, the use case of data warehouses is to give a stable querying ability and performance for structured data. AI-driven frameworks integrate these paradigms and help in intelligent data discovery, automated transformations, and faster analytics. Experimental results show that these systems overcome traditional bottlenecks, optimize ETL processes, and enable real-time decision-making. However, governance, data quality, and ethical AI usage have persisted. This study points to the potential of taking action on such synergy to gain actionable insights towards enterprise data strategy moving ahead.

References:

[1] Coleman, S. S., & Watson, R. W. (1993). The emerging paradigm shift in storage system architectures. Proceedings of the IEEE, 81(4), 607-620.

[2] Amarasinghe, S. C., & Fernando, N. (2023). Evaluating Scalability and Performance in Data Lake Architectures: Opportunities and Challenges. International Journal of Applied Machine Learning and Computational Intelligence, 13(5), 1-15.

[3] Manchana, R. Building a Modern Data Foundation in the Cloud: Data Lakes and Data Lakehouses as Key Enablers. J Artif Intell Mach Learn & Data Sci 2023, 1(1), 1098-1108.

[4] Althati, C., Tomar, M., & Shanmugam, L. (2024). Enhancing Data Integration and Management: The Role of AI and Machine Learning in Modern Data Platforms. Journal of Artificial Intelligence General Science (JAIGS) ISSN: 3006-4023, 2(1), 220-232.

[5] Gad-Elrab, A. A. (2021). Modern business intelligence: Big data analytics and artificial intelligence for creating the data-driven value. E-Business-Higher Education and Intelligence Applications, 135.

[6] Bai, M., & Tahir, F. (2023). Data lakes and data warehouses: Managing big data architectures. Tech. Rep., EasyChair.

[7] Nambiar, A., & Mundra, D. (2022). An overview of data warehouse and data lake in modern enterprise data management. Big data and cognitive computing, 6(4), 132.

[8] Vemulapalli, G. (2023). Optimizing Analytics: Integrating Data Warehouses and Lakes for Accelerated Workflows. International Scientific Journal for Research, 5(5), 1-27.

[9] Khosravi, H., Sadiq, S., & Amer-Yahia, S. (2023). Data management of AI-powered education technologies: Challenges and opportunities. Learning Letters.

[10] Pasholikov, M., & Dudakov, G. (2020). Technological innovations: application, prospects, development trends. In E3S Web of Conferences (Vol. 164, p. 10003). EDP Sciences.

[11] Bianchini, D., De Antonellis, V., & Garda, M. (2024). A semantics-enabled approach for personalized Data Lake exploration. Knowledge and Information Systems, 66(2), 1469-1502.

[12] Sauter, V. L. (2014). Decision support systems for business intelligence. John Wiley & Sons.

[13] Turban, E. (2011). Decision support and business intelligence systems. Pearson Education India.

[14] Yang, S. (2017). IoT stream processing and analytics in the fog. IEEE Communications Magazine, 55(8), 21-27.

[15] Kaur, J. (2023). Streaming Data Analytics: Challenges and Opportunities. International Journal of Applied Engineering & Technology, 5(S4), 10-16.

[16] From BI to AI-Why use a Data Lakehouse instead of a Data Lake for AI?, online. https://www.linkedin.com/pulse/from-bi-ai-why-use-data-lakehouse-instead-lake-ai-bahram-khanlarov-kds2e

[17] Akbar, A., Khan, A., Carrez, F., & Moessner, K. (2017). Predictive analytics for complex IoT data streams. IEEE Internet of Things Journal, 4(5), 1571-1582.

[18] Patel, J. M., & Patel, J. M. (2020). Natural Language Processing (NLP) and Text Analytics. Getting Structured Data from the Internet: Running Web Crawlers/Scrapers on a Big Data Production Scale, 135-223.

[19] Doherty, N. F., & Doig, G. (2011). The role of enhanced information accessibility in realizing the benefits from data warehousing investments. Journal of Organisational Transformation & Social Change, 8(2), 163-182.

[20] Anderson, R. J. (1996). Reducing and controlling overhead costs. Drug Information Journal, 30(1), 89-96.

[21] Iyer, L. S., Gupta, B., & Johri, N. (2005). Performance, scalability and reliability issues in web applications. Industrial Management & Data Systems, 105(5), 561-576.

[22] Bertsimas, D., & Kallus, N. (2020). From predictive to prescriptive analytics. Management Science, 66(3), 1025-1044.

[23] Deka, G. C. (2014). Big data predictive and prescriptive analytics. In Handbook of research on cloud infrastructures for Big Data analytics (pp. 370-391). IGI Global.

Keywords:

Data Lakes, AI-Powered, Data Warehouses, Data Science, Data Management, Data Governance.