Build scalable and reliable data ecosystems using Data Mesh, Databricks Spark, and Kafka Key Features Develop modern data skills used in emerging technologies Learn pragmatic design methodologies such as Data Mesh and data lakehouses Gain a deeper understanding of data governance Purchase of the print or Kindle book includes a free PDF eBook Book DescriptionModern Data Architectures with Python will teach you how to seamlessly incorporate your machine learning and data science work streams into your open data platforms. You’ll learn how to take your data and create open lakehouses that work with any technology using tried-and-true techniques, including the medallion architecture and Delta Lake. Starting with the fundamentals, this book will help you build pipelines on Databricks, an open data platform, using SQL and Python. You’ll gain an understanding of notebooks and applications written in Python using standard software engineering tools such as git, pre-commit, Jenkins, and Github. Next, you’ll delve into streaming and batch-based data processing using Apache Spark and Confluent Kafka. As you advance, you’ll learn how to deploy your resources using infrastructure as code and how to automate your workflows and code development. Since any data platform's ability to handle and work with AI and ML is a vital component, you’ll also explore the basics of ML and how to work with modern MLOps tooling. Finally, you’ll get hands-on experience with Apache Spark, one of the key data technologies in today’s market. By the end of this book, you’ll have amassed a wealth of practical and theoretical knowledge to build, manage, orchestrate, and architect your data ecosystems.What you will learn Understand data patterns including delta architecture Discover how to increase performance with Spark internals Find out how to design critical data diagrams Explore MLOps with tools such as AutoML and MLflow Get to grips with building data products in a data mesh Discover data governance and build confidence in your data Introduce data visualizations and dashboards into your data practice Who this book is forThis book is for developers, analytics engineers, and managers looking to further develop a data ecosystem within their organization. While they’re not prerequisites, basic knowledge of Python and prior experience with data will help you to read and follow along with the examples.

Discover all the essential design and architectural patterns in one place to help you rapidly build and deploy your modern data platform using AWS services Key Features Learn to build modern data platforms on AWS using data lakes and purpose-built data services Uncover methods of applying security and governance across your data platform built on AWS Find out how to operationalize and optimize your data platform on AWS Purchase of the print or Kindle book includes a free PDF eBook Book DescriptionMany IT leaders and professionals are adept at extracting data from a particular type of database and deriving value from it. However, designing and implementing an enterprise-wide holistic data platform with purpose-built data services, all seamlessly working in tandem with the least amount of manual intervention, still poses a challenge. This book will help you explore end-to-end solutions to common data, analytics, and AI/ML use cases by leveraging AWS services. The chapters systematically take you through all the building blocks of a modern data platform, including data lakes, data warehouses, data ingestion patterns, data consumption patterns, data governance, and AI/ML patterns. Using real-world use cases, each chapter highlights the features and functionalities of numerous AWS services to enable you to create a scalable, flexible, performant, and cost-effective modern data platform. By the end of this book, you’ll be equipped with all the necessary architectural patterns and be able to apply this knowledge to efficiently build a modern data platform for your organization using AWS services.What you will learn Familiarize yourself with the building blocks of modern data architecture on AWS Discover how to create an end-to-end data platform on AWS Design data architectures for your own use cases using AWS services Ingest data from disparate sources into target data stores on AWS Build data pipelines, data sharing mechanisms, and data consumption patterns using AWS services Find out how to implement data governance using AWS services Who this book is for This book is for data architects, data engineers, and professionals creating data platforms. The book's use case–driven approach helps you conceptualize possible solutions to specific use cases, while also providing you with design patterns to build data platforms for any organization. It's beneficial for technical leaders and decision makers to understand their organization's data architecture and how each platform component serves business needs. A basic understanding of data & analytics architectures and systems is desirable along with beginner’s level understanding of AWS Cloud.

Key Features Discover the key drivers of successful Azure architecture Practical guidance Focus on scalability and performance Expert authorship Book Description This book presents a guide to design and implement scalable, secure, and efficient data solutions in the Azure cloud environment. It provides Data Architects, developers, and IT professionals who are responsible for designing and implementing data solutions in the Azure cloud environment with the knowledge and tools needed to design and implement data solutions using the latest Azure data services. It covers a wide range of topics, including data storage, data processing, data analysis, and data integration. In this book, you will learn how to select the appropriate Azure data services, design a data processing pipeline, implement real-time data processing, and implement advanced analytics using Azure Databricks and Azure Synapse Analytics. You will also learn how to implement data security and compliance, including data encryption, access control, and auditing. Whether you are building a new data architecture from scratch or migrating an existing on premises solution to Azure, the Azure Data Architecture Guidelines are an essential resource for any organization looking to harness the power of data in the cloud. With these guidelines, you will gain a deep understanding of the principles and best practices of Azure data architecture and be equipped to build data solutions that are highly scalable, secure, and cost effective. What You Need to Use this Book? To use this book, it is recommended that readers have a basic understanding of data architecture concepts and data management principles. Some familiarity with cloud computing and Azure services is also helpful. The book is designed for data architects, data engineers, data analysts, and anyone involved in designing, implementing, and managing data solutions on the Azure cloud platform. It is also suitable for students and professionals who want to learn about Azure data architecture and its best practices.

Dr.Sudhakar.K, Associate Professor, Department of Artificial Intelligence & Data Science, NITTE Meenakshi Institute of Technology, Bangalore, Karnataka, India. Mrs.Sangeetha Suresh Harikantra, Assistant Professor, Department of Artificial Intelligence & Data Science, NITTE Meenakshi Institute of Technology, Bangalore, Karnataka, India. Mrs.Anu.D, Assistant Professor, Department of Artificial Intelligence & Data Science, NITTE Meenakshi Institute of Technology, Bangalore, Karnataka, India. Mrs.Rajeshwari Patil, Assistant Professor, Department of Artificial Intelligence & Data Science, NITTE Meenakshi Institute of Technology, Bangalore, Karnataka, India.

Data miner’s survival kit for explainable, effective, and efficient algorithms enabling responsible decision-making KEY FEATURES ● Accessible, and case-based exploration of the most effective data mining techniques in Python. ● An indispensable guide for utilizing AI potential responsibly. ● Actionable insights on modeling techniques, deployment technologies, business needs, and the art of data science, for risk mitigation and better business outcomes. DESCRIPTION "Modern Data Mining with Python" is a guidebook for responsibly implementing data mining techniques that involve collecting, storing, and analyzing large amounts of structured and unstructured data to extract useful insights and patterns. Enter into the world of data mining and machine learning. Use insights from various data sources, from social media to credit card transactions. Master statistical tools, explore data trends, and patterns. Understand decision trees and artificial neural networks (ANNs). Manage high-dimensional data with dimensionality reduction. Explore binary classification with logistic regression. Spot concealed patterns with unsupervised learning. Analyze text with recurrent neural networks (RNNs) and visuals with convolutional neural networks (CNNs). Ensure model compliance with regulatory standards. After reading this book, readers will be equipped with the skills and knowledge necessary to use Python for data mining and analysis in an industry set-up. They will be able to analyze and implement algorithms on large structured and unstructured datasets. WHAT YOU WILL LEARN ● Explore the data mining spectrum ranging from data exploration and statistics. ● Gain hands-on experience applying modern algorithms to real-world problems in the financial industry. ● Develop an understanding of various risks associated with model usage in regulated industries. ● Gain knowledge about best practices and regulatory guidelines to mitigate model usage-related risk in key banking areas. ● Develop and deploy risk-mitigated algorithms on self-serve ModelOps platforms. WHO THIS BOOK IS FOR This book is for a wide range of early career professionals and students interested in data mining or data science with a financial services industry focus. Senior industry professionals, and educators, trying to implement data mining algorithms can benefit as well. TABLE OF CONTENTS 1. Understanding Data Mining in a Nutshell 2. Basic Statistics and Exploratory Data Analysis 3. Digging into Linear Regression 4. Exploring Logistic Regression 5. Decision Trees with Bagging and Boosting 6. Support Vector Machines and K-Nearest Neighbors 7. Putting Dimensionality Reduction into Action 8. Beginning with Unsupervised Models 9. Structured Data Classification using Artificial Neural Networks 10. Language Modeling with Recurrent Neural Networks 11. Image Processing with Convolutional Neural Networks 12. Understanding Model Risk Management for Data Mining Models 13. Adopting ModelOps to Manage Model Risk

Provides an up-to-date analysis of big data and multi-agent systems The term Big Data refers to the cases, where data sets are too large or too complex for traditional data-processing software. With the spread of new concepts such as Edge Computing or the Internet of Things, production, processing and consumption of this data becomes more and more distributed. As a result, applications increasingly require multiple agents that can work together. A multi-agent system (MAS) is a self-organized computer system that comprises multiple intelligent agents interacting to solve problems that are beyond the capacities of individual agents. Modern Big Data Architectures examines modern concepts and architecture for Big Data processing and analytics. This unique, up-to-date volume provides joint analysis of big data and multi-agent systems, with emphasis on distributed, intelligent processing of very large data sets. Each chapter contains practical examples and detailed solutions suitable for a wide variety of applications. The author, an internationally-recognized expert in Big Data and distributed Artificial Intelligence, demonstrates how base concepts such as agent, actor, and micro-service have reached a point of convergence—enabling next generation systems to be built by incorporating the best aspects of the field. This book: Illustrates how data sets are produced and how they can be utilized in various areas of industry and science Explains how to apply common computational models and state-of-the-art architectures to process Big Data tasks Discusses current and emerging Big Data applications of Artificial Intelligence Modern Big Data Architectures: A Multi-Agent Systems Perspective is a timely and important resource for data science professionals and students involved in Big Data analytics, and machine and artificial learning.

Data fabric, data lakehouse, and data mesh have recently appeared as viable alternatives to the modern data warehouse. These new architectures have solid benefits, but they're also surrounded by a lot of hyperbole and confusion. This practical book provides a guided tour of these architectures to help data professionals understand the pros and cons of each. James Serra, big data and data warehousing solution architect at Microsoft, examines common data architecture concepts, including how data warehouses have had to evolve to work with data lake features. You'll learn what data lakehouses can help you achieve, as well as how to distinguish data mesh hype from reality. Best of all, you'll be able to determine the most appropriate data architecture for your needs. With this book, you'll: Gain a working understanding of several data architectures Learn the strengths and weaknesses of each approach Distinguish data architecture theory from reality Pick the best architecture for your use case Understand the differences between data warehouses and data lakes Learn common data architecture concepts to help you build better solutions Explore the historical evolution and characteristics of data architectures Learn essentials of running an architecture design session, team organization, and project success factors Free from product discussions, this book will serve as a timeless resource for years to come.

Learn the concepts and exercises needed to confidently prepare for the Databricks Associate Developer for Apache Spark 3.0 exam and validate your Spark skills with an industry-recognized credential Key Features Understand the fundamentals of Apache Spark to design robust and fast Spark applications Explore various data manipulation components for each phase of your data engineering project Prepare for the certification exam with sample questions and mock exams Purchase of the print or Kindle book includes a free PDF eBook Book DescriptionSpark has become a de facto standard for big data processing. Migrating data processing to Spark saves resources, streamlines your business focus, and modernizes workloads, creating new business opportunities through Spark’s advanced capabilities. Written by a senior solutions architect at Databricks, with experience in leading data science and data engineering teams in Fortune 500s as well as startups, this book is your exhaustive guide to achieving the Databricks Certified Associate Developer for Apache Spark certification on your first attempt. You’ll explore the core components of Apache Spark, its architecture, and its optimization, while familiarizing yourself with the Spark DataFrame API and its components needed for data manipulation. You’ll also find out what Spark streaming is and why it’s important for modern data stacks, before learning about machine learning in Spark and its different use cases. What’s more, you’ll discover sample questions at the end of each section along with two mock exams to help you prepare for the certification exam. By the end of this book, you’ll know what to expect in the exam and gain enough understanding of Spark and its tools to pass the exam. You’ll also be able to apply this knowledge in a real-world setting and take your skillset to the next level.What you will learn Create and manipulate SQL queries in Apache Spark Build complex Spark functions using Spark's user-defined functions (UDFs) Architect big data apps with Spark fundamentals for optimal design Apply techniques to manipulate and optimize big data applications Develop real-time or near-real-time applications using Spark Streaming Work with Apache Spark for machine learning applications Who this book is for This book is for data professionals such as data engineers, data analysts, BI developers, and data scientists looking for a comprehensive resource to achieve Databricks Certified Associate Developer certification, as well as for individuals who want to venture into the world of big data and data engineering. Although working knowledge of Python is required, no prior knowledge of Spark is necessary. Additionally, experience with Pyspark will be beneficial.

Success in data science depends on the flexible and appropriate use of tools. That includes Python and R, two of the foundational programming languages in the field. This book guides data scientists from the Python and R communities along the path to becoming bilingual. By recognizing the strengths of both languages, you'll discover new ways to accomplish data science tasks and expand your skill set. Authors Rick Scavetta and Boyan Angelov explain the parallel structures of these languages and highlight where each one excels, whether it's their linguistic features or the powers of their open source ecosystems. You'll learn how to use Python and R together in real-world settings and broaden your job opportunities as a bilingual data scientist. Learn Python and R from the perspective of your current language Understand the strengths and weaknesses of each language Identify use cases where one language is better suited than the other Understand the modern open source ecosystem available for both, including packages, frameworks, and workflows Learn how to integrate R and Python in a single workflow Follow a case study that demonstrates ways to use these languages together

Unlock the potential of data with "Streamlining ETL: A Practical Guide to Building Pipelines with Python and SQL," the definitive resource for creating high-performance ETL pipelines. This essential guide is meticulously designed for data professionals seeking to harness the data-intensive capabilities of Python and SQL. From establishing a development environment and extracting raw data to optimizing and securing data processes, this book offers comprehensive coverage of every aspect of ETL pipeline development. Whether you're a data engineer, IT professional, or a scholar in data science, this book provides step-by-step instructions, practical examples, and expert insights necessary for mastering the creation and management of robust ETL pipelines. By the end of this guide, you will possess the skills to transform disparate data into meaningful insights, ensuring your data processes are efficient, scalable, and secure. Dive into advanced topics with ease and explore best practices that will make your data workflows more productive and error-resistant. With this book, elevate your organization's data strategy and foster a data-driven culture that thrives on precision and performance. Embrace the journey to becoming an adept data professional with a solid foundation in ETL processes, equipped to handle the challenges of today's data demands.