POSTS

Data engineering is a skill that’s been steadily rising in demand over the years. Data engineers are responsible for making raw data used for further data-driven work by data scientists, business analysts, and all other end users within an organization.

Data Engineering

What Is Data Engineering? Data engineering is designing and building systems for collecting, storing, and analyzing data at scale. A subdiscipline of software engineering that entirely focuses on the transportation, transformation, and storage of data, data engineering involves designing and building pipelines that convert data into usable formats for end users.

Data engineering forms the foundation of any data-driven company, which is why data engineers are in high demand these days. However, this role demands a lot of data literacy skills.

What are the Responsibilities of a Data Engineer?

1. training machine learning (ML) models,

2. finding and correcting errors in data,

3. performing exploratory data analysis,

4. giving data a standard format,

5. populating fields in an application with outside data,

6. removing duplicate copies of data.

In a nutshell, business intelligence, data science, or any other data-related teams are the end users of data engineering teams.

Data Engineering vs. Data Science

Data engineering makes data reliable and consistent for analysis, while data science uses this reliable data for analytical projects such as machine learning, data exploration, etc. This works the same way humans put their physical needs before social ones. Often, companies have to satisfy a few prerequisites that generally fall under the data engineering umbrella to create a foundation for data scientists to work on.

Therefore, it is correct to say that data scientists rely on data engineers to gather and prepare data for analysis. We could even claim that there can be no data science without data engineering.

Challenges and Solutions in Python

There are some Common Challenges and Solutions in Python. Now that we have a better understanding of data engineering, it’s time to dive deeper into understanding the typical data engineering tasks, fields of action, data pipelines, etc.

Data Transformation

As the name suggests, data transformation refers to converting data from one format to another. Typically, most collected data demands adjustments to align with the system architecture standards.

So, under transformation, a data engineer will perform data normalization and cleaning to make the information more accessible to users. This includes changing or removing incorrect, duplicate, corrupted, or incomplete data in a dataset, casting the same data into a single type, ensuring dates are in the same format, and more. As all these transformations are performed on substantial amounts of data, there also arises a need for parallel computing.

What is Data Transformation?

Data transformation involves changing, eliminating, or fixing a dataset’s incorrect, duplicate, corrupted, or inaccurate data. It can be a pretty tedious task. According to IBM Data Analytics, data scientists spend up to 80% of their time cleaning data.

Pandas can be used here, as it features a data manipulation function that can be used to access data and carry out data cleaning.

Data Transformation: Challenges and Solutions

Before we move on to the challenges and solutions, look at the following table. Here we have some general information about customers, such as name, gender, country, etc.—including other fields like “date” and a column named “param.”

Finding and Filling in Missing Values

We could either use “na” or “null” to see the missing values, but it doesn’t happen immediately. So, adding an “any” function at the end can help.

However, what if we want to know the exact number of missing values for a specific column? Well, we can use “sum” instead of “any,” and we’ll get something like this:

To fill missing values, use the “fillna” function, and for all the missing names, we get a dash, while the missing age is filled with “0.” Alternatively, we can also use drop rows with at least one or all NaN (Not a Number) values with the function “dropna,” and so on.

An even better way to find and fill missing values is to use Pandas.

In Pandas, missing data is represented by None and NaN. The various functions used to detect, remove, and replace null values in Pandas DataFrame include:

1. isnull()

2. notnull()

3. dropna()

4. fillna()

5. replace()

6. interpolate()

Beyond finding and filling in missing values, Pandas can perform functions like counting, sums, averages, joining data, applying to conditions, and more. However, these are all essential functions. So, let’s move to something bigger: Apache Spark.

Apache Spark is a potent and fast analytics engine for big data and machine learning, particularly beneficial in accessing CSV files. Another critical use case of Apache Spark is parallel processing; the tool is designed to process data in a distributed way.

Additional features include lazy evaluation and caching intermediate results in memory. If you are processing or transforming millions or billions of rows at once, Apache Spark, with proper infrastructure, is probably the best tool for the job.

Data Orchestration

The last step is data orchestration—combining and organizing siloed data from various storage locations and making it available for data analysis—as data pipelines comprise several elements: data sources, transformations, and data sink or targets. Hence, data pipelines are built from separate, relatively smaller pieces using different technologies rather than writing them as a large block of code.

A data engineer should connect the separate pieces, schedule the process, and sometimes make decisions based on incoming data, replay part of the pipeline, or apply parallelization to some features. Performing all these tasks demands a more comprehensive and efficient tool than, for instance, Crontab.

What is Data Orchestration?

Imagine a pipeline of tasks that should be run once a day or once a week. These tasks should be run in a specific order. However, they grow and become a network of tasks with dynamic branches called DAGs—Directed Acyclic Graphs. So, orchestration tools are essential to ensure data in DAGs flows in one direction, or in other words, to ensure it has no cycles.

But that’s not all. Other than organizing tasks into DAGs and scheduling them, we also often want to:

1. be able to monitor them easily;

2. dynamically parallel some tasks;

3. wait for a file to appear before it can be processed;

4. use some common or shared variables;

5. in case of failure, replay DAG starting from a particular task.

So, which Python-based tool should be used to address this challenge?

Well, you have plenty of options to choose from. You may use Apache Airflow, Dagster, Luigi, Prefect, Kubeflow, MLflow, Mara, or Kedro. Nevertheless, Airflow is the most popular option, as it has a wide array of features. On the other hand, Luigi (designed by Spotify) and Prefect are much easier to get started with, though they lack some of Airflow’s features.

For instance, Airflow can run multiple DAGs at once and trigger a workflow at specified intervals or times. Also, it’s way simpler to build more complex pipelines, where, for example, we want one task to begin before the previous task has ended. Additionally, Prefect is open-core, while Luigi and Airflow are both open-source.

Data Ingestion

The first step of a data engineering project lifecycle—data ingestion—involves moving data from various sources to a specific database or data warehouse where it can be used for data transformations and analytics.

Storage is also worth mentioning here since the core purpose of data engineering is connecting to various storage types, extracting data from it, and saving it.

One challenge with this is that data comes in various file formats such as comma, tab-separated, JSON, and column-oriented like Parquet or ORC files. So, data engineers often have to deal with structured and unstructured data. Additionally, this data might be present in various SQL and NoSQL databases and data lakes, or they might have to scrape data from websites, streaming services, APIs, etc.

What is Data ingestion and Storage?

SQLAlchemy is the most commonly used tool for connecting databases. An object-relational model library, it supports MySQL, MariaDB, PostgreSQL, Microsoft SQL Server, OracleDB, and SQLite. It consists of two distinct components: the Core and the Object Relational Mapper (ORM).

The Core is a fully featured SQL toolkit allowing users to interact with various DB APIs. In ORM, classes can be mapped to the database schema. The ORM is optional, but it is the main feature that makes SQLAlchemy so popular. Other connectors include:

1. MySQL connector for Python;

2. pyodbc for Microsoft Server;

3. PyMongo for MongoDB;

4. redis-py for Redis.

Every known database or data warehouse—whether it’s Snowflake Elastic or ClickHouse—has its own Python connector or at least recommends using a generic one. There are well-known request libraries for websites and APIs that are often used in conjunction with BeautifulSoup. This helpful utility allows you to get specific elements out of a webpage, such as a list of images.

There is also Scrappy, a tool that allows users to write small amounts of Python codes to create a “spider”—an automated bot that can go through web pages and scrape them. With the help of Scrappy, data engineers can download, clean, and save data from the web without additional hard work.

Getting started with a database is fairly simple. The steps are quite similar for most databases, with subtle variations. For instance, querying a MySQL database with a MySQL connector involves the following steps:

1. Step #1: Create a connection

2. Step #2: Write a query

3. Step #3: Execute it

4. Step #4: Fetch the results

Dealing with Multiple File Formats

Another issue during the data ingestion and storage phase is the various file formats. This is where Pandas come in. A software library written for Python, Pandas, allows us to interact with almost every known file format. Still, for a few formats, there are only reader functions, like ORC, SAS, or SPSS, and for Latex (Lay-tech), there is only the writer.

But the good news is that even if Pandas doesn’t directly support a file format, there is always some workaround solution on how that specific data can be read to Pandas data frames. As far as streaming services are concerned, Kafka is the ideal option. Three predominant Python libraries work with Kafka:

1. Confluent Python Kafka,

2. Kafka-Python,

3. PyKafka.

Confluent Python Kafka is the best option since it was created by Confluent, the same people who developed Kafka. Additionally, PyKafka is no longer supported, though it can be seen in many examples and articles.

For streaming services like Amazon Kinesis, there are dedicated Python libraries:

1. Lay-tech—A document design system where you define the author, titles, subtitles, and so on, and it formats the text for you.

2. Feather—Hails from Apache Arrow and helps efficiently store Pandas DataFrame objects on the disk or in available tables or data frames.

3. Hierarchical Data Format—Uses a file directory-like structure that allows you to organize data within the file in different structured ways.

4. Pickle—Primarily used in serializing and deserializing a Python object structure. “Pickling” is the process where a Python object is converted into a byte stream.