Lab Work 2#

Variables and Data Types#

In Python, you can declare variables by simply assigning values to them. There is no need for explicit variable type declaration:

1
name = "Alice"       # String
2
age = 30             # Integer
3
height = 5.7         # Float
4
is_student = False   # Boolean

Basic Data Structures#

Python has several data structures that are regularly used in data analysis:

1. Lists

   • Ordered, mutable collections of elements.
   • Example:

     1
     fruits = ["apple", "banana", "cherry"]
     

   • You can access elements by index:

     1
     print(fruits[0])  # "apple"
     

   • Elements can be appended:

     1
     fruits.append("durian")
     

2. Tuples

   • Ordered, immutable collections of elements.
   • Example:

     1
     coordinates = (10, 20)
     

3. Dictionaries

   • Key-value pairs, highly efficient for lookups.
   • Example:

     1
     person = {
     2
         "name": "Alice",
     3
         "age": 30
     4
     }
     

   • Access values by key:

     1
     print(person["name"])  # "Alice"
     

Control Flow#

• If-Else Statements

  1
  if age > 18:
  2
      print("Adult")
  3
  else:
  4
      print("Minor")
  

• For Loops

  1
  for fruit in fruits:
  2
      print(fruit)
  

• While Loops

  1
  count = 0
  2
  while count < 5:
  3
      print(count)
  4
      count += 1
  

Functions#

Functions group reusable code blocks. Define them with the def keyword:

1
def greet(name):
2
    return f"Hello, {name}!"
3

4
message = greet("Alice")
5
print(message)

With these basics, you can comfortably read and write Python scripts. Next, we delve deeper into the data structures vital for data analysis and introduce key libraries used throughout the data science ecosystem.

pandas DataFrame#

The cornerstone of Python-based data analysis is the pandas library, which introduces the DataFrame, a two-dimensional labeled data structure typically employed for tabular data. Think of a DataFrame as an in-memory representation of a CSV or a spreadsheet.

Here’s how to import pandas and create a simple DataFrame:

1
import pandas as pd
2

3
data = {
4
    "Name": ["Alice", "Bob", "Charlie"],
5
    "Age": [25, 30, 35],
6
    "City": ["New York", "Paris", "London"]
7
}
8

9
df = pd.DataFrame(data)
10
print(df)

Output:

Common pandas operations include:

• Indexing rows and columns.
• Filtering rows based on conditions.
• Grouping & aggregating data.
• Merging or concatenating multiple DataFrames.

NumPy Arrays#

While pandas is specialized for tabular data, NumPy is the fundamental package for scientific computing in Python. NumPy arrays are powerful n-dimensional arrays that offer:

• Vectorized arithmetic operations (much faster than pure Python loops).
• Broadcasting capabilities (operations on arrays of different shapes).
• A wide array of mathematical functions.

Example of using NumPy arrays:

1
import numpy as np
2

3
arr = np.array([1, 2, 3, 4, 5])
4
print(arr * 2)  # [ 2  4  6  8 10 ]

Table: Comparison of Core Data Structures#

Understanding these data structures is crucial for analyzing, reshaping, summarizing, and modeling your data. Next, we’ll explore how to load, inspect, and glean insights from real-world datasets using pandas.

Loading Data#

You can gather data from diverse sources such as CSV files, spreadsheets, SQL databases, or JSON endpoints. The most common function to start with is pd.read_csv:

1
import pandas as pd
2

3
df = pd.read_csv("your_data.csv")
4
print(df.head())

Inspecting Data#

1. Head and Tail

   • df.head(n) shows the first n rows.
   • df.tail(n) shows the last n rows.

2. Shape and Columns

   • df.shape returns (number_of_rows, number_of_columns).
   • df.columns returns the column names.

3. Info and Describe

   • df.info() displays details about column data types and missing values.
   • df.describe() provides summary statistics like mean, standard deviation, min, max, and quartiles for numeric columns.

Data Filtering and Indexing#

1. Boolean Indexing

   1
   adults = df[df["Age"] > 18]
   

2. Loc and iLoc

   • df.loc[row_label, column_label] filters using labels.
   • df.iloc[row_position, column_position] filters using indices.

3. Multiple Conditions

   1
   city_paris_aged_30 = df[(df["City"] == "Paris") & (df["Age"] == 30)]
   

Grouping and Aggregation#

Group data by specific features to summarize or detect patterns:

1
grouped_data = df.groupby("City")["Age"].mean()
2
print(grouped_data)

This snippet calculates the average age per city. You can use various aggregation functions such as sum, count, min, max, and std.

Matplotlib#

Matplotlib is the grandfather of Python plotting, providing versatile and customizable visualizations. Here’s an example of creating a basic line plot and customizing it:

1
import matplotlib.pyplot as plt
2

3
x = [1, 2, 3, 4, 5]
4
y = [10, 12, 8, 15, 7]
5

6
plt.figure(figsize=(8, 4))
7
plt.plot(x, y, marker='o', linestyle='-', color='b', label='Line Example')
8
plt.title("Basic Line Plot")
9
plt.xlabel("X-axis")
10
plt.ylabel("Y-axis")
11
plt.legend()
12
plt.show()

pandas Built-in Visualizations#

pandas has built-in plotting functions that wrap around matplotlib, making it simpler to create bar plots, histograms, box plots, etc.:

1
df["Age"].hist(bins=10, figsize=(6,4))
2
plt.title("Histogram of Ages")
3
plt.xlabel("Age")
4
plt.ylabel("Frequency")
5
plt.show()

Seaborn#

Seaborn is built on top of matplotlib and offers more sophisticated plots with fewer lines of code. It excels at statistical visualizations. For instance, to create a scatter plot colored by a categorical variable:

1
import seaborn as sns
2

3
sns.scatterplot(data=df, x="Age", y="Height", hue="Gender")
4
plt.show()

Plotly and Other Interactive Libraries#

For interactive dashboards and web applications, libraries like Plotly, Bokeh, or Altair are popular. They allow you to hover over points, zoom in on graphs, or create dynamic filters.

Handling Missing Values#

1. Detection

   1
   missing_counts = df.isnull().sum()
   2
   print(missing_counts)
   

2. Removal

   • Remove rows or columns with missing values:

     1
     df_clean = df.dropna()
     

   • This is acceptable only if dropping does not lead to significant data loss.

3. Imputation

   • Replace missing values with mean, median, or mode:

     1
     df["Age"].fillna(df["Age"].mean(), inplace=True)
     

   • Forward-fill or backward-fill for time-series data:

     1
     df["Price"].fillna(method="ffill", inplace=True)
     

Dealing with Duplicates#

1
duplicates_count = df.duplicated().sum()
2
df_unique = df.drop_duplicates()

Data Type Conversion#

Sometimes numeric data might be stored as string, or date fields might be read in as objects. Use these conversions:

1
df["Date"] = pd.to_datetime(df["Date"])
2
df["Gender"] = df["Gender"].astype("category")

String Operations#

pandas offers numerous functions to manipulate string columns, such as str.lower(), str.upper(), str.contains(), etc.:

1
df["Name"] = df["Name"].str.title()
2
contains_substring = df["Address"].str.contains("Street")

Outlier Detection#

Outliers can skew your analysis. Methods for detecting outliers:

• Box plots (sns.boxplot) to visualize the distribution.
• Z-score or standard deviation threshold.
• Interquartile Range (IQR) approach.

Common ML Terminology#

• Features (X): Independent variables, used as inputs to the model.
• Target (y): Dependent variable, which the model aims to predict.
• Training Set: Subset of your data used to train the model.
• Test Set: Held-out portion of data used to evaluate the model’s performance on unseen data.
• Overfitting: When the model memorizes training data rather than generalizing.
• Underfitting: When the model is too simple and fails to capture the data’s trends.

Simple Example: Linear Regression#

Let’s go through a basic example of predicting house prices (target) based on variables such as square footage, number of bedrooms, etc. We’ll assume you have a dataset loaded into a DataFrame df:

1
import pandas as pd
2
from sklearn.model_selection import train_test_split
3
from sklearn.linear_model import LinearRegression
4

5
# Suppose df has "price" as target and "sqft", "bedrooms" as features
6
X = df[["sqft", "bedrooms"]]
7
y = df["price"]
8

9
X_train, X_test, y_train, y_test = train_test_split(X, y,
10
                                                    test_size=0.2,
11
                                                    random_state=42)
12

13
model = LinearRegression()
14
model.fit(X_train, y_train)
15

16
y_pred = model.predict(X_test)
17
print(y_pred[:5])  # First 5 predictions

Once predictions are generated, evaluate the model:

1
from sklearn.metrics import mean_squared_error, r2_score
2

3
mse = mean_squared_error(y_test, y_pred)
4
r2 = r2_score(y_test, y_pred)
5

6
print(f"MSE: {mse}")
7
print(f"R^2: {r2}")

Classification Example: Logistic Regression#

For classification tasks (e.g., predicting if a customer will buy a product or not):

1
from sklearn.linear_model import LogisticRegression
2

3
features = df[["age", "income"]]
4
target = df["purchase_made"]  # 1 if made a purchase, 0 otherwise
5

6
X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.2)
7
clf = LogisticRegression()
8
clf.fit(X_train, y_train)
9
predictions = clf.predict(X_test)

Compute performance metrics like accuracy, precision, recall, or the confusion matrix:

1
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
2

3
acc = accuracy_score(y_test, predictions)
4
print(f"Accuracy: {acc}")
5
print("Classification Report:")
6
print(classification_report(y_test, predictions))
7
print("Confusion Matrix:")
8
print(confusion_matrix(y_test, predictions))

Hyperparameter Tuning#

Algorithms often have parameters that can significantly affect performance. Hyperparameter tuning is the process of systematically searching for optimal parameter combinations. For instance, you might tune the number of trees in a Random Forest or the regularization parameter in Logistic Regression. scikit-learn provides tools like GridSearchCV or RandomizedSearchCV for this purpose:

1
from sklearn.model_selection import GridSearchCV
2
from sklearn.ensemble import RandomForestRegressor
3

4
param_grid = {
5
    "n_estimators": [50, 100, 200],
6
    "max_depth": [None, 5, 10]
7
}
8

9
rf = RandomForestRegressor()
10
grid_search = GridSearchCV(rf, param_grid, cv=3)
11
grid_search.fit(X_train, y_train)
12

13
best_model = grid_search.best_estimator_
14
print(grid_search.best_params_)

Pipeline Creation#

Data cleaning, feature engineering, and modeling can be combined into a single pipeline. This ensures a predictable, repeatable approach:

1
from sklearn.pipeline import Pipeline
2
from sklearn.preprocessing import StandardScaler
3

4
pipeline = Pipeline([
5
    ("scaler", StandardScaler()),
6
    ("regressor", LinearRegression())
7
])
8

9
pipeline.fit(X_train, y_train)
10
pipeline_predictions = pipeline.predict(X_test)

Feature Engineering#

Sometimes raw data is not enough. Domain-specific transformations can drastically improve model performance:

• Combining multiple features (e.g., combining latitude and longitude into a “distance from city center�?metric).
• Extracting text-based features (e.g., using TF-IDF on textual data).
• Transforming non-linear relationships (e.g., using log-transform for highly skewed distributions).

Deep Learning#

For complex tasks like image recognition or natural language processing, deep learning frameworks like TensorFlow or PyTorch come into play. They allow you to create multi-layered neural networks that can handle massive datasets. Basic usage with TensorFlow could look like:

1
import tensorflow as tf
2

3
model = tf.keras.models.Sequential([
4
    tf.keras.layers.Dense(64, activation='relu', input_shape=(X_train.shape[1],)),
5
    tf.keras.layers.Dense(1)  # for regression
6
])
7

8
model.compile(optimizer='adam', loss='mse')
9
model.fit(X_train, y_train, epochs=10, batch_size=32)

Big Data and Distributed Computing#

When dealing with extremely large datasets that don’t fit into your machine’s memory, you can consider:

• Dask for parallel computing on local clusters.
• Apache Spark for distributed data processing and MLlib-based machine learning.
• SQL or NoSQL databases for structured or unstructured data storage and querying.

Transitioning into these areas typically requires a deeper understanding of distributed systems, cluster configurations, and parallel algorithms.