Data Science

Data Science/U20CSCJ11

UNIT-1

PART-A

1. Outline of NumPy

NumPy (Numerical Python) is a powerful library for numerical computing in Python. It provides:

Multidimensional array objects (ndarray).
Functions for performing mathematical, logical, shape manipulation, and statistical operations.
Tools for integrating C/C++ and Fortran code.
Support for large datasets with fast operations due to optimized C code.

2. Creating 1D and 2D Arrays in NumPy

1D Array:

import numpy as np

array_1d = np.array([1, 2, 3, 4, 5])

print(array_1d)

2D Array:

array_2d = np.array([[1, 2, 3], [4, 5, 6]])

print(array_2d)

3. Outline of Pandas

Pandas is a data analysis and manipulation library in Python. It offers:

Data structures: Series (1D) and DataFrame (2D).
Tools for data cleaning, transformation, and aggregation.
Support for handling missing data.
Built-in functions for merging, joining, and reshaping data.

Key Benefits:

Simplifies handling structured data like spreadsheets and databases.
Integrates seamlessly with NumPy for numerical operations.

4. Creating 3D Arrays in NumPy

import numpy as np

array_3d = np.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])

print(array_3d)

5. Sketch about Data Science

Data Science involves extracting insights and knowledge from data using:

Statistics and mathematics.
Programming languages like Python and R.
Machine Learning and AI for predictive analysis.
Tools like Pandas, NumPy, TensorFlow, and more.

Components of Data Science:

Data Collection
Data Cleaning
Data Analysis
Model Building
Visualization and Reporting

6. Benefits of Data Science

Enhanced Decision-Making: Data-driven insights lead to better business strategies.
Automation: Predictive models reduce human intervention.
Personalization: Tailors customer experiences (e.g., recommendation systems).
Optimization: Improves efficiency across industries like healthcare, finance, and logistics.

7. Uses of Data Science

Healthcare: Predicting diseases, personalized treatments.
E-Commerce: Recommendations, trend analysis.
Finance: Fraud detection, credit scoring.
Marketing: Customer segmentation, sentiment analysis.
Transportation: Traffic optimization, autonomous vehicles.

8. Types of Data

Structured Data: Tabular format, easily stored in databases.
Unstructured Data: Free-form like text, images, audio.
Semi-Structured Data: Contains elements of both, like JSON, XML.
Time-Series Data: Data points indexed in time order.

9. Operations using NumPy

Mathematical Operations:

import numpy as np

a = np.array([1, 2, 3])

b = np.array([4, 5, 6])

print(a + b) # Element-wise addition

Matrix Multiplication:

a = np.array([[1, 2], [3, 4]])

b = np.array([[5, 6], [7, 8]])

print(np.dot(a, b))

Statistical Operations:

arr = np.array([1, 2, 3, 4, 5])

print(np.mean(arr), np.std(arr))

10. Pandas DataFrame

DataFrame is a 2D data structure with labeled rows and columns.

Creating a DataFrame:
import pandas as pd
data = {'Name': ['Alice', 'Bob'], 'Age': [25, 30]}
df = pd.DataFrame(data)
print(df)

Viewing Data:

print(df.head()) # First few rows

print(df.tail()) # Last few rows

Viewing Data:

print(df.head()) # First few rows

print(df.tail()) # Last few rows

Filtering:

print(df[df['Age'] > 25])

UNIT-1

PART-B

1.Summarize the Pandas in python and Write a sample program.

Pandas is a popular Python library used for data manipulation and analysis. It provides two main data structures:

Series: One-dimensional labeled array capable of holding any data type (integer, string, float, etc.).
DataFrame: Two-dimensional labeled data structure, similar to a spreadsheet or SQL table.

Key Features of Pandas:

Handles missing data effectively.
Data alignment for operations on data with differing indexes.
Built-in grouping and aggregation functions.
Tools for merging, reshaping, and filtering data.
Excellent for data cleaning and preprocessing.

import pandas as pd

# Create a DataFrame

data = {

'Name': ['Alice', 'Bob', 'Charlie', 'Diana'],

'Age': [25, 30, 35, 40],

'City': ['New York', 'Los Angeles', 'Chicago', 'Houston']

}

df = pd.DataFrame(data)

# Display the DataFrame

print("Original DataFrame:")

print(df)

# Add a new column

df['Salary'] = [70000, 80000, 75000, 85000]

# Filter rows where Age > 30

filtered_df = df[df['Age'] > 30]

# Display the modified DataFrame

print("\nModified DataFrame:")

print(df)

# Display the filtered rows

print("\nFiltered DataFrame (Age > 30):")

print(filtered_df)

# Save the DataFrame to a CSV file

df.to_csv('sample_data.csv', index=False)

o/p

Name Age City

0 Alice 25 New York

1 Bob 30 Los Angeles

2 Charlie 35 Chicago

3 Diana 40 Houston

2.Outline of Box-Plot and write syntax to create it.

A Box Plot (or Box-and-Whisker Plot) is a graphical representation of data distribution based on five summary statistics:

Minimum: The smallest value in the dataset.
First Quartile (Q1): The median of the lower half of the data.
Median (Q2): The middle value of the dataset.
Third Quartile (Q3): The median of the upper half of the data.
Maximum: The largest value in the dataset.

Additionally:

The Interquartile Range (IQR) is $Q 3 - Q 1$ .
Whiskers extend to the smallest and largest values within $1.5 \times IQR$ from the quartiles.
Outliers are points outside the whiskers, often shown as individual dots.

Purpose of a Box Plot:

Summarizes data distribution.
Identifies outliers.
Compares distributions across different groups.

sample code:

import matplotlib.pyplot as plt

# Sample data

data = [7, 8, 8, 5, 4, 6, 7, 8, 3, 9, 10, 6, 4]

# Create a box plot

plt.boxplot(data)

plt.title('Box Plot Example')

plt.ylabel('Values')

plt.show()

3.Outline of NumPy in python? Write a sample program.

NumPy (Numerical Python) is a fundamental library for numerical computing in Python, offering support for arrays, matrices, and many mathematical functions. Below is a structured outline:

. Core Features

N-dimensional Arrays: Homogeneous multidimensional arrays (ndarray).
Mathematical Operations: Element-wise operations, linear algebra, and random number generation.
Broadcasting: Apply operations on arrays of different shapes.
Indexing and Slicing: Access array elements using slicing and indexing.

. Key Functions

Array Creation:
- numpy.array() - Create arrays from lists/tuples.
- numpy.zeros() / numpy.ones() - Initialize arrays with zeros/ones.
- numpy.arange() - Create evenly spaced values.
- numpy.linspace() - Create linearly spaced values.
Array Operations:
- Addition, subtraction, multiplication, division, dot products.
Reshaping Arrays:
- numpy.reshape(), numpy.transpose().
Statistical Operations:
- numpy.mean(), numpy.median(), numpy.std().
Linear Algebra:
- numpy.linalg.inv(), numpy.dot(), numpy.linalg.eig().
Random Numbers:

numpy.random.rand(), numpy.random.randint().

sample code:

import numpy as np

# 1. Creating Arrays

a = np.array([1, 2, 3])

b = np.array([[1, 2], [3, 4]])

# 2. Array Operations

c = a + 5 # Element-wise addition

d = b * 2 # Element-wise multiplication

# 3. Mathematical Operations

sum_array = np.sum(a)

mean_array = np.mean(b)

product = np.dot([1, 2], [3, 4]) # Dot product

# 4. Array Reshaping

reshaped = b.reshape(4, 1)

# 5. Random Numbers

random_array = np.random.rand(3, 3)

# Display Results

print("Array a:", a)

print("Array b:\n", b)

print("Array c (a + 5):", c)

print("Array d (b * 2):\n", d)

print("Sum of a:", sum_array)

print("Mean of b:", mean_array)

print("Dot Product:", product)

print("Reshaped Array b:\n", reshaped)

print("Random Array:\n", random_array)

o/p:

Array a: [1 2 3]

Array b:

[[1 2]

[3 4]]

Array c (a + 5): [6 7 8]

Array d (b * 2):

[[ 2 4]

[ 6 8]]

Sum of a: 6

Mean of b: 2.5

Dot Product: 11

Reshaped Array b:

[[1]

[2]

[3]

[4]]

Random Array:

[[0.41677682 0.93526423 0.04242474]

[0.71555686 0.48251262 0.8344023 ]

[0.08776882 0.9741319 0.33967011]]

4.Develop a data frame using list.

In Python, you can use the pandas library to create a DataFrame from a list. A DataFrame is a two-dimensional, tabular data structure that is ideal for handling structured data. Below is an example:

import pandas as pd

# Single-dimensional list

data = [10, 20, 30, 40, 50]

# Create DataFrame from list

df_single = pd.DataFrame(data, columns=['Numbers'])

print("DataFrame from Single-dimensional List:")

print(df_single)

# Two-dimensional list (list of lists)

data_2d = [

[1, 'Alice', 23],

[2, 'Bob', 27],

[3, 'Charlie', 22]

]

# Create DataFrame from 2D list

df_2d = pd.DataFrame(data_2d, columns=['ID', 'Name', 'Age'])

print("\nDataFrame from Two-dimensional List:")

print(df_2d)

o/p:

DataFrame from Single-dimensional List:

Numbers

0 10

1 20

2 30

3 40

4 50

DataFrame from Two-dimensional List:

ID Name Age

0 1 Alice 23

1 2 Bob 27

2 3 Charlie 22

5.Finding the Transpose of a NumPy array - transpose() method and Reshaping a

NumPy array

Finding the Transpose of a NumPy Array

The numpy.transpose() method or .T attribute is used to get the transpose of an array. The transpose swaps the array's rows and columns.

Syntax -

numpy.transpose(a, axes=None)

a: The array to be transposed.

axes: Optional; specify the order of axes. Default is to reverse axes.

Example: Transpose of a NumPy Array

import numpy as np

# Original Array

array = np.array([[1, 2, 3], [4, 5, 6]])

# Transpose using .T

transpose1 = array.T

# Transpose using numpy.transpose()

transpose2 = np.transpose(array)

print("Original Array:")

print(array)

print("\nTranspose using .T:")

print(transpose1)

print("\nTranspose using numpy.transpose():")

print(transpose2)

output -

Original Array:

[[1 2 3]

[4 5 6]]

Transpose using .T:

[[1 4]

[2 5]

[3 6]]

Transpose using numpy.transpose():

[[1 4]

[2 5]

[3 6]]

Reshaping a NumPy Array

The numpy.reshape() method changes the shape of an array without altering its data.

Syntax

numpy.reshape(a, newshape)

- a: The array to be reshaped.

- newshape: A tuple specifying the desired shape. The product of dimensions must match the size of the original array.

Example: Reshaping a NumPy Array

import numpy as np

# Original Array

array = np.array([1, 2, 3, 4, 5, 6])

# Reshape into a 2x3 array

reshaped1 = array.reshape(2, 3)

# Reshape into a 3x2 array

reshaped2 = array.reshape(3, 2)

# Reshape using -1 (automatic dimension calculation)

reshaped_auto = array.reshape(2, -1)

print("Original Array:")

print(array)

print("\nReshaped into 2x3 Array:")

print(reshaped1)

print("\nReshaped into 3x2 Array:")

print(reshaped2)

print("\nReshaped with Automatic Calculation (2x-1):")

print(reshaped_auto)

Output

Original Array:

[1 2 3 4 5 6]

Reshaped into 2x3 Array:

[[1 2 3]

[4 5 6]]

Reshaped into 3x2 Array:

[[1 2]

[3 4]

[5 6]]

Reshaped with Automatic Calculation (2x-1):

[[1 2 3]

[4 5 6]]

Key Points

Transpose: Rearranges rows and columns.
Reshape: Changes the dimensionality of the array while preserving data.
Use -1 in reshape() to let NumPy calculate one dimension automatically.

6.Finding Mean, Median and Standard deviation on NumPy arrays

Explanation:

Mean: Average of all elements in the array.
Median: Middle value when the elements are sorted.
Standard Deviation: Measure of the amount of variation or dispersion of data values.

Program:

import numpy as np

# Create a NumPy array

data = np.array([4, 2, 7, 5, 9])

# Calculate Mean

mean = np.mean(data) # Sum of all elements divided by total count

# Calculate Median

median = np.median(data) # Middle value in sorted array

# Calculate Standard Deviation

std_dev = np.std(data) # Square root of the average of squared deviations from the mean

print("Array:", data)

print("Mean:", mean)

print("Median:", median)

print("Standard Deviation:", std_dev)

Output:

Array: [4 2 7 5 9]

Mean: 5.4

Median: 5.0

Standard Deviation: 2.280350850198276

7.Write a program to print the eigen values and eigen vectors of a matrix

Explanation:

Eigenvalues: Scalars associated with a matrix that provide insights into its properties.
Eigenvectors: Non-zero vectors that change only in scale when a linear transformation is applied

Program:

import numpy as np

# Create a square matrix

matrix = np.array([[4, 2],

[3, 1]])

# Compute Eigenvalues and Eigenvectors

eigen_values, eigen_vectors = np.linalg.eig(matrix)

print("Matrix:")

print(matrix)

print("\nEigenvalues:")

print(eigen_values)

print("\nEigenvectors:")

print(eigen_vectors)

Output:

Matrix:

[[4 2]

[3 1]]

Eigenvalues:

[5.37228132 -0.37228132]

Eigenvectors:

[[ 0.82456484 -0.41597356]

[ 0.56576746 0.90937671]]

8.Compute and display the sine and cosine of a matrix.

Explanation:

Sine (sin) and cosine (cos) are trigonometric functions.
These are applied element-wise to a matrix.

program

import numpy as np

# Create a matrix

matrix = np.array([[0, np.pi/2],

[np.pi, 3*np.pi/2]])

# Compute Sine

sine_matrix = np.sin(matrix) # Sine of each element in the matrix

# Compute Cosine

cosine_matrix = np.cos(matrix) # Cosine of each element in the matrix

print("Matrix:")

print(matrix)

print("\nSine of Matrix:")

print(sine_matrix)

print("\nCosine of Matrix:")

print(cosine_matrix)

output

Matrix:

[[0. 1.57079633]

[3.14159265 4.71238898]]

Sine of Matrix:

[[ 0.0000000e+00 1.0000000e+00]

[ 1.2246468e-16 -1.0000000e+00]]

Cosine of Matrix:

[[ 1.0000000e+00 6.1232340e-17]

[-1.0000000e+00 -1.8369702e-16]]

UNIT-1

PART-C

1.i) Illustrate the Data Science with its types.

ii) Take any two list, construct a program to plot a scatter plot along with a grid

with appropriate tiles for the graph, X -axis & Y-axis

i) Illustrating Data Science and its Types

What is Data Science?

Data Science is an interdisciplinary field that combines domain expertise, programming skills, and knowledge of mathematics and statistics to extract meaningful insights from structured and unstructured data. It involves processes such as data collection, cleaning, exploration, modeling, and visualization to solve real-world problems.

Types of Data Science

Descriptive Analytics: Focuses on summarizing historical data to understand what happened in the past. Example: Dashboards showing sales trends.
Diagnostic Analytics: Explores data to determine why a certain event occurred. Example: Identifying reasons for a decline in website traffic.
Predictive Analytics: Uses statistical models and machine learning algorithms to forecast future outcomes. Example: Predicting stock prices.
Prescriptive Analytics: Suggests actions to achieve desired outcomes. Example: Recommending products on e-commerce platforms.
Exploratory Data Analysis (EDA): Involves visualizing data to uncover patterns and relationships. Example: Scatter plots, histograms.
Machine Learning/AI Models: Algorithms used for classification, regression, clustering, and more. Example: Fraud detection systems.

ii) Python Program to Plot a Scatter Plot with a Grid

Below is a program that demonstrates how to create a scatter plot using two lists. It includes adding a grid and setting appropriate titles for the graph, X-axis, and Y-axis.

Code Example

import matplotlib.pyplot as plt

# Data for scatter plot

x = [10, 20, 30, 40, 50] # X-axis data

y = [15, 30, 10, 45, 25] # Y-axis data

# Creating the scatter plot

plt.scatter(x, y, color='blue', label='Data Points', s=100) # 's' for marker size

# Adding grid to the plot

plt.grid(True, linestyle='--', linewidth=0.5, alpha=0.7)

# Adding titles and labels

plt.title('Scatter Plot Example with Grid', fontsize=14, fontweight='bold')

plt.xlabel('X-Axis Label', fontsize=12)

plt.ylabel('Y-Axis Label', fontsize=12)

# Adding legend

plt.legend()

# Displaying the plot

plt.show()

Explanation of the Code

Importing Libraries:
- matplotlib.pyplot is imported for plotting the graph.
Data Initialization:
- Two lists, x and y, contain the data points for the X and Y axes.
Scatter Plot:
- plt.scatter(x, y, ...) creates the scatter plot.
- The color of points is set to blue, and the size (s) is increased for better visibility.
Grid Addition:
- plt.grid() adds a grid to the background with customizable line style, width, and transparency (alpha).
Labels and Titles:
- plt.title() sets the graph's title.
- plt.xlabel() and plt.ylabel() add labels to the X and Y axes.
Legend:
- plt.legend() provides a label for the data points.
Display:

plt.show() renders the scatter plot.

Output:

The plot will feature:

A set of scattered points.
A grid with dashed lines for clarity.
Title: Scatter Plot Example with Grid.
X-axis label: X-Axis Label.
Y-axis label: Y-Axis Label.

Let me know if you'd like assistance running this or customizing it further!

2.Explain SciPy and construct a program to calculate the Inverse and Psuedo-Inverse of a input Matrix.

SciPy: Overview

What is SciPy?

SciPy is a Python library used for scientific and technical computing. It builds on the NumPy library and provides a wide range of modules for optimization, integration, interpolation, eigenvalue problems, algebraic equations, and other standard scientific computing tasks.

Key Features of SciPy:

Scientific Functions: Advanced math functions like integration, optimization, and solving differential equations.
Linear Algebra: Matrix operations, eigenvalues, singular value decomposition (SVD), and solving linear systems.
Statistical Functions: Probability distributions, statistical tests, and random sampling.
Signal and Image Processing: Tools for filtering, Fourier transforms, and image manipulation.

Program to Calculate the Inverse and Pseudo-Inverse of a Matrix

The inverse of a square matrix exists only if the matrix is non-singular (determinant ≠ 0). The pseudo-inverse (Moore-Penrose inverse) is a generalized inverse that can be computed for any matrix, including non-square or singular matrices.

Code Example

import numpy as np

from scipy.linalg import inv, pinv # Importing inverse and pseudo-inverse functions

# Input matrix

matrix = np.array([[1, 2], [3, 4]])

# Calculate inverse

try:

inverse_matrix = inv(matrix) # Compute inverse using SciPy

print("Inverse of the Matrix:")

print(inverse_matrix)

except np.linalg.LinAlgError:

print("Matrix is singular and does not have an inverse.")

# Calculate pseudo-inverse

pseudo_inverse_matrix = pinv(matrix) # Compute pseudo-inverse using SciPy

print("\nPseudo-Inverse of the Matrix:")

print(pseudo_inverse_matrix)

Explanation of the Code

Importing Libraries:
- numpy is used to create and handle matrices.
- scipy.linalg.inv computes the matrix inverse.
- scipy.linalg.pinv computes the pseudo-inverse of the matrix.
Input Matrix:
- A 2x2 matrix is initialized as [[1, 2], [3, 4]].
Matrix Inverse:
- inv(matrix) computes the inverse.
- If the matrix is singular (non-invertible), an exception is raised, which is handled using a try-except block.
Matrix Pseudo-Inverse:
- pinv(matrix) computes the Moore-Penrose pseudo-inverse, which exists for all matrices.
Output:

The program prints the inverse (if possible) and the pseudo-inverse of the given matrix.

Output Example

For the input matrix:

Matrix = [\begin{array}{cc} 1 & 2 \\ 3 & 4 \end{array}]

The output will be:

[\begin{array}{cc} 1 & 2 \\ 2 & 4 \end{array}]

UNIT-2

PART-C

1.Explain in various Measuring central tendency with sample program

1. Measuring Central Tendency

Central tendency refers to a single value that represents the center or typical value of a dataset. The three main measures of central tendency are mean, median, and mode. Each of these measures provides insights into the distribution and characteristics of the data

Mean

Definition: The mean is the average of all data points. It is calculated by summing all values and dividing by the total number of values.
Formula:
$Mean = \frac{\sum X}{N}$
Where $X$ is the data points and $N$ is the number of data points.
Example in Python:

import numpy as np

data = [1, 2, 3, 4, 5]

mean = np.mean(data)

print(f"Mean: {mean}")

Advantages:

Easy to calculate.
Takes into account all data points.

Disadvantages:

Sensitive to extreme values (outliers).

Median

Definition: The median is the middle value in a sorted dataset. If the dataset has an even number of values, the median is the average of the two middle values.
Steps to Find Median:
1. Sort the data in ascending order.
2. Find the middle value (or average of two middle values).
Example in Python:
data = [1, 3, 3, 6, 7, 8, 9]
median = np.median(data)
print(f"Median: {median}")

Advantages:
- Not affected by outliers.
Disadvantages:

Does not consider the magnitude of all values.

Mode
- Definition: The mode is the most frequently occurring value in a dataset. A dataset can have no mode, one mode, or multiple modes.
- Example in Python:
  from scipy import stats
  
  data = [1, 2, 2, 3, 4, 4, 4, 5]
  mode = stats.mode(data)
  print(f"Mode: {mode.mode[0]}, Frequency: {mode.count[0]}")
- Advantages:
  - Useful for categorical data.
- Disadvantages:
  - May not exist in some datasets.

2.Explain Normal and Poisson Distribution

Normal Distribution

Definition: A continuous probability distribution that is symmetric and follows a bell-shaped curve.
Key Characteristics:
- Mean ( $μ$ ) and standard deviation ( $σ$ ) define the distribution.
- Approximately 68% of values fall within one standard deviation of the mean, 95% within two, and 99.7% within three.
Real-World Examples:
- Heights of people.
- Test scores.
Python Example:

import numpy as np

import matplotlib.pyplot as plt

from scipy.stats import norm

x = np.linspace(-10, 10, 1000)

normal_pdf = norm.pdf(x, loc=0, scale=1)

plt.plot(x, normal_pdf, label="Normal Distribution")

plt.title("Normal Distribution")

plt.legend()

plt.show()

Poisson Distribution

Definition: A discrete probability distribution used to model the number of events occurring in a fixed interval of time or space.
Key Characteristics:
- Parameterized by $λ$ (average rate of occurrence).
- Events are independent.
- Example: Number of cars passing a checkpoint per hour.
Real-World Examples:
- Calls received by a call center.
- Number of customers arriving at a store.
Python Example:
from scipy.stats import poisson
import matplotlib.pyplot as plt

k = range(0, 15)
poisson_pmf = poisson.pmf(k, mu=5)

plt.stem(k, poisson_pmf, basefmt=" ", label="Poisson Distribution")
plt.title("Poisson Distribution")
plt.legend()
plt.show()

3.Implement Anova test using python
ANOVA (Analysis of Variance) is a statistical method used to determine whether there are significant differences between the means of three or more groups. It evaluates the variability within and between groups to assess whether the group means differ more than would be expected by random chance.

Key Concepts of ANOVA
Null Hypothesis ( $H_{0}$ ):
Assumes that all group means are equal.
Example: In an experiment comparing test scores across three teaching methods, $H_{0}$ states that all teaching methods result in the same mean test score.
Alternative Hypothesis ( $H_{1}$ ):
At least one group mean is different from the others.
F-Statistic:
A ratio of the variance between groups to the variance within groups.
Formula: $F = \frac{Variance Between Groups}{Variance Within Groups}$
A high $F$ -statistic value suggests significant differences between group means.
P-Value:
The probability of observing the data (or something more extreme) if $H_{0}$ is true.
If $p < (commonly 0.05), we reject$ $H_{0}$ .

Steps to Perform ANOVA

Define Groups: Identify the different groups whose means you want to compare.
Calculate Variances:
- Between-group variance: Measures variability due to differences between group means.
- Within-group variance: Measures variability within each group.
Compute F-Statistic: Use the variances to calculate the $F$ -statistic.
Evaluate P-Value: Determine if the $F$ -statistic corresponds to a p-value less than the significance level.

Types of ANOVA

One-Way ANOVA:
- Tests for differences between means of groups based on one independent variable.
- Example: Comparing test scores among three different schools.
Two-Way ANOVA:

Tests the influence of two independent variables on the dependent variable.
Example: Examining the effect of teaching methods and gender on test scores
.

Python Implementation: One-Way ANOVA

Here’s an example where we compare test scores across three different study methods:

from scipy.stats import f_oneway

# Define data for three groups (e.g., three study methods)

method1 = [75, 80, 85, 90, 95]

method2 = [70, 78, 85, 88, 92]

method3 = [65, 72, 78, 85, 88]

# Perform one-way ANOVA

f_stat, p_value = f_oneway(method1, method2, method3)

print("ANOVA Results")

print(f"F-Statistic: {f_stat}")

print(f"P-Value: {p_value}")

# Interpretation

alpha = 0.05

if p_value < alpha:

print("Reject the null hypothesis: At least one group mean is significantly different.")

else:

print("Fail to reject the null hypothesis: No significant difference between group means.")

Output Explanation:

The $F$ -statistic quantifies the ratio of between-group variance to within-group variance.
The $p$ -value determines if the observed differences are statistically significant.

Python Implementation: Two-Way ANOVA

Two-way ANOVA is used when there are two independent variables. Here's an example with two factors: study method and gender.

import pandas as pd

import statsmodels.api as sm

from statsmodels.formula.api import ols

# Create a dataset

data = pd.DataFrame({

"Score": [85, 78, 92, 88, 95, 72, 80, 85, 90, 88],

"Method": ["A", "A", "A", "B", "B", "B", "C", "C", "C", "C"],

"Gender": ["M", "F", "M", "F", "M", "F", "M", "F", "M", "F"]

})

# Fit the two-way ANOVA model

model = ols('Score ~ C(Method) + C(Gender) + C(Method):C(Gender)', data=data).fit()

anova_table = sm.stats.anova_lm(model, typ=2)

print(anova_table)

Output Explanation:

C(Method): Tests the main effect of study methods.
C(Gender): Tests the main effect of gender.
C(Method):C(Gender): Tests interaction between study method and gender.

Visualizing ANOVA Results

Visualizations help in understanding the group differences visually.

import matplotlib.pyplot as plt

import seaborn as sns

# Boxplot to visualize group differences

sns.boxplot(x="Method", y="Score", hue="Gender", data=data)

plt.title("Scores by Study Method and Gender")

plt.show()

interpreting Results

If $F$ -statistic is high and $p$ -value is less than 0.05:
- Reject $H_{0}$ : At least one group mean is significantly different.
If $F$ -statistic is low or $p$ -value is greater than 0.05:

Fail to reject $H_{0}$ : No significant difference in group means.

Advantages of ANOVA

Efficiently compares multiple groups simultaneously.
Reduces the risk of Type I error compared to performing multiple $t$ -tests.

Disadvantages of ANOVA

Assumes normality of data.
Sensitive to unequal variances across groups.

When to Use ANOVA

To test hypotheses about group differences when you have:
1. Continuous dependent variable.
2. Categorical independent variable(s).
3. More than two groups to compare.

This comprehensive explanation and Python implementation should give you a deep understanding of ANOVA and its applications. Let me know if you'd like to explore further or need clarification

4.Develop a python program to implement two sampled t-test.

import numpy as np
from scipy.stats import ttest_ind
# Input data
# Replace these lists with your own data
sample1 = [25, 30, 35, 40, 45]
sample2 = [20, 22, 27, 30, 35]
# Perform the two-sample t-test
t_stat, p_value = ttest_ind(sample1, sample2, equal_var=False) # Set equal_var=True if variances are assumed equal
# Output results
print("Two-Sample T-Test Results")
print(f"T-Statistic: {t_stat:.3f}")
print(f"P-Value: {p_value:.3f}")
# Interpretation of the results
alpha = 0.05 # Significance level
if p_value < alpha:
print("Reject the null hypothesis: The two samples have significantly different means.")
else:
print("Fail to reject the null hypothesis: No significant difference between the sample means.")

Explanation:

Input Data: Replace sample1 and sample2 with your actual datasets.
ttest_ind Function:
- equal_var: Set to True if the two samples are assumed to have equal variances; otherwise, set to False.
- Returns t_stat (t-statistic) and p_value (probability of observing the data if the null hypothesis is true).
Significance Level:
- The alpha value (commonly 0.05) is the threshold for deciding whether to reject the null hypothesis.
Output:

Prints the t-statistic and p-value.
Interprets the result based on the p-value compared to the significance level.

output:

Two-Sample T-Test Results

T-Statistic: 1.359

P-Value: 0.208

Fail to reject the null hypothesis: No significant difference between the sample means.

5.Develop programs for one sampled t-test and Paired sampled t-test.

One-Sample T-Test

The one-sample t-test checks if the mean of a sample differs significantly from a known population mean.

Code for One-Sample T-Test:

from scipy.stats import ttest_1samp

# Sample data

sample = [12, 15, 14, 10, 13, 14, 15, 16]

# Hypothesized population mean

pop_mean = 14

# Perform the one-sample t-test

t_stat, p_value = ttest_1samp(sample, pop_mean)

# Output results

print("One-Sample T-Test Results")

print(f"T-Statistic: {t_stat:.3f}")

print(f"P-Value: {p_value:.3f}")

# Interpretation

alpha = 0.05 # Significance level

if p_value < alpha:

print("Reject the null hypothesis: The sample mean is significantly different from the population mean.")

else:

print("Fail to reject the null hypothesis: The sample mean is not significantly different from the population mean.")

Paired-Sample T-Test

The paired-sample t-test (or dependent t-test) is used when comparing the means of two related groups to see if they are significantly different from each other.

Code for Paired-Sample T-Test:

from scipy.stats import ttest_rel

# Sample data (before and after treatment)

sample1 = [85, 89, 91, 87, 92] # Before treatment

sample2 = [88, 90, 92, 85, 95] # After treatment

# Perform the paired sample t-test

t_stat, p_value = ttest_rel(sample1, sample2)

# Output results

print("Paired-Sample T-Test Results")

print(f"T-Statistic: {t_stat:.3f}")

print(f"P-Value: {p_value:.3f}")

# Interpretation

alpha = 0.05 # Significance level

if p_value < alpha:

print("Reject the null hypothesis: There is a significant difference between the paired samples.")

else:

print("Fail to reject the null hypothesis: No significant difference between the paired samples.")

One-Sample T-Test:

One-Sample T-Test Results

T-Statistic: -1.414

P-Value: 0.203

Fail to reject the null hypothesis: The sample mean is not significantly different from the population mean.

Paired-Sample T-Test:

Paired-Sample T-Test Results

T-Statistic: -2.828

P-Value: 0.047

Reject the null hypothesis: There is a significant difference between the paired samples.

6.Identify the important parameters of Hypothesis testing.

In hypothesis testing, several important parameters are considered to evaluate whether there is enough evidence to accept or reject the null hypothesis. The key parameters involved in hypothesis testing are:

1. Null Hypothesis (H₀)

The null hypothesis represents the default assumption or the statement that there is no effect, no difference, or no relationship.
Example: "There is no significant difference between the two groups."

2. Alternative Hypothesis (H₁ or Ha)

The alternative hypothesis represents the statement that contradicts the null hypothesis. It is what the researcher is trying to prove.
Example: "There is a significant difference between the two groups."

3. Test Statistic

The test statistic is a numerical value that summarizes the data for the hypothesis test. It is used to determine whether to reject the null hypothesis.
The specific test statistic depends on the type of test (e.g., t-statistic for t-tests, z-statistic for z-tests).
Example: For a t-test, the test statistic is calculated as: $t = \frac{\overset{ˉ}{X} - μ}{\frac{s}{\sqrt{n}}}$ where $\overset{ˉ}{X}$ is the sample mean, $μ$ is the population mean, $s$ is the sample standard deviation, and $n$ is the sample size.

4. Significance Level (α)

The significance level (alpha) is the probability of rejecting the null hypothesis when it is actually true (Type I error). It represents the threshold for deciding whether the result is statistically significant.
Common values are 0.05, 0.01, or 0.10.
Example: If $α = 0.05$ , it means you are willing to accept a 5% chance of incorrectly rejecting the null hypothesis.

5. P-Value

The p-value is the probability of obtaining a test statistic as extreme as, or more extreme than, the observed result, under the assumption that the null hypothesis is true.
If the p-value is less than the significance level (α), you reject the null hypothesis.
Example: If $p = 0.03$ and $α = 0.05$ , you reject the null hypothesis, as 0.03 < 0.05.

6. Critical Value

The critical value is a point (or points) on the test statistic distribution that marks the boundary for rejecting the null hypothesis.
For a two-tailed test, there are two critical values (one for each tail of the distribution). For a one-tailed test, there is one critical value.
Example: In a z-test with $α = 0.05$ , the critical z-values for a two-tailed test would be approximately ±1.96.

7. Confidence Interval

A confidence interval is a range of values, derived from the sample, that is likely to contain the true population parameter with a given level of confidence (usually 95% or 99%).
If the confidence interval does not include the value under the null hypothesis (e.g., 0 for a difference), you may reject the null hypothesis.
Example: A 95% confidence interval for the difference in means might be [1.2, 5.4], suggesting that the difference is likely between 1.2 and 5.4, and thus not equal to 0.

8. Type I Error (α)

Type I error occurs when the null hypothesis is incorrectly rejected when it is actually true.
Example: Concluding that there is a difference between two groups when, in fact, there is none.

9. Type II Error (β)

Type II error occurs when the null hypothesis is incorrectly accepted (or not rejected) when it is actually false.
Example: Concluding that there is no difference between two groups when, in fact, there is one.

10. Power of the Test

The power of a test is the probability that it will correctly reject the null hypothesis when it is false (i.e., avoid a Type II error). Power is equal to $1 - β$ .
Example: A test with 80% power means there is an 80% chance of detecting an effect if it truly exists.

11. Sample Size (n)

The sample size is the number of observations or data points in the sample used for hypothesis testing. A larger sample size typically increases the power of the test, making it easier to detect a significant effect.
Example: A sample size of 100 is more likely to provide reliable results than a sample size of 10.

Summary of Key Parameters:

Null Hypothesis (H₀) and Alternative Hypothesis (H₁)
Test Statistic (t-statistic, z-statistic, etc.)
Significance Level (α)
P-Value
Critical Value
Confidence Interval
Type I Error (α) and Type II Error (β)
Power of the Test
Sample Size (n)

Understanding these parameters is crucial for interpreting the results of hypothesis testing and making decisions based on statistical evidence

UNIT-3

PART-A

1. Identify Downsampling

Downsampling refers to the process of reducing the frequency of the data. For instance, if you have minute-wise data, you might aggregate it to hourly data by taking the mean, sum, or other appropriate aggregation method. This helps in simplifying the data for analysis, reducing noise, or matching the data to the frequency required for modeling or analysis.

2. Show the note on Timedelta

Timedelta is a concept in Python (specifically in the pandas library) that represents the difference between two datetime objects. It can be used for performing arithmetic on date and time values, such as calculating the difference between two dates or adding/subtracting time from a datetime object.

import pandas as pd

# Create datetime objects

date1 = pd.to_datetime('2024-12-01')

date2 = pd.to_datetime('2024-12-04')

# Calculate the time difference

delta = date2 - date1

print(delta) # Output: 3 days

3. Classify AR & MA

AR (Autoregressive Model): A time series model where the current value is based on the previous values of the series. The current value of the series depends linearly on its previous values.
MA (Moving Average Model): A time series model that models the current value as a linear combination of past error terms (or residuals). It uses the error terms from previous time steps to predict future values.

4. Outline of Resampling

Resampling is the process of changing the frequency of the time series data. There are two main types of resampling:
- Downsampling: Reducing the frequency of data (e.g., from minute-level data to hourly data).
- Upsampling: Increasing the frequency of data (e.g., from daily data to hourly data), typically by adding missing data through interpolation.

5. Short Notes on Upsampling

Upsampling refers to increasing the frequency of time series data. This typically involves filling in missing values or interpolating values between existing data points. Upsampling is useful when you need to work with a higher resolution of data but don’t have measurements for those periods. Common techniques include forward-fill or linear interpolation.

6. Classify ARMA & ARIMA

ARMA (Autoregressive Moving Average): A combination of AR and MA models used when the time series is stationary (i.e., no trend or seasonality). It models the relationship between past values and past errors.
ARIMA (Autoregressive Integrated Moving Average): Extends ARMA by adding an integration (I) component, which is used to model non-stationary time series data. The integration step removes trends by differencing the data before applying ARMA modeling.

7. What is an Autocorrelation Function?

The Autocorrelation Function (ACF) is a tool used to measure the correlation between a time series and its lagged version. It helps to identify repeating patterns such as seasonality and cyclic behavior within the data. High autocorrelation values at specific lags indicate that past values are significantly correlated with future values at those lags.

8. What are the Advantages of Time Series Forecasting?

Advantages:
- Trend Identification: Helps identify trends in data over time, which can be used for making predictions.
- Seasonality Detection: Allows for the detection of seasonal patterns, which can be leveraged for better forecasting.
- Improved Decision Making: Provides data-driven predictions that help businesses in planning, inventory management, and resource allocation.
- Cost Efficiency: Accurate forecasts help reduce costs by avoiding overproduction or stockouts.

9. State the Applications of Time Series Forecasting

Applications:
- Stock Market Prediction: Predicting future stock prices based on historical trends.
- Sales Forecasting: Predicting future sales to help businesses manage inventory and demand.
- Weather Prediction: Forecasting weather patterns based on historical data.
- Economic Forecasting: Predicting economic indicators such as GDP, unemployment rates, and inflation.
- Energy Demand Forecasting: Estimating future energy consumption for power grid management.

10. Identify the Type of Data with Time Stamp and Explore its Uses

Time-Stamped Data refers to data that includes a timestamp or a date-time value associated with each observation. This type of data is typically used in time series analysis.

Uses:

Tracking Events Over Time: Used in domains like stock prices, sensor readings, and IoT devices, where the data points are indexed by time.
Forecasting: Used in predictive modeling where future values are forecasted based on past data.
Trend and Pattern Analysis: Helps in identifying long-term trends, seasonal patterns, and anomalies in data collected over time.

UNIT-3

PART-A

1. Explain Upsampling with Polynomial Interpolation

Upsampling with Polynomial Interpolation is a technique used to increase the frequency of time series data. It involves generating new data points between existing points by fitting a polynomial to the data and then using it to interpolate missing values.
Steps:
1. Increase the frequency: Upsample the time series by adding the desired number of missing time steps.
2. Fit a polynomial: Fit a polynomial function to the existing data points using polynomial interpolation.
3. Generate new values: Use the fitted polynomial to compute values for the new time steps.

code:

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

from numpy.polynomial.polynomial import Polynomial

# Example data

date_rng = pd.date_range(start='2024-12-01', end='2024-12-05', freq='D')

data = np.array([1, 2, 3, 4, 5])

df = pd.DataFrame(data, index=date_rng, columns=['Value'])

# Upsample by interpolating polynomially (quadratic interpolation here)

df_upsampled = df.resample('H').interpolate(method='polynomial', order=2)

# Plotting the original and upsampled series

df.plot(label='Original Data')

df_upsampled.plot(label='Upsampled Data', linestyle='--')

plt.legend()

plt.show()

2. Explain About Resampling

Resampling in time series refers to the process of changing the frequency of your data points. There are two main types of resampling:
- Downsampling: Reducing the frequency by aggregating values (e.g., converting minute data to hourly data by averaging or summing).
- Upsampling: Increasing the frequency of data, typically by adding missing values or interpolating existing values (e.g., converting daily data to hourly data).
Methods:
- Resample: pandas.DataFrame.resample() is used to change the frequency, and you can apply aggregation functions like mean, sum, etc., to handle the downsampling.
- Interpolation: For upsampling, you can use interpolation methods like linear or polynomial interpolation.

3. Construct a Note on TimeDelta

Timedelta is a concept in Python (in the pandas library) that represents the difference between two datetimeobjects. It allows you to perform arithmetic operations with dates and times.
Example Uses:

Adding or subtracting time from a datetime object.
Calculating the difference between two dates/times.

code:

import pandas as pd

date1 = pd.to_datetime('2024-12-01')

date2 = pd.to_datetime('2024-12-04')

delta = date2 - date1

print(f"Time Difference: {delta}")

4. Explain About Downsampling

Downsampling refers to the process of reducing the frequency of data. It is commonly used when you want to aggregate time series data over a larger time period to simplify analysis, reduce noise, or focus on higher-level trends.
Methods:
- You can use aggregation functions like mean, sum, min, max to resample the data from a higher frequency (e.g., minute-wise) to a lower frequency (e.g., hourly or daily).

code:

df.resample('D').mean() # Resampling data to daily frequency, calculating the mean

5. Identify the Function to Calculate the Autocorrelation of a Time Series

The Autocorrelation Function (ACF) is used to measure the correlation between a time series and its lagged version. In Python, you can use the acf function from the statsmodels library to calculate the autocorrelation.

code:

from statsmodels.tsa.stattools import acf

import pandas as pd

import numpy as np

# Example time series data

data = np.random.randn(100)

acf_values = acf(data, nlags=20)

print(acf_values)

6. Develop a Program to Illustrate the Concept of Time Deltas

import pandas as pd

# Create two datetime objects

date1 = pd.to_datetime('2024-12-01')

date2 = pd.to_datetime('2024-12-05')

# Calculate the difference

delta = date2 - date1

print(f"Time difference: {delta}")

# Add the delta to a date

new_date = date1 + delta

print(f"New Date: {new_date}")

7. Apply Functions in Python to Develop an AutoRegressive (AR) Model

from statsmodels.tsa.ar_model import AutoReg

import pandas as pd

import numpy as np

# Example data

data = pd.Series(np.random.randn(100))

# Fit AR model

model = AutoReg(data, lags=1) # AR(1) model

model_fitted = model.fit()

# Get the coefficients

print("AR Model Coefficients:", model_fitted.params)

8. State the Differences Between Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF)

ACF (Autocorrelation Function): Measures the correlation between a time series and its lagged versions. It shows how the series is correlated with its previous values over various lags.
PACF (Partial Autocorrelation Function): Measures the correlation between a time series and its lagged versions, after removing the effect of shorter lags. It helps in determining the order of an AR (AutoRegressive) model.

9. Develop a Small Program to Print a Time Series

Program:

import pandas as pd

date_rng = pd.date_range(start='2024-12-01', periods=10, freq='D')

data = pd.Series(range(10), index=date_rng)

print(data)

10. Develop a Small Program to Plot a Time Series

import pandas as pd

import matplotlib.pyplot as plt

# Generate a time series

date_rng = pd.date_range(start='2024-12-01', periods=10, freq='D')

data = pd.Series(range(10), index=date_rng)

# Plot the time series

data.plot()

plt.title('Time Series Plot')

plt.xlabel('Date')

plt.ylabel('Value')

plt.show()

UNIT-3

PART-B

1.Explain upsampling with a polynomial interpolation

1. Upsampling with Polynomial Interpolation

Upsampling is the process of increasing the number of samples in a signal or dataset by inserting additional data points between the existing points. This process is commonly used in time series analysis or signal processing when you need to convert a dataset to a higher resolution. Polynomial interpolation is one way to achieve upsampling, where we fit a polynomial to the known data points and use this polynomial to estimate the values of the newly inserted data points.
Polynomial interpolation works by finding a polynomial that exactly fits the set of given data points. Once the polynomial is found, it can be evaluated at new points (i.e., the points between the original data points), thereby "upsampling" the signal.
The BarycentricInterpolator from the `scipy.interpolate` module allows us to perform polynomial interpolation effectively.

Code Example for Polynomial Interpolation:

import numpy as np

from scipy.interpolate import BarycentricInterpolator

import matplotlib.pyplot as plt

# Original data points (known values)

x = np.array([0, 1, 2, 3, 4])

y = np.array([0, 1, 4, 9, 16]) # y = x^2 (just for example)

# Polynomial interpolation

interpolator = BarycentricInterpolator(x, y)

x_new = np.linspace(0, 4, 100) # New x values (higher resolution)

y_new = interpolator(x_new) # Interpolated y values

# Plot the original and interpolated data

plt.plot(x, y, 'o', label='Original Data')

plt.plot(x_new, y_new, label='Interpolated Data')

plt.legend()

plt.title('Upsampling with Polynomial Interpolation')

plt.show()

In this example, we create a set of original data points (x, y), then use polynomial interpolation to estimate new data points between the original values (x_new). We plot both the original and interpolated data, showing how the interpolation fills in the new values smoothly.

2.Explain about Resampling

A. Resampling

Resampling refers to the process of changing the frequency or the number of data points in a time series. It can either involve downsampling (reducing the frequency or data points) or upsampling (increasing the frequency or inserting more data points). The main goal of resampling is to adjust the data so that it meets certain requirements, such as uniform time intervals or different sampling rates.

Downsampling involves aggregating data points over time. For example, converting minute-level data to hourly data by averaging or summing the data.
Upsampling involves inserting data points where none exist. This is typically done by interpolating the values, as we discussed in the previous section.

Resampling is widely used in various fields such as finance, time series forecasting, and signal processing.

Example in Python with pandas:

import pandas as pd

# Create a time series with daily frequency

dates = pd.date_range('2020-01-01', periods=10, freq='D')

data = pd.Series(range(10), index=dates)

# Upsampling: changing the frequency to hourly and filling missing values by forward fill

upsampled_data = data.resample('H').ffill()

print("Upsampled Data:")

print(upsampled_data)

# Downsampling: changing the frequency to weekly and summing the values for each week

downsampled_data = data.resample('W').sum()

print("Downsampled Data:")

print(downsampled_data)

In this example, resample() is used to change the frequency of the data. ffill() is applied to upsample the data by forward filling missing values, while sum() is used to aggregate the data when downsampling to weekly frequency.

3.Construct a note on TimeDelta.

TimeDelta

A TimeDelta represents the difference between two datetime objects. It is a fundamental concept in working with time and date arithmetic. In Python, we use the datetime module to perform operations involving time deltas.

A timedelta object has attributes like days, seconds, and microseconds, and can be used to perform arithmetic operations on date or time objects.

Example:

from datetime import datetime, timedelta

# Create two datetime objects

dt1 = datetime(2024, 12, 1, 10, 0, 0) # 1st December 2024, 10:00:00 AM

dt2 = datetime(2024, 12, 4, 15, 0, 0) # 4th December 2024, 3:00:00 PM

# Calculate the time delta (difference between two dates)

delta = dt2 - dt1

# Print the time delta

print(f"Time Delta: {delta}")

print(f"Total Days: {delta.days}")

print(f"Total Seconds: {delta.total_seconds()}")

In this example:

The difference between dt1 and dt2 results in a timedelta object.
We can then access various attributes of the timedelta, such as days and total_seconds()

4.Explain about Downsampling

Downsampling is the process of reducing the number of samples in a signal or dataset. For time series data, downsampling typically involves aggregating the data points into fewer, more representative data points.

For example, you may have minute-level data, but you might want to reduce the frequency to hourly data by calculating the average or sum of the values for each hour.

Example of Downsampling:

import pandas as pd

# Create a time series with hourly frequency

dates = pd.date_range('2024-12-01', periods=24, freq='H')

data = pd.Series(range(24), index=dates)

# Downsample to daily frequency by summing the values for each day

downsampled_data = data.resample('D').sum()

print("Downsampled Data:")

print(downsampled_data)

In this example:

The original data is sampled at hourly intervals.
The data is downsampled to daily frequency by summing up the hourly data for each day.

5.Identify the function to calculate the autocorrelation of a time series

Autocorrelation Function (ACF)

The Autocorrelation Function (ACF) measures the correlation between a time series and its lagged version. It helps to determine how well the past values of a time series predict future values. The ACF is essential in identifying the dependencies and patterns in time series data, and it is widely used in time series forecasting and model building (e.g., ARIMA models).

ACF Calculation with Python:

from statsmodels.tsa.stattools import acf

import numpy as np

# Generate a random time series for demonstration

time_series = np.random.randn(100)

# Compute the ACF of the time series

autocorr = acf(time_series, nlags=10)

print("Autocorrelation function (ACF):")

print(autocorr)

In this example:

We generate a random time series and then calculate its ACF using the acf function from statsmodels.
The nlags parameter specifies the number of lags to calculate the autocorrelation for.

6.Develop a program to illustrate the concept of Time Deltas.

Program to Illustrate Time Deltas

This example shows how to use timedelta to compute the difference between two dates and perform time-related arithmetic operations:

from datetime import datetime, timedelta

# Create datetime objects

dt1 = datetime(2024, 12, 1, 10, 0, 0)

dt2 = datetime(2024, 12, 4, 15, 0, 0)

# Calculate time difference

delta = dt2 - dt1

# Display results

print(f"Time difference: {delta}")

print(f"Days difference: {delta.days} days")

print(f"Total seconds: {delta.total_seconds()} seconds")

In this case, we calculate the difference between two datetime objects (dt1 and dt2) and print out both the total days and total seconds.

7.Apply functions in python to develop an AutoRegressive(AR) Model

AutoRegressive (AR) Model in Python

An AutoRegressive (AR) Model is a time series model where the value of a series at time t depends linearly on its previous values. For instance, an AR(1) model would use only the previous value to predict the next one.

AR Model in Python:

from statsmodels.tsa.ar_model import AutoReg

import pandas as pd

# Sample data (time series)

data = pd.Series([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

# Fit an AutoRegressive model

model = AutoReg(data, lags=1) # AR(1) model

model_fitted = model.fit()

# Display the fitted model summary

print(model_fitted.summary())

Here:

We use AutoReg from statsmodels to fit an AR model.
We specify lags=1, meaning that the model uses only the previous time point for prediction.

8.State the differences between Autocorrelation function(ACF) and Partial

Correlation function(PACF).

The Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) are both used in time series analysis to assess the relationships between a time series and its lagged values. However, they differ in how they measure these relationships.

Autocorrelation Function (ACF):

Definition: ACF measures the linear relationship or correlation between a time series and its own lagged versions, i.e., how well the past values of a series predict its future values at different lags.
Purpose: ACF is used to identify the degree of correlation between the series and its lagged values. It helps in identifying whether the data has repeating patterns or periodic trends.
Key Feature: ACF includes the effect of all intermediate lags. This means that when calculating the ACF for a given lag, it accounts for the correlations at previous lags as well.
Interpretation: If the ACF value is significantly high at a particular lag, it indicates that the values at that lag are strongly correlated with the current value.

Partial Autocorrelation Function (PACF):

Definition: PACF measures the linear relationship between a time series and its lagged values, but it does so after removing the effects of the intermediate lags. In other words, PACF isolates the direct correlation between the series and a particular lag, without the influence of previous lags.
Purpose: PACF is used to determine the "order" of an autoregressive (AR) model. It helps in identifying which lags are significant when constructing an AR model for time series forecasting.
Key Feature: PACF eliminates the indirect effects of other lags and focuses on the immediate influence of a particular lag. For instance, PACF for lag 3 gives the correlation between the series and its third lag after removing the influences of lags 1 and 2.
Interpretation: The PACF is particularly helpful when determining the number of AR terms to include in an ARIMA model. A significant PACF spike at lag 1, for example, suggests that only the first lag is important for modeling, while further lags may be unnecessary.

Key Differences:

ACF: Accounts for all previous lags (longer dependencies).
PACF: Only measures the correlation after accounting for the influence of intermediate lags (shorter dependencies).
Use Case: ACF is useful for identifying the MA (Moving Average) part of a time series model, while PACF is used to determine the AR (Autoregressive) part.

Graphical Example:

ACF: If you plot the ACF for a time series, you will observe a series of bars that represent the correlation of the series with its lags.
PACF: If you plot the PACF, you’ll typically see a sharp cutoff after the first few lags, indicating that the correlations for higher lags are not direct but mediated through previous lags.

9.Develop a small program to print a time series.

To print a time series in Python, we can use the pandas library to generate a time series of data and display it. Here is a simple program to demonstrate this:

import pandas as pd

# Generate a time series with daily frequency

dates = pd.date_range('2024-01-01', periods=10, freq='D') # 10 days starting from January 1, 2024

data = pd.Series([100, 200, 300, 400, 500, 600, 700, 800, 900, 1000], index=dates)

# Print the time series

print("Time Series:")

print(data)

Explanation:

The pd.date_range function generates a range of dates, starting from January 1, 2024, with a total of 10 days.
We create a time series with corresponding values (e.g., sales numbers) for each date.
The pd.Series function takes the list of values and the generated dates as the index, creating the time series.
Finally, we print the time series using print(data).

10.Develop a small program to plot a time series.

To plot a time series, we can use the matplotlib library to visualize the data. Here’s a small program to plot a time series:

Code Example:

import pandas as pd

import matplotlib.pyplot as plt

# Generate a time series with daily frequency

dates = pd.date_range('2024-01-01', periods=10, freq='D')

data = pd.Series([100, 200, 300, 400, 500, 600, 700, 800, 900, 1000], index=dates)

# Plot the time series

plt.figure(figsize=(10, 6))

plt.plot(data.index, data.values, marker='o', linestyle='-', color='b', label='Sales Data')

# Add labels and title

plt.xlabel('Date')

plt.ylabel('Sales')

plt.title('Time Series Plot of Sales Data')

plt.legend()

# Show the plot

plt.grid(True)

plt.show()

Explanation:

We generate a time series using the same pd.date_range and pd.Series method as in the previous example.
We use matplotlib.pyplot to plot the data. The plt.plot() function is used to create the plot, specifying the dates as the x-axis and the values as the y-axis.
We also set the figure size, add markers, lines, and a blue color (color='b'), and label the axes and title.
plt.grid(True) adds a grid to the plot for better readability, and plt.legend() shows the label for the data line.
Finally, plt.show() displays the plot.

Output: The result is a graphical plot showing the sales data over the 10 days:

The x-axis represents the dates.
The y-axis represents the sales numbers.
The points on the plot are connected by a line, and markers (o) indicate each data point.

UNIT-3

PART-C

1.i) Illustrate the ANOVA Test.

ii) Explain P-Test in Exponential Hypothesis.

The ANOVA (Analysis of Variance) test is a statistical method used to compare the means of three or more groups to determine if at least one group mean is significantly different from the others. It is used when you have more than two groups and you want to test the null hypothesis that all group means are equal.

Steps in ANOVA:

Null Hypothesis (H₀): The means of the groups are equal.
- $H_{0} : μ_{1} = μ_{2} = μ_{3} = . . . = μ_{k}$
Alternative Hypothesis (H₁): At least one group mean is different.
Assumptions:
- The samples are independent.
- The data in each group are approximately normally distributed.
- The variance across groups is equal (homogeneity of variance).

ANOVA Calculation:

F-statistic: It is the ratio of the variance between the groups to the variance within the groups. $F = \frac{Between-group variance}{Within-group variance}$
- If the F-statistic is large, the null hypothesis is rejected, indicating that at least one group mean is significantly different.

Example:

Suppose you want to test if three different teaching methods have the same effect on student performance. You collect student scores from three groups, each using a different method. You can perform an ANOVA test to determine if there is a significant difference in the means of the three groups.

Python Example:

import numpy as np

import scipy.stats as stats

# Data for three groups

group1 = np.array([83, 90, 78, 95, 88])

group2 = np.array([70, 75, 72, 68, 74])

group3 = np.array([80, 85, 82, 90, 87])

# Perform ANOVA test

f_statistic, p_value = stats.f_oneway(group1, group2, group3)

print("F-statistic:", f_statistic)

print("P-value:", p_value)

# Interpretation

if p_value < 0.05:

print("Reject the null hypothesis: There is a significant difference between the groups.")

else:

print("Fail to reject the null hypothesis: There is no significant difference between the groups.")

Explanation:

We used scipy.stats.f_oneway to perform a one-way ANOVA test on three independent groups.
If the p-value is less than 0.05, we reject the null hypothesis, implying a significant difference between the group means.

ii) Explain P-Test in Exponential Hypothesis

A P-test is used in hypothesis testing to determine the statistical significance of a test result. In the context of an Exponential Hypothesis, it is used to test hypotheses about the exponential distribution, such as the mean or rate parameter of an exponentially distributed population.

Exponential Distribution:

The probability density function (PDF) of an exponential distribution is given by: $f (x ∣ λ) = λ e^{- λ x}, x \geq 0$ where $λ$ is the rate parameter (the inverse of the mean).

P-Test for Exponential Hypothesis:

Null Hypothesis (H₀): The data follows an exponential distribution with a specific rate parameter $λ_{0}$ .
Alternative Hypothesis (H₁): The data does not follow the exponential distribution with rate parameter $λ_{0}$ .

Test Statistic:

The test statistic often used is the likelihood ratio test or a Chi-square test for the goodness of fit.
p-value: The p-value is calculated by comparing the test statistic with the expected distribution under the null hypothesis. A small p-value indicates evidence against the null hypothesis.

2.Explain the Time series forecasting with ARMA & ARIMA.

ARMA (AutoRegressive Moving Average):

ARMA is a time series model that combines two components: AutoRegressive (AR) and Moving Average (MA).

AR: The autoregressive component models the dependency between an observation and a number of lagged observations.
- The general form is: $X_{t} = ϕ_{1} X_{t - 1} + ϕ_{2} X_{t - 2} + \dots + ϕ_{p} X_{t - p} + ϵ_{t}$ where $ϕ_{1}, ϕ_{2}, \dots, ϕ_{p}$ are the coefficients and $ϵ_{t}$ is white noise.
MA: The moving average component models the dependency between an observation and a residual error from a moving average model applied to lagged observations.
- The general form is: $X_{t} = θ_{1} ϵ_{t - 1} + θ_{2} ϵ_{t - 2} + \dots + θ_{q} ϵ_{t - q} + ϵ_{t}$ where $θ_{1}, θ_{2}, \dots, θ_{q}$ are the coefficients and $ϵ_{t}$ is white noise.
ARMA Model: The ARMA model is a combination of both AR and MA models and is used when the time series data is stationary.

ARIMA (AutoRegressive Integrated Moving Average):

ARIMA is an extension of ARMA that also includes differencing to make the series stationary.

I: Stands for Integration, which means differencing the time series data to make it stationary.
The general form of an ARIMA model is: $(p, d, q)$ where:
- $p$ : The order of the AR term.
- $d$ : The degree of differencing.
- $q$ : The order of the MA term.

Forecasting:

ARMA is used for stationary time series data (no trend or seasonality).
ARIMA is used for non-stationary data, where differencing is required to make the series stationary.

Python Example (using `statsmodels`):

from statsmodels.tsa.arima.model import ARIMA

import pandas as pd

# Example time series data

data = pd.Series([10, 12, 14, 16, 18, 20, 22, 24, 26, 28])

# ARIMA Model (p=1, d=1, q=1)

model = ARIMA(data, order=(1, 1, 1))

model_fit = model.fit()

# Forecasting

forecast = model_fit.forecast(steps=5)

print("Forecasted values:", forecast)

3.Illustrate the Z-Test & T-Test in Exponential Hypothesis.

Z-Test:

The Z-test is used to determine if there is a significant difference between the sample mean and the population mean when the sample size is large (typically $n > 30$ ) or when the population variance is known.

Z-statistic is given by: $Z = \frac{\overset{ˉ}{X} - μ}{\frac{σ}{\sqrt{n}}}$ where:
- $\overset{ˉ}{X}$ is the sample mean.
- $μ$ is the population mean.
- $σ$ is the population standard deviation.
- $n$ is the sample size.

T-Test:

The T-test is used when the sample size is small ( $n < 30$ ) or when the population variance is unknown.

T-statistic is given by: $T = \frac{\overset{ˉ}{X} - μ}{\frac{s}{\sqrt{n}}}$
- $s$ is the sample standard deviation.

Both tests are used to test the hypothesis about population parameters, such as mean or variance.

Exponential Hypothesis: When testing the hypothesis about an exponential distribution, the tests would check if the observed data follows an exponential distribution with a given rate parameter.

4.Explain the Time series forecasting with AR & MA.

Time series forecasting involves predicting future values of a series based on its past values. The AR (AutoRegressive)and MA (Moving Average) models are two popular approaches used for forecasting time series data.

AR (AutoRegressive) Model:

The AR model assumes that the current value of the time series is linearly dependent on its previous values. It is based on the principle that the past values have a strong relationship with the present value.

AR(p): In an AR model of order p, the value at time $t$ $t$ , denoted as $X_{t}$ $X_{t}$ , is modeled as a linear combination of the previous p values of the series. The formula is given by: $X_{t} = ϕ_{1} X_{t - 1} + ϕ_{2} X_{t - 2} + \dots + ϕ_{p} X_{t - p} + ϵ_{t}$ where:
- $X_{t}$ is the current value of the time series.
- $ϕ_{1}, ϕ_{2}, \dots, ϕ_{p}$ are the AR coefficients.
- $ϵ_{t}$ is the white noise or error term at time $t.p$
- $p$ is the order of the AR model (the number of past values used for prediction).

MA (Moving Average) Model:

The MA model assumes that the current value of the time series is linearly dependent on the past errors (residuals). The errors are modeled as a linear combination of past error terms.

MA(q): In an MA model of order q, the value at time $t$ $t$ , denoted as $X_{t}$ $X_{t}$ , is given by: $X_{t} = ϵ_{t} + θ_{1} ϵ_{t - 1} + θ_{2} ϵ_{t - 2} + \dots + θ_{q} ϵ_{t - q}$ where:
- $X_{t}$ is the observed value.
- $θ_{1}, θ_{2}, \dots, θ_{q}$ are the coefficients of the MA model.
- $ϵ_{t}$ is the white noise or error term at time $t$ .
- $q$ is the order of the MA model (the number of past errors).

AR vs MA:

The AR model is based on the time series' past values, while the MA model is based on past error terms.
AR is suitable when past values can explain future values, and MA is suitable when the future value is influenced by the errors from the past predictions.

Combining AR and MA - ARMA:

An ARMA (AutoRegressive Moving Average) model combines both AR and MA components to capture both the relationship between past values and past errors.

The ARMA model of order (p, q) is given by: $X_{t} = ϕ_{1} X_{t - 1} + \dots + ϕ_{p} X_{t - p} + ϵ_{t} + θ_{1} ϵ_{t - 1} + \dots + θ_{q} ϵ_{t - q}$ where p is the order of the AR part, and q is the order of the MA part.

Example of ARMA Model:

5.Select and apply the appropriate function in python to perform downsampling.

Downsampling in time series refers to reducing the frequency of the data by aggregating data over specific intervals. It is typically done when you have high-frequency data (e.g., minute-level) and want to reduce it to lower frequency (e.g., hourly or daily).

In Python, the `pandas` library provides convenient functions to perform downsampling.

resample() function in pandas is commonly used to downsample time series data.

Steps in Downsampling:

Resample: Choose a new time frequency, such as converting daily data to monthly data, or minute-level data to hourly data.
Aggregate: Apply an aggregation function like mean, sum, or median to the downsampled data.

Example of Downsampling:

import pandas as pd

# Example time series data with minute frequency

data = {'timestamp': pd.date_range('2023-01-01', periods=10, freq='T'),

'value': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}

df = pd.DataFrame(data)

df.set_index('timestamp', inplace=True)

# Downsampling to hourly frequency, using the 'mean' aggregation

downsampled_df = df.resample('H').mean()

print(downsampled_df)

In this example:

resample('H') resamples the data to hourly frequency.
mean() calculates the mean of values in each hour.

Common Downsampling Frequencies:

'D': Day
'W': Week
'M': Month
'H': Hour
'T': Minute

6) Develop a Program in Python to Decompose a Time Series Into Its Components

Time series decomposition involves breaking down a time series into its main components: trend, seasonality, and residual (noise). This is helpful to understand the underlying patterns in the data.
Decomposition using statsmodels:
The statsmodels.tsa.seasonal module provides a function called seasonal_decompose to decompose a time series.
Steps in Decomposition:

Trend: The long-term movement in the data.

Seasonality: Regular repeating patterns over fixed periods.

Residual: The noise or irregular component.

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

from statsmodels.tsa.seasonal import seasonal_decompose

# Generate a sample time series with trend and seasonality

np.random.seed(0)

time = pd.date_range('2020-01-01', periods=100, freq='D')

trend = np.linspace(10, 50, 100) # Linear trend

seasonality = 10 * np.sin(np.linspace(0, 2 * np.pi, 100)) # Seasonal component

residual = np.random.normal(0, 2, 100) # Random noise

data = trend + seasonality + residual

# Create DataFrame

df = pd.DataFrame(data, index=time, columns=['value'])

# Decompose the time series

decomposition = seasonal_decompose(df['value'], model='additive', period=30)

# Plot the decomposition

decomposition.plot()

plt.show()

7) Apply Functions in Python to Develop an AutoRegressive (AR) Model

The AutoRegressive (AR) model is used to forecast future values of a time series based on its past values. In Python, we can use statsmodels to implement AR models.

Steps to Create an AR Model:

Data Preparation: Import and format the time series data.
Model Creation: Use the AutoReg class from statsmodels.
Model Fit: Fit the AR model to the data and use it for forecasting.

Example of AR Model:

from statsmodels.tsa.ar_model import AutoReg

import pandas as pd

# Example time series data (monthly sales)

data = [10, 12, 14, 16, 18, 20, 22, 24, 26, 28]

df = pd.Series(data)

# Create an AutoReg model (p=3, meaning using 3 previous lags)

model = AutoReg(df, lags=3)

model_fit = model.fit()

# Forecast the next 5 values

forecast = model_fit.predict(start=len(df), end=len(df)+4)

print("Forecasted values:", forecast)

UNIT-4

PART-A

1.Dissect CSV.

A CSV (Comma Separated Values) file is a common data format used to store tabular data. Each row represents a record, and each value within a row is separated by a delimiter (commonly a comma). In Python, we can use libraries like pandas or csv to dissect and analyze CSV files.

2) List the Needs of Data Pre-Processing

Data preprocessing is a critical step in the machine learning pipeline that ensures the quality of the data before feeding it into a model. The goal is to prepare the data in such a way that the model can learn patterns and relationships without being biased by irrelevant or flawed information.

Summary of Preprocessing Needs:

Cleaning: Handle missing values, duplicates, and outliers.
Transformation: Normalize/standardize data, encode categorical variables, process dates.
Feature Engineering: Create new features and select important ones.
Splitting: Split data into training and testing sets.
Handling Imbalance: Address class imbalances using resampling.
Integration: Merge datasets from multiple sources.
Reduction: Apply dimensionality reduction techniques

3) Examine Python with MongoDB

Python can interact with MongoDB using the pymongo library. MongoDB is a NoSQL database that stores data in a flexible, JSON-like format called BSON. To use MongoDB in Python, you typically follow these steps:

Install pymongo: pip install pymongo
Connect to the MongoDB server
Perform database operations such as insert, query, update, and delete.

4) Dissect JSON

JSON (JavaScript Object Notation) is a lightweight data format used for data interchange. It represents data as key-value pairs, similar to dictionaries in Python. To "dissect" JSON in Python means parsing and accessing the values stored in a JSON string or file.

5) Inspect Rollback Operation

A rollback is used in database management to undo changes that were made during a transaction. In Python, rollback operations are typically handled using transaction management features in SQL databases. While MongoDB is not transactional in the traditional sense (before version 4.0), some databases like MySQL or PostgreSQL support explicit rollbacks in transactions.

5) Identify the Function to Read from a CSV File

In Python, the pandas library is widely used for reading CSV files. The function to read a CSV file is pd.read_csv().

import pandas as pd

# Read CSV file into a DataFrame

df = pd.read_csv("data.csv")

# Display the first few rows

print(df.head())

6) Illustrate the Use of Dialect Parameter in CSV File

The dialect parameter in Python's csv module defines the formatting rules of the CSV file. It can specify delimiters, quoting characters, and line terminators, among other options. You can define your own dialect using the csv.register_dialect() function.

import csv

# Register a custom dialect

csv.register_dialect('my_dialect', delimiter=';', quoting=csv.QUOTE_ALL)

# Open and read a CSV using the custom dialect

with open('data.csv', 'r') as file:

reader = csv.reader(file, dialect='my_dialect')

for row in reader:

print(row)

7) Discuss the Key Characteristics of a JSON File Format

JSON (JavaScript Object Notation) is widely used for representing structured data. Some key characteristics include:

Human-readable: JSON is text-based and easy for humans to read and write.
Data Representation: Data is represented as key-value pairs (objects), and values can be strings, numbers, arrays, or other objects.
Lightweight: JSON is a compact format, making it efficient for data exchange.
Language-Independent: JSON is language-agnostic and can be used across different programming languages.

Extensible: JSON can be easily extended to represent complex nested structure

8) State Any Two Differences Between SQL and NoSQL Databases

SQL and NoSQL databases have distinct characteristics, and here are two key differences:

Data Model:
- SQL: Uses a structured schema with tables, rows, and columns. Data must fit into a predefined schema (relational model).
- NoSQL: Flexible data model, often document-based (e.g., MongoDB) or key-value pairs (e.g., Redis), and does not require a fixed schema.
Scalability:
- SQL: Vertical scaling, where you increase the capacity of a single server by adding more CPU, RAM, or storage.
- NoSQL: Horizontal scaling, where you add more servers to distribute the load, making NoSQL databases more suitable for large-scale applications.

UNIT-4

PART-B

1) Simplify About JSON with a Sample Program

JSON (JavaScript Object Notation) is a lightweight, text-based data format that is used to store and exchange data. It represents data in key-value pairs, and it is language-independent but widely used in web development.

{

"name": "John Doe",

"age": 30,

"city": "New York"

}

In Python, the json module is used to work with JSON data.

Python Program to Work with JSON:

import json

# Sample JSON string

json_data = '{"name": "John Doe", "age": 30, "city": "New York"}'

# Parse JSON string into Python dictionary

data = json.loads(json_data)

# Access data from the dictionary

print("Name:", data["name"])

print("Age:", data["age"])

print("City:", data["city"])

# Convert Python dictionary back to JSON

json_string = json.dumps(data)

print("JSON String:", json_string)

2) Discover the Way of Creating a Database Table in SQL

In SQL, creating a database table involves using the CREATE TABLE statement. You define the table name, the columns, and the data types of those columns.

SQL Query to Create a Table:

CREATE TABLE Employees (

EmployeeID INT PRIMARY KEY,

Name VARCHAR(100),

Age INT,

Department VARCHAR(50)

);

3) Examine Normalizing Dataset

Normalization is a technique used to adjust the values in a dataset to a common scale without distorting differences in the ranges of values. It is particularly useful for algorithms that depend on the magnitude of features, like k-nearest neighbors and gradient descent.

Methods of Normalization:

Min-Max Normalization: Rescales the data to a fixed range, usually [0, 1].
- Formula:
  $Normalized Value = \frac{X - min(X)}{max(X) - min(X)}$
Z-Score Normalization (Standardization): Centers the data around 0 with a standard deviation of 1.
- Formula:
  $Z-Score = \frac{X - μ}{σ}$
- Where μ is the mean and σ is the standard deviation.

Example in Python (Using `pandas` and `sklearn`):

import pandas as pd

from sklearn.preprocessing import MinMaxScaler, StandardScaler

# Sample Data

data = {'Age': [22, 45, 38, 50, 30]}

df = pd.DataFrame(data)

# Min-Max Normalization

scaler = MinMaxScaler()

df['Age_normalized'] = scaler.fit_transform(df[['Age']])

# Z-Score Normalization

std_scaler = StandardScaler()

df['Age_zscore'] = std_scaler.fit_transform(df[['Age']])

print(df)

4) Examine About Binarize Data

Binarization is a technique used to convert data into a binary format. It is commonly used in machine learning to convert continuous variables into binary (0 or 1) based on a threshold value. This is useful when you need to categorize data into two groups (e.g., Yes/No, True/False).

Example:

Convert age to 1 if age is above 30, otherwise 0.

Python Example using `Binarizer` from `sklearn`:

from sklearn.preprocessing import Binarizer

# Sample data

data = {'Age': [22, 45, 38, 50, 30]}

df = pd.DataFrame(data)

# Binarize data based on a threshold (e.g., threshold of 30)

binarizer = Binarizer(threshold=30)

df['Age_binarized'] = binarizer.fit_transform(df[['Age']])

print(df)

5) Identify the Function to Connect with MongoDB and Explain the Same

To connect with MongoDB in Python, you use the MongoClient from the pymongo library. This allows you to interact with the MongoDB database.

Python Code to Connect to MongoDB:

from pymongo import MongoClient

# Connect to MongoDB server (local in this case)

client = MongoClient("mongodb://localhost:27017/")

# Access a specific database

db = client['test_db']

# Access a specific collection

collection = db['test_collection']

print("Connected to MongoDB!")

*) Differentiate MySQL and MongoDB Data

MySQL (Relational Database) and MongoDB (NoSQL Database) have several differences:

Data Structure:
- MySQL: Stores data in tables with rows and columns (structured schema).
- MongoDB: Stores data in collections as documents in BSON format (flexible schema).
Scalability:
- MySQL: Typically scales vertically (upgrading hardware).
- MongoDB: Scales horizontally (adding more servers).
Query Language:
- MySQL: Uses SQL (Structured Query Language) for querying.
- MongoDB: Uses MongoDB Query Language (MQL), which is JavaScript-like.
Transactions:
- MySQL: Supports ACID-compliant transactions.
- MongoDB: Supports transactions but with some limitations, especially before version 4.0.
Use Cases:

MySQL: Ideal for structured data with relationships.
MongoDB: Ideal for unstructured or semi-structured data and big data applications.

UNIT-4

PART-C

1.Analyse about interacting with Data in NoSQL.

NoSQL databases are designed to handle a wide variety of data types and structures. Unlike SQL databases, which rely on structured tables and schemas, NoSQL databases are schema-less, offering greater flexibility for developers to store unstructured or semi-structured data.

Key Characteristics:

Data Storage Formats: Common formats include documents (MongoDB), key-value pairs (Redis), wide-column stores (Cassandra), and graph databases (Neo4j).
Scalability: NoSQL databases are horizontally scalable, meaning you can add more servers to handle increased data loads.
Flexibility: NoSQL allows for rapid development and iteration since the schema can adapt to changing data needs.
Use Cases: Real-time analytics, IoT applications, social media platforms, and any application requiring high write speeds or flexible schemas.

Example (Python and MongoDB):

from pymongo import MongoClient

# Connect to MongoDB

client = MongoClient("mongodb://localhost:27017/")

db = client["example_db"]

collection = db["example_collection"]

# Insert a document

collection.insert_one({"name": "Alice", "age": 25, "city": "New York"})

# Retrieve documents

for doc in collection.find():

print(doc)

2.Analyse about interacting with Data in SQL

SQL databases are ideal for structured data stored in tabular formats. They enforce strict schema rules, making them suitable for applications requiring consistent data integrity.

Key Characteristics:

Data Storage Formats: Relational tables with rows and columns, defined by schemas.
ACID Compliance: Ensures atomicity, consistency, isolation, and durability, making them reliable for financial or transactional systems.
Querying: Use SQL for complex queries, joins, aggregations, and updates.
Scalability: Generally vertically scalable, requiring hardware upgrades for increased capacity.

Example (Python and MySQL):

import mysql.connector

# Connect to MySQL

connection = mysql.connector.connect(

host="localhost",

user="root",

password="password",

database="example_db"

)

cursor = connection.cursor()

# Insert data

cursor.execute("INSERT INTO users (name, age, city) VALUES ('Bob', 30, 'Boston')")

connection.commit()

# Retrieve data

cursor.execute("SELECT * FROM users")

for row in cursor.fetchall():

print(row)

connection.close()

3.Analyse the advantages of CSV file format and apply functions in python to

interact with CSV files.

CSV (Comma-Separated Values) is a simple, widely-used file format for storing tabular data.

Advantages:

Simplicity: Easy to read and understand.
Compatibility: Supported by almost all software, from spreadsheets to databases.
Lightweight: Plain text format with minimal overhead.
Portability: Platform-independent and suitable for data exchange.
Human-Readable: The content is easily accessible and editable.

Python Functions to Interact with CSV Files:

csv.reader(): Reads CSV files row by row.
csv.writer(): Writes data to CSV files.
pandas.read_csv(): Reads CSV into a DataFrame for data analysis.
pandas.to_csv(): Writes DataFrame to a CSV file.

Example with csv.reader:

import csv

# Read from a CSV file

with open('example.csv', 'r') as file:

reader = csv.reader(file)

for row in reader:

print(row)

Example with Pandas:

import pandas as pd

# Read CSV

df = pd.read_csv("example.csv")

print(df.head())

# Write to CSV

df.to_csv("output.csv", index=False)

4.List the advantages of JSON file format and apply functions in python to interact

with JSON files.

JSON (JavaScript Object Notation) is a popular data interchange format.

Advantages:

Readability: Human-readable and easy to edit.
Compatibility: Works seamlessly with APIs and web applications.
Lightweight: Minimal syntax overhead.
Hierarchical Structure: Ideal for nested or hierarchical data.
Language-Independent: Supported by almost all modern programming languages.

Python Functions for JSON:

json.load(): Reads JSON from a file.
json.dump(): Writes Python objects to a JSON file.
json.loads(): Converts a JSON string to a Python dictionary.
json.dumps(): Converts Python objects to a JSON string.

Example:

import json

# Write JSON data

data = {"name": "Alice", "age": 25, "city": "New York"}

with open("data.json", "w") as file:

json.dump(data, file)

# Read JSON data

with open("data.json", "r") as file:

content = json.load(file)

print(content)

5.Identify the functions in python to perform basic operations with MySQL

database.

Python’s mysql.connector library provides functions to interact with MySQL databases.

Common Functions:

connect(): Establishes a connection to the MySQL database.
cursor(): Creates a cursor object to execute queries.
execute(): Executes a SQL query.
fetchall(): Retrieves all rows from the last executed query.
commit(): Saves changes to the database.
close(): Closes the connection.

Example:

import mysql.connector

connection = mysql.connector.connect(

host="localhost",

user="root",

password="password",

database="example_db"

)

cursor = connection.cursor()

# Create a table

cursor.execute("CREATE TABLE IF NOT EXISTS users (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(255), age INT)")

# Insert data

cursor.execute("INSERT INTO users (name, age) VALUES ('Alice', 25)")

connection.commit()

# Fetch data

cursor.execute("SELECT * FROM users")

print(cursor.fetchall())

connection.close()

6.Develop a program in python to interact with a MongoDB database and perform

basic operations.

from pymongo import MongoClient

# Connect to MongoDB

client = MongoClient("mongodb://localhost:27017/")

db = client["example_db"]

collection = db["example_collection"]

# Insert data

collection.insert_one({"name": "Alice", "age": 25})

# Retrieve data

for document in collection.find():

print(document)

# Update data

collection.update_one({"name": "Alice"}, {"$set": {"age": 26}})

# Delete data

collection.delete_one({"name": "Alice"})

7.Construct a program in python to read from a CSV file using csv.reader().

import csv

# Open and read CSV file

with open("example.csv", "r") as file:

reader = csv.reader(file)

for row in reader:

print(row)

UNIT-5

PART-A

1.Inspect Filtering Data.

Filtering data is the process of isolating specific subsets of data based on given conditions or criteria. It is an essential step in data preprocessing and analysis to remove irrelevant or noisy information.

Applications:

Data Cleaning: Remove invalid or duplicate entries.
Focused Analysis: Work on subsets of data relevant to the task.
Improved Performance: Reduce the dataset size for faster computations.

2.Simplify the term Manipulating.

Manipulating refers to changing, modifying, or restructuring data to make it suitable for analysis. This can involve tasks like adding new columns, renaming columns, altering values, or reshaping data.

Common Data Manipulations:

Adding or removing rows and columns.
Replacing missing or erroneous values.
Converting data types or formats.
Applying functions to transform data.

3.Inspect the Grouping

Grouping refers to aggregating data based on a specific feature or category. It helps summarize data and uncover patterns by applying aggregation functions like sum, mean, or count.

Applications:

Summarizing large datasets.
Analyzing trends within categories.
Data visualization and reporting.

4. Discover to Write a CSV File

Writing a CSV file involves saving data in a tabular format using a delimiter (e.g., commas). Python provides libraries like csv and pandas to write data to CSV files.

5.Simplify on Sorting.

Sorting arranges data in a specific order, either ascending or descending, based on one or more criteria. Sorting helps organize data for better readability and analysis.

6.List types of Data Mining.

Classification: Assigns data to predefined categories (e.g., spam detection).

Clustering: Groups similar data points together (e.g., customer segmentation).

Regression: Predicts continuous values (e.g., sales forecasting).

Association Rule Mining: Identifies relationships between variables (e.g., market basket analysis).

Anomaly Detection: Finds unusual patterns or outliers (e.g., fraud detection).

Text Mining: Analyzes textual data (e.g., sentiment analysis).

Time-Series Analysis: Analyzes trends over time (e.g., stock market predictions).

7 Illustrate the effect of outliers in data mining

Outliers are extreme values that differ significantly from the majority of data. While they may represent errors or rare events, they can have a significant impact on analysis.

Effects of Outliers:

Skewed Results: Can distort statistical measures like mean and standard deviation.
Misleading Models: Influence machine learning models, leading to poor performance.
Increased Variability: Affect clustering and regression accuracy.

8 What are the types of imputation?

Imputation is the process of replacing missing data with substituted values.

Types of Imputation:

Mean Imputation: Replace missing values with the mean of the available data.
Median Imputation: Use the median value, robust against outliers.
Mode Imputation: Replace with the most frequent value, useful for categorical data.
K-Nearest Neighbors (KNN): Predict missing values based on similarity with other data points.
Regression Imputation: Use a regression model to predict missing values.
Multiple Imputation: Generate multiple imputed datasets and combine results for robust analysis.

9. Illustrate the various steps in data mining.

Data mining is the process of discovering patterns and insights from large datasets. The key steps are:

Data Collection: Gather data from various sources like databases, files, or sensors.
Data Cleaning: Handle missing values, remove duplicates, and correct inconsistencies.
Data Integration: Combine data from multiple sources into a single, unified dataset.
Data Transformation: Convert data into the required format, normalize it, or scale values.
Data Reduction: Reduce the size of the dataset by feature selection, aggregation, or sampling.
Data Mining: Apply algorithms like classification, clustering, or association rule mining to find patterns.
Pattern Evaluation: Assess the discovered patterns for usefulness and accuracy.
Knowledge Presentation: Visualize and present the results using graphs, charts, or reports.

10.Classify the types of encoding used in transforming categorical attributes.

Label Encoding: Assigns each unique category a number (e.g., "Red" → 0, "Blue" → 1).

One-Hot Encoding: Creates binary columns for each category (e.g., "Red" → [1, 0, 0], "Blue" → [0, 1, 0]).

Ordinal Encoding: Assigns numerical values to categories based on a defined order (e.g., "Low" → 1, "Medium" → 2, "High" → 3).

Binary Encoding: Converts categories into binary numbers and represents them as columns.

Frequency Encoding: Encodes categories based on their frequency in the dataset.

Target Encoding: Replaces categories with the mean of the target variable for each category.

UNIT-5

PART-B

1.Function a program to handle CSV files using pandas library.

1. Function a program to handle CSV files using pandas library

Detailed Answer:
The `pandas` library is a powerful Python library for data manipulation and analysis. One of its most widely used features is handling CSV files, which are commonly used for storing and exchanging data in tabular format.
To handle CSV files:
Reading Data: Use the `read_csv()` method to load the CSV data into a pandas DataFrame. This function supports multiple parameters for customization, such as specifying column names, handling missing values, or skipping rows.
Exploring Data: Once loaded, methods like `.head()`, `.tail()`, and `.info()` can provide a quick overview of the dataset.
Writing Data: The `to_csv()` method writes a pandas DataFrame back to a CSV file.
Example Code:
import pandas as pd

# Reading a CSV file
data = pd.read_csv('sample.csv')
print("Preview of the Data:\n", data.head())

# Describing the dataset to understand its structure
print("Dataset Information:\n", data.info())

# Writing modified data to a new CSV file
data['New_Column'] = data['Existing_Column'] * 2 # Example modification
data.to_csv('output.csv', index=False)
print("Modified data has been written to 'output.csv'.")

2.Function a program to retrieve data using Python.

Detailed Answer:

Retrieving specific data is a key task in data analysis. With pandas, you can filter rows, select columns, or retrieve subsets of the dataset based on conditions.

Steps to Retrieve Data:

Select Columns: Use data[['Column1', 'Column2']] to extract specific columns.
Retrieve Rows: Use .iloc[] for positional indexing or .loc[] for label-based indexing.
Apply Conditions: Filter rows using conditional statements.

Example Code:

import pandas as pd

# Sample DataFrame

data = pd.DataFrame({

'Name': ['Alice', 'Bob', 'Charlie'],

'Age': [25, 30, 35],

'Score': [80, 90, 85]

})

# Retrieving specific columns

selected_columns = data[['Name', 'Score']]

print("Selected Columns:\n", selected_columns)

# Retrieving rows based on conditions

filtered_data = data[data['Age'] > 25]

print("Rows where Age > 25:\n", filtered_data)

# Retrieving a specific row by index

row = data.iloc[1]

print("Second Row:\n", row)

3.Classify a note on data pre-processing.

Detailed Answer:

Data pre-processing is a critical step in the data analysis pipeline. It involves cleaning and transforming raw data to improve its quality and make it suitable for analysis or modeling. Poor-quality data can lead to inaccurate results, so pre-processing ensures the dataset is ready for further operations.

Steps in Data Pre-processing:

Handling Missing Values:
- Replace missing values with mean, median, or mode.
- Remove rows or columns with too many missing values.
Normalization:
- Scale numerical data to a standard range (e.g., 0 to 1) to ensure consistency in the dataset.
Encoding Categorical Data:
- Convert non-numerical labels into numerical values using methods like one-hot encoding.
Feature Scaling:
- Techniques like standardization or min-max scaling ensure that all features contribute equally to the model.

Example Code:

import pandas as pd

from sklearn.preprocessing import MinMaxScaler, LabelEncoder

# Sample dataset

data = pd.DataFrame({

'Age': [25, 30, None, 35],

'Salary': [50000, None, 60000, 70000],

'Gender': ['Male', 'Female', 'Female', 'Male']

})

# Handling missing values

data['Age'].fillna(data['Age'].mean(), inplace=True)

data['Salary'].fillna(data['Salary'].median(), inplace=True)

# Normalization

scaler = MinMaxScaler()

data[['Age', 'Salary']] = scaler.fit_transform(data[['Age', 'Salary']])

# Encoding categorical data

encoder = LabelEncoder()

data['Gender'] = encoder.fit_transform(data['Gender'])

print("Pre-processed Data:\n", data)

4.Examine about filtering data.

Detailed Answer:

Filtering data involves selecting specific rows or columns from a dataset based on certain conditions. This is essential when working with large datasets where you need to focus on a subset of data.

Why Filtering is Important?

Focuses analysis on relevant data.
Reduces the dataset size for faster computation.
Helps in exploratory data analysis by isolating patterns.

Techniques for Filtering:

Using Conditions: Filter rows that meet a specific criterion.
Using Indexing: Select subsets of data based on labels or positions.
Combining Filters: Use logical operators (&, |) for multiple conditions.

Example Code:

import pandas as pd

# Sample dataset

data = pd.DataFrame({

'Name': ['Alice', 'Bob', 'Charlie', 'Diana'],

'Age': [25, 30, 35, 40],

'Score': [80, 85, 90, 95]

})

# Filter rows where Age > 30

filtered_data = data[data['Age'] > 30]

print("Filtered Rows (Age > 30):\n", filtered_data)

# Filter rows where Score is between 80 and 90

filtered_data = data[(data['Score'] >= 80) & (data['Score'] <= 90)]

print("Filtered Rows (Score 80-90):\n", filtered_data)

5.Apply Binarization to rescale the data.

Binarization is the process of transforming numerical data into binary values (0 or 1) based on a threshold. It’s a form of discretization and is particularly useful for simplifying data for certain machine-learning models, like logistic regression.

Steps in Binarization:

Identify the feature(s) that need to be binarized.
Define a threshold value. All values above the threshold are converted to 1, and those below or equal to the threshold are converted to 0.
Use libraries like sklearn for easy implementation.

from sklearn.preprocessing import Binarizer

# Sample dataset

data = [[1.2, -0.5, 3.7, 0.0]]

# Define the binarizer with a threshold

binarizer = Binarizer(threshold=0.5)

# Apply binarization

binary_data = binarizer.fit_transform(data)

print("Original Data:", data)

print("Binarized Data:\n", binary_data)

6.Apply methods in Python to split attributes into dependent and independent attributes.

Detailed Answer:

In machine learning, data is divided into:

Independent Attributes (Features): The input data used for predictions.
Dependent Attribute (Target/Label): The output we want to predict.

Steps to Split Attributes:

Identify the dependent and independent variables.
Use pandas to separate these into different variables or arrays.

import pandas as pd

# Sample dataset

data = pd.DataFrame({

'Height': [5.0, 5.5, 6.0, 6.2],

'Weight': [50, 60, 65, 70],

'Gender': [0, 1, 1, 0] # 0 for Female, 1 for Male

})

# Independent attributes

X = data[['Height', 'Weight']]

# Dependent attribute

y = data['Gender']

print("Independent Attributes (Features):\n", X)

print("Dependent Attribute (Target):\n", y)

7.Analyse the techniques in Python to handle missing values.

Detailed Answer:

Handling missing data is crucial to prevent biased results or errors during data analysis or model training. Several strategies can be employed based on the dataset and the type of missing data.

Common Techniques:

Remove Missing Values:
- Use .dropna() to remove rows or columns containing missing data.
Replace Missing Values (Imputation):
- Replace missing data with statistical measures like mean, median, or mode.
- Use predictive modeling for advanced imputation.
Forward or Backward Fill:
- Fill missing values using adjacent values in time-series data.
Interpolate:

Estimate missing values using interpolation.

import pandas as pd

import numpy as np

# Sample dataset with missing values

data = pd.DataFrame({

'Age': [25, 30, None, 35],

'Salary': [50000, None, 60000, 70000]

})

# Dropping missing values

data_dropped = data.dropna()

print("Data After Dropping Missing Values:\n", data_dropped)

# Replacing missing values with mean

data['Age'].fillna(data['Age'].mean(), inplace=True)

data['Salary'].fillna(data['Salary'].median(), inplace=True)

print("\nData After Filling Missing Values:\n", data)

8.Identify a function to split a dataset into training and test data.

Detailed Answer:

Splitting a dataset into training and testing sets is essential for evaluating the performance of a machine learning model. The training set is used to train the model, while the testing set evaluates its performance.

Function: The train_test_split function from sklearn is commonly used for this purpose.

Parameters:

test_size: Fraction of the data used for testing.
random_state: Ensures reproducibility by fixing the random seed.

Example Code:

from sklearn.model_selection import train_test_split

# Sample dataset

X = [[1], [2], [3], [4], [5]] # Features

y = [10, 20, 30, 40, 50] # Target

# Splitting data into training and test sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print("Training Features:", X_train)

print("Test Features:", X_test)

print("Training Labels:", y_train)

print("Test Labels:", y_test)

9.Discuss data cleaning.

Detailed Answer:

Data cleaning is the process of identifying and rectifying errors or inconsistencies in raw data to ensure it is accurate, complete, and reliable for analysis or modeling. It is one of the most time-consuming yet crucial steps in data preprocessing.

Steps in Data Cleaning:

Identify Missing Data:
- Use .isnull() to check for missing values.
Handle Missing Data:
- Remove rows or columns with excessive missing values.
- Impute missing data with appropriate strategies.
Remove Duplicates:
- Identify and remove duplicate entries using .drop_duplicates().
Handle Outliers:
- Use statistical techniques (e.g., Z-score, IQR) to detect and treat outliers.
Standardize Data:
- Ensure data is formatted consistently (e.g., date formats, units).

Example Code:

import pandas as pd

import numpy as np

# Sample dataset with missing values

data = pd.DataFrame({

'Age': [25, 30, None, 35],

'Salary': [50000, None, 60000, 70000]

})

# Dropping missing values

data_dropped = data.dropna()

print("Data After Dropping Missing Values:\n", data_dropped)

# Replacing missing values with mean

data['Age'].fillna(data['Age'].mean(), inplace=True)

data['Salary'].fillna(data['Salary'].median(), inplace=True)

print("\nData After Filling Missing Values:\n", data)

10.Develop a program to filter and replace missing values.

import pandas as pd

import numpy as np

# Sample dataset with missing values

data = pd.DataFrame({

'Name': ['Alice', 'Bob', 'Charlie', None],

'Age': [25, None, 30, 35],

'Score': [85, 90, None, 95]

})

print("Original Data:")

print(data)

# 1. Identify Missing Values

print("\nMissing Values Check:")

print(data.isnull())

# 2. Filter Rows with Missing Values

rows_with_missing = data[data.isnull().any(axis=1)]

print("\nRows with Missing Values:")

print(rows_with_missing)

# 3. Replace Missing Values

# Replace missing values in 'Name' column with 'Unknown'

data['Name'].fillna('Unknown', inplace=True)

# Replace missing values in 'Age' column with the mean of the column

data['Age'].fillna(data['Age'].mean(), inplace=True)

# Replace missing values in 'Score' column with the median of the column

data['Score'].fillna(data['Score'].median(), inplace=True)

print("\nData After Replacing Missing Values:")

print(data)

Explanation of the Program:

Original Data:
- A dataset is created with missing values (represented as None or NaN).
Identifying Missing Values:
- .isnull() is used to identify missing values in the dataset. It returns a Boolean DataFrame indicating Truewhere values are missing.
Filtering Rows with Missing Values:
- Rows containing any missing value are filtered using .any(axis=1).
Replacing Missing Values:
- Name Column: Missing values are replaced with the string 'Unknown'.
- Age Column: Missing values are replaced with the mean of the Age column.
- Score Column: Missing values are replaced with the median of the Score column.

UNIT-5

PART-C

1.Classify the four main steps for Data pre-processing in detail.
1. Classify the four main steps for Data pre-processing in detail.
Data pre-processing is crucial to ensure that the data used for modeling is clean, consistent, and suitable for the machine learning process. The four main steps involved in data pre-processing are:
a. Data Cleaning
Data cleaning involves handling missing values, removing duplicates, and correcting inconsistencies in the data. Techniques like imputation (filling missing values with mean, median, or mode), dropping rows or columns with missing data, or using algorithms that can handle missing data (e.g., tree-based models) are commonly used.
b. Data Transformation
Data transformation focuses on converting the data into a format suitable for analysis. This could involve normalization, standardization, or encoding categorical data. For example, numerical features may be scaled to a similar range using Min-Max scaling or Z-score normalization, and categorical variables may be converted into numerical form through methods like Label Encoding or One-Hot Encoding.
c. Data Reduction
Data reduction is a technique to reduce the number of variables or data points to make the analysis more efficient while retaining essential information. It involves dimensionality reduction methods like Principal Component Analysis (PCA) or feature selection techniques that remove irrelevant or redundant features.
d. Data Integration
Data integration involves combining data from multiple sources into a unified dataset. This is especially important in scenarios where data comes from heterogeneous sources. For example, integrating databases, flat files, and real-time data streams into a cohesive structure for analysis.

2.Analyse in detail about Data Mining.
Data mining is the process of discovering patterns, correlations, trends, and useful information from large datasets using methods at the intersection of machine learning, statistics, and database systems. It involves extracting meaningful insights from large volumes of raw data, which can be used for decision-making.
a. Data Mining Techniques
- Classification: This is used to predict the categorical class labels of new data points based on previously labeled data (e.g., spam email classification, medical diagnosis).
- Clustering: It groups a set of objects such that objects in the same group (or cluster) are more similar to each other than to those in other groups. K-means and hierarchical clustering are common methods.
- Association Rule Learning: This technique is used to discover interesting relationships or patterns between variables in large datasets (e.g., market basket analysis to find associations between products purchased together).
- Regression: It predicts a continuous outcome based on input features. It is widely used in forecasting and trend analysis.
- Anomaly Detection: Identifying unusual patterns or outliers in data that do not conform to expected behavior (e.g., fraud detection in financial transactions).
- Sequential Pattern Mining: Identifying regular sequences of events or patterns over time.
b. Data Mining Process
The typical data mining process involves:
- Data Collection: Gathering data from various sources.
- Data Cleaning and Preprocessing: Ensuring that the data is clean, relevant, and in a suitable format.
- Pattern Discovery: Applying algorithms to identify patterns, correlations, or trends.
- Evaluation and Interpretation: Analyzing the discovered patterns for usefulness and relevance.
- Deployment: Applying the results to real-world applications or decision-making processes.
c. Applications of Data Mining
- Healthcare: Predicting disease outbreaks, diagnosing illnesses, and improving patient care.
- Finance: Fraud detection, credit scoring, and investment analysis.
- Retail: Market basket analysis, customer segmentation, and inventory management.
- Marketing: Targeted advertising, customer relationship management (CRM), and campaign analysis.
3.Analyse the importance of preprocessing in machine learning.
Preprocessing is a critical step in machine learning because it directly affects the quality of the input data and, in turn, the performance of the model. Here's why preprocessing is essential:
a. Handling Missing Data
Machine learning algorithms cannot handle missing data well. Preprocessing steps like imputation or removing missing values ensure that the data fed into the model is complete, preventing errors or biased predictions.
b. Scaling and Normalization
Many machine learning algorithms, especially those based on distance (like K-nearest neighbors, and SVM), assume that the features are on the same scale. Normalization (scaling features to a specific range) and standardization (scaling features to have a mean of 0 and standard deviation of 1) are crucial to ensure that one feature doesn't dominate the others in terms of scale.
c. Encoding Categorical Variables
Machine learning models require numerical data, so categorical variables must be encoded into numerical formats. Techniques like one-hot encoding or label encoding allow the machine learning algorithms to work with categorical data effectively.
d. Reducing Dimensionality
High-dimensional datasets can lead to overfitting and increased computational cost. Dimensionality reduction techniques like PCA (Principal Component Analysis) can help reduce the number of features while retaining most of the information.
e. Feature Engineering
Preprocessing often involves creating new features or modifying existing ones to improve model performance. For example, creating interaction terms, polynomial features, or aggregating data can help uncover hidden patterns in the data.
f. Noise Reduction
Real-world data can be noisy. Preprocessing helps to remove or reduce noise, ensuring that the model learns from the most relevant features and avoids overfitting to irrelevant patterns.

4.Identify the functions in Python to perform grouping of data.
In Python, grouping of data can be done primarily using the pandas library. Some functions used for grouping data are:
a. groupby()
The groupby() function in pandas allows you to group data based on one or more columns. It is commonly used to split the data into groups, apply functions to each group, and then combine the results.
import pandas as pd
df = pd.DataFrame({
'Category': ['A', 'B', 'A', 'B', 'A'],
'Value': [10, 20, 30, 40, 50]
})
grouped = df.groupby('Category')
b. agg()
The agg() function is used to apply aggregation functions on grouped data. You can pass multiple aggregation functions like sum, mean, count, etc.

grouped.agg('sum')

c. pivot_table()
The pivot_table() function allows you to create a pivot table for data summarization. It's useful when you want to aggregate and reshape data.

df.pivot_table(values='Value', index='Category', aggfunc='sum')

d. value_counts()
The value_counts() function is used to count occurrences of unique values in a column.

df['Category'].value_counts()

5.Differentiate One Hot encoding and Label encoding.
Both One-Hot Encoding and Label Encoding are used to convert categorical variables into numerical format, but they work in different ways.
a. One-Hot Encoding
- Concept: One-Hot Encoding converts each categorical value into a new column, creating a binary vector where only the column corresponding to the category value is marked as 1, and all other columns are 0.
- Use Case: One-Hot Encoding is ideal when the categorical variable is nominal (no ordinal relationship).
- Example:
  For a column with values ['Red', 'Green', 'Blue'], One-Hot Encoding would produce:
Red Green Blue
1 0 0
0 1 0
0 0 1
b. Label Encoding
- Concept: Label Encoding assigns a unique integer to each category. It transforms each category label into a numerical value.
- Use Case: Label Encoding is ideal for ordinal data where the categories have a natural ordering (e.g., low, medium, high).
- Example:
  For a column with values ['Red', 'Green', 'Blue'], Label Encoding would produce:
Red -> 0
Green -> 1
Blue -> 2
6.Identify the functions in Python to perform data ranking.
Ranking data in Python can be done using pandas. Some useful functions are:
a. rank()
The rank() function assigns ranks to values in a column, with ties receiving the average rank.
df['Rank'] = df['Value'].rank()
b. sort_values()
This function can sort the data, and by sorting in descending order, you can also identify the top-ranked values.

df.sort_values(by='Value', ascending=False)

7.Classify the rescaling techniques in Python.
Rescaling techniques are used to scale the data into a specific range, which helps some machine learning models perform better. Common rescaling techniques include:
a. Min-Max Scaling
Min-Max Scaling transforms the data into a fixed range, typically [0, 1]. It is done using the formula:
X_scaled = (X - X.min()) / (X.max() - X.min())
b. Standardization (Z-Score Normalization)
Standardization rescales the data to have a mean of 0 and a standard deviation of 1. It is useful when the data has a Gaussian distribution.

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

c. Robust Scaling
Robust Scaling is based on the median and the interquartile range (IQR), making it robust to outliers

from sklearn.preprocessing import RobustScaler
scaler = RobustScaler()
X_scaled = scaler.fit_transform(X)

d. MaxAbs Scaling
MaxAbs Scaling scales the data by dividing by the maximum absolute value, ensuring that the data is within [-1, 1].

from sklearn.preprocessing import MaxAbsScaler
scaler = MaxAbsScaler()
X_scaled = scaler.fit_transform(X)

Menu

Data Science

1. Outline of NumPy

2. Creating 1D and 2D Arrays in NumPy

3. Outline of Pandas

5. Sketch about Data Science

6. Benefits of Data Science

7. Uses of Data Science

8. Types of Data

9. Operations using NumPy

Mathematical Operations:

Key Features of Pandas:

Purpose of a Box Plot:

. Core Features

. Key Functions

Finding the Transpose of a NumPy Array

Key Points

Explanation:

Explanation:

i) Illustrating Data Science and its Types

What is Data Science?

Types of Data Science

ii) Python Program to Plot a Scatter Plot with a Grid

Code Example

Explanation of the Code

SciPy: Overview

What is SciPy?

Key Features of SciPy:

Program to Calculate the Inverse and Pseudo-Inverse of a Matrix

Code Example

Explanation of the Code

Output Example

1. Measuring Central Tendency

Mean

Median

Mode

Normal Distribution

Poisson Distribution

Key Concepts of ANOVA

Steps to Perform ANOVA

Types of ANOVA

Python Implementation: One-Way ANOVA

Python Implementation: Two-Way ANOVA

Visualizing ANOVA Results

interpreting Results

Advantages of ANOVA

Disadvantages of ANOVA

When to Use ANOVA

One-Sample T-Test

Code for One-Sample T-Test:

Paired-Sample T-Test

Code for Paired-Sample T-Test:

1. Null Hypothesis (H₀)

2. Alternative Hypothesis (H₁ or Ha)

3. Test Statistic

4. Significance Level (α)

5. P-Value

6. Critical Value

7. Confidence Interval

8. Type I Error (α)

9. Type II Error (β)

10. Power of the Test

11. Sample Size (n)

Summary of Key Parameters:

1. Identify Downsampling

3. Classify AR & MA

4. Outline of Resampling

5. Short Notes on Upsampling

6. Classify ARMA & ARIMA

7. What is an Autocorrelation Function?

8. What are the Advantages of Time Series Forecasting?

9. State the Applications of Time Series Forecasting

10. Identify the Type of Data with Time Stamp and Explore its Uses

2. Explain About Resampling

3. Construct a Note on TimeDelta

4. Explain About Downsampling

5. Identify the Function to Calculate the Autocorrelation of a Time Series

8. State the Differences Between Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF)

9. Develop a Small Program to Print a Time Series

1. Upsampling with Polynomial Interpolation