Experimental Project One: Hadoop and Spark Setup and Application Development
Ⅰ Experimental principle
Apache Hadoop is a software framework that can perform distributed processing on large clusters with thousands of nodes and petabyte-level data. Users of Hadoop can quickly develop parallel applications, thus focusing on business logic without having to undertake heavy tasks such as distributing data, distributing code for parallel processing, and handling faults. Spark is a fast enterprise-level large-scale data processing engine. It provides API support for Hadoop and enables the development of Spark applications based on the Hadoop environment.
Ⅱ Teaching Requirements
Install the Hadoop and Spark environments on the virtual machine or Linux system, and start the daemon process. Use Spark to implement and run the WordCount program.
2.1. Students are required to be able to set up Hadoop and Spark environments.
2.2. Students are required to be able to start the Spark service process correctly.
2.3. Students are required to be able to implement Spark applications and run them correctly.
Ⅲ Experimental result
3.1. Demonstration of hadoop and spark environments
In the terminal, "java -version", "hadoop version", and "spark-submit --version" were executed to confirm that the environment variables and versions were correct.
3.1.1 Java version
Figure 1-1 Java version
3.1.2 Hadoop version
Figure 1-2 Hadoop version
3.1.3 Spark version
Figure 1-3 Spark version
3.1.4 Environment variable configuration
Figure 1-4 .bashrc file
3.1.5 View the Hadoop process with jps
After starting HDFS and YARN by executing 'start -DPs.sh' and 'start-yarn.sh', use jps to confirm that all major processes (NameNode, DataNode, ResourceManager, NodeManager) have been running.
Figure 1-5 jps
Figure 1-5 shows the NameNode, DataNode, ResourceManager, NodeManager and other processes viewed through jps after startup.
3.1.6 Access the Hadoop Web UI
Access 192.168.75.139:9870 in the browser, and you can see that the NameNode UI displays the HDFS status (such as formatted, running time, etc.) normally.
Figure 1-6 The status interface of the NameNode
Similarly, it indicates that the ResourceManager UI is operating normally
Figure 1-7 The interface of ResourceManager
3.2 HDFS formatting & Simple file operation verification
3.2.1 HDFS formatting
The hdfs namenode -format command was executed to initialize the HDFS file system of Hadoop. The output is as follows, indicating that the formatting was successful and a new cluster ID was generated
Figure 1-8 Formatting successful
3.2.2 Start the Hadoop daemon process
After starting the HDFS daemon, it can be seen using jps that the three processes, NameNode, DataNode, and SecondaryNameNode, are all running normally.
Figure 1-9 Start the Hadoop daemon process
3.2.3 Basic file operations of HDFS (Verifying the availability of HDFS)
To verify that HDFS can be used normally, the HDFS directory /user/rc was created and the local file testfile.txt was uploaded to prove that the file has been successfully stored.
Figure 1-10 Verifying the availability of HDFS
3.3 Run the Spark application: WordCount example
To verify whether Spark can run correctly, write and submit the WordCount program using PySpark. This program reads the text file /input/input.txt from HDFS, counts the word frequency and outputs the result. The running results are as follows, indicating that Spark successfully connected to Hadoop and completed distributed task computing
Figure 1-11 Prepare the test file and upload it to HDFS
Figure 1-12 Write the PySpark WordCount program
Figure 1-13 Output result example
Experimental Project Two: Implementing the linear regression algorithm with Spark MLlib
Ⅰ Experimental principle
Regression analysis is a statistical analysis method used to determine the quantitative relationship of interdependence among two or more variables and is widely applied. In MLlib, linear regression is a regression method that can predict specific data relatively accurately. It predicts unknown data with the help of a prediction algorithm through a given series of training data.
Ⅱ Teaching Requirements
The linear regression algorithm program is implemented using MLlib under Spark, and the input data set can be fitted to obtain the requirement regression formula. Verify the fitted curve.
2.1. Students are required to accurately understand the basic principles of the linear regression analysis algorithm
2.2. Students are required to be able to implement and run the basic linear regression algorithm using MLlib.
2.3. Students are required to be able to run the linear regression algorithm to obtain the fitting curve and conduct the fitting effect analysis
Ⅲ Experimental result
3.1. Dataset download
This step completes the download and placement operations of the real public dataset required for the experiment. By obtaining the Ames Housing dataset from GitHub and correctly saving it as the ames_iowa_housing.csv file to the local project directory, it is ensured that it can be successfully read by PySpark later. Meanwhile, the file integrity and encoding format were verified, and the loading failure problems caused by path errors or the default reading of HDFS were eliminated. This step provides standard and normative input data for the data analysis and modeling stage of the experiment, ensuring the reliability of the data source and the reproducibility of the experiment.
Figure 2-1 the examples of the first five lines
3.2. Data reading and initial cleaning
The following indicates that the dataset contains a total of 1,460 records and 81 fields (80 attributes + 1 target variable SalePrice). Next, the missing values and non-numerical features need to be cleaned and preprocessed to facilitate the subsequent model training.
3.2.1 Write the PySpark script data_read_clean.py to read CSV
# 文件名：data_read_clean.py
# 功能：读取 ames_iowa_housing.csv 并展示 Schema、示例数据、行列统计
from pyspark.sql import SparkSession
def main():
# 1. 创建 SparkSession
spark = SparkSession.builder \
.appName("Step3_ReadData") \
.getOrCreate()
# 2. 读取 CSV 文件（建议使用绝对路径或加 file://）
df = spark.read.csv("file:///home/hadoop/projects/linear_regression/ames_iowa_housing.csv",
header=True, inferSchema=True)
# 3. 输出数据结构（Schema）
print("=== 数据 Schema ===")
df.printSchema()
# 4. 展示前 5 行数据内容（全字段不截断）
print("=== 前 5 行数据示例 ===")
df.show(5, truncate=False)
# 5. 统计数据维度（行数与列数）
total_rows = df.count()
total_cols = len(df.columns)
print(f"=== 数据集维度 ===\n总行数: {total_rows}, 总列数: {total_cols}")
# 6. 关闭 SparkSession
spark.stop()
if __name__ == "__main__":
main()
3.2.2 Schema output (Field structure)
Figure 2-2 Part of the Schema output (field structure)
By analyzing the reading results of the data, it can be seen that Spark successfully identified the field types of each column in the dataset, including numerical types (such as LotArea, GrLivArea) and categorical types (such as MSZoning, Street, etc.). Meanwhile, the first five rows of sample data displayed indicate that the data content is complete. No garbled characters or reading errors occurred, and the field structure was consistent with the original CSV file. The data dimensions obtained by using count() and len(df.columns) are 1,460 records and 81 fields respectively, further indicating that the data scale is moderate and suitable for machine learning modeling and multi-feature analysis.
3.2.3 The first five lines of data are displayed
Figure 2-3 The first five lines of data are displayed
3.2.4 Data dimension statistics
Figure 2-4 Data dimension statistics
This 3.2 step aims to verify Spark's correct reading and preliminary parsing of the experimental data set, laying the foundation for subsequent analysis. Through automatic type inference (inferSchema), data preview (show), and dimension statistics (count, printSchema), the validity of data integrity and read parameters (such as header) was confirmed, and the rationality of the distributed environment configuration was verified at the same time. Its advantage lies in leveraging Spark's native capabilities to enhance reading efficiency and reduce the risk of manual intervention. It enables rapid analysis of data structures, samples, and scales through concise code, providing a basis for subsequent cleaning and feature engineering.
3.3. Missing value processing and category feature encoding
3.3.1 Overall code
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when, isnan, count
from pyspark.ml.feature import Imputer, StringIndexer, OneHotEncoder
def main():
spark = SparkSession.builder.appName("Step4_Preprocessing").getOrCreate()
# 1. 读取数据
df = spark.read.csv("file:///home/hadoop/projects/linear_regression/ames_iowa_housing.csv",
header=True, inferSchema=True)
# 2. 显式将 LotFrontage 转换为 double（防止类型推断失败）
df = df.withColumn("LotFrontage", col("LotFrontage").cast("double"))
# 3. 缺失值统计（处理前）
print("=== 缺失值统计（处理前） ===")
missing_counts = df.select([count(when(col(c).isNull() | isnan(col(c)), c)).alias(c) for c in df.columns])
missing_counts.show(truncate=False)
# 4. 用中位数填充 LotFrontage
imputer = Imputer(inputCols=["LotFrontage"], outputCols=["LotFrontage_imputed"]).setStrategy("median")
df = imputer.fit(df).transform(df).drop("LotFrontage").withColumnRenamed("LotFrontage_imputed", "LotFrontage")
# 5. 其他缺失值填充
df = df.fillna({
"GarageYrBlt": 0,
"MasVnrArea": 0,
"MasVnrType": "None",
"Electrical": "SBrkr"
})
# 6. 再次统计缺失值（处理后）
print("=== 缺失值统计（处理后） ===")
missing_counts_after = df.select([count(when(col(c).isNull() | isnan(col(c)), c)).alias(c) for c in df.columns])
missing_counts_after.show(truncate=False)
# 7. 类别特征编码
categorical_cols = ["Neighborhood", "HouseStyle", "ExterQual"]
index_cols = [c + "_Idx" for c in categorical_cols]
ohe_cols = [c + "_OH" for c in categorical_cols]
for c in categorical_cols:
indexer = StringIndexer(inputCol=c, outputCol=c + "_Idx", handleInvalid="keep")
df = indexer.fit(df).transform(df)
encoder = OneHotEncoder(inputCols=index_cols, outputCols=ohe_cols, handleInvalid="keep")
df = encoder.fit(df).transform(df)
df = df.drop(*categorical_cols).drop(*index_cols)
# 8. 显示结果
print("=== Schema ===")
df.printSchema()
print("=== 示例数据 ===")
df.select("LotFrontage", "GarageYrBlt", "MasVnrArea", "MasVnrType", "Electrical",
"Neighborhood_OH", "HouseStyle_OH", "ExterQual_OH", "SalePrice").show(5, truncate=False)
# 9. 保存结果
df.write.mode("overwrite").parquet("file:///home/hadoop/projects/linear_regression/ames_iowa_preprocessed.parquet")
spark.stop()
if __name__ == "__main__":
main()
The full processing of missing values in numerical and category fields is completed through Imputer (median filling) and fillna (zero value filling), effectively improving data integrity. For the three-column category features (such as Neighborhood), a new sparse vector format feature is generated through StringIndexer sequence number encoding and OneHotEncoder exclusive hot mapping. The Schema and the first five lines of examples verify that the encoding logic is consistent with the original data, meeting the input requirements of the model.
3.3.2 Statistics of Missing Values before Processing (Partial)
Figure 2-5 Statistics of Missing Values before Processing (Partial)
3.3.2 Statistics (confirmation) of missing values after processing
Figure 2-6 Statistics (confirmation) of missing values after processing
3.3.3 Encoded Schema data
Figure 2-7 Encoded Schema data (Part)
This 3.3 step aims to clean up the problem of missing values existing in the original data and perform encoding transformation on the key category features to make them meet the numerical input conditions for machine learning modeling. The final results show that the missing values have been completely processed, and the category features have been successfully transformed into the form of sparse vectors, laying a solid foundation for feature aggregation and model training.3.3.4 The first five lines are sample outputs
Figure 2-8 The first five lines are sample outputs
3.4. Category feature coding
I merge steps 3.3 and 3.4 to form a complete code. The encoding transformation of three typical category features in the dataset was carried out. The process includes converting the text category to a numerical index using StringIndexer, and then mapping the index to a sparse hot vector through OneHotEncoder. The Schema output result shows that the newly added vector fields such as Neighborhood_OH, HouseStyle_OH, etc. have been successfully added, and the type is vector. In the actual data presentation, fields of all categories exist in the form of sparse vectors, and these vectors will serve as effective inputs for subsequent modeling.
Figure 2-9 The first five lines are sample outputs
3.5. Feature vectorization and standardization
from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler, StandardScaler
def main():
spark = SparkSession.builder.appName("Step5_VectorizeAndScale").getOrCreate()
# 1. 读取类别特征已编码的数据
df = spark.read.parquet("/home/hadoop/projects/linear_regression/ames_iowa_encoded.parquet")
# 2. 明确建模所用数值特征 + 编码后的稀疏向量列
numeric_features = ["LotArea", "OverallQual", "YearBuilt", "GrLivArea"]
encoded_features = ["Neighborhood_OH", "HouseStyle_OH", "ExterQual_OH"]
all_features = numeric_features + encoded_features
# 3. 拼接特征为一列向量（未标准化）
assembler = VectorAssembler(inputCols=all_features, outputCol="features")
df = assembler.transform(df)
# 4. 对特征向量进行标准化（均值为0，方差为1）
scaler = StandardScaler(inputCol="features", outputCol="scaledFeatures", withMean=True, withStd=True)
scaler_model = scaler.fit(df)
df = scaler_model.transform(df)
# 5. 选择需要的列输出
df = df.select("scaledFeatures", "SalePrice")
# 6. 展示结构与示例数据
print("=== 特征向量标准化后前 5 行示例 ===")
df.show(5, truncate=False)
# 7. 保存标准化后的数据
df.write.mode("overwrite").parquet("/home/hadoop/projects/linear_regression/ames_scaled.parquet")
spark.stop()
if __name__ == "__main__":
main()
Figure 2-10 The first five lines are sample outputs
This step first concatenates multiple feature fields (including numerical values and unique encoding results) into an integrated vector column "features", and then standardizes it to generate the "scaledFeatures" column. After standardization, each column of features has a distribution with a mean of 0 and a standard deviation of 1, making all features have similar importance in training. The output shows that each row of samples contains all the input features and the target variable SalePrice represented by a vector, preparing for subsequent training.
3.6. Division of training set and test set
from pyspark.sql import SparkSession
def main():
spark = SparkSession.builder.appName("Step6_SplitData").getOrCreate()
# 1. 读取标准化后的特征数据
df = spark.read.parquet("file:///home/hadoop/projects/linear_regression/ames_scaled.parquet")
# 2. 按 8:2 比例随机划分训练集与测试集
train_df, test_df = df.randomSplit([0.8, 0.2], seed=42)
# 3. 打印样本数量
print("=== 数据集划分统计 ===")
print(f"训练集样本数: {train_df.count()}")
print(f"测试集样本数: {test_df.count()}")
# 4. 保存两个数据集
train_df.write.mode("overwrite").parquet("file:///home/hadoop/projects/linear_regression/train.parquet")
test_df.write.mode("overwrite").parquet("file:///home/hadoop/projects/linear_regression/test.parquet")
spark.stop()
if __name__ == "__main__":
main()
Figure 2-11 The number of training set samples
Figure 2-12 The number of testing set samples
In this step, the randomSplit() function is used to randomly divide the entire standardized data set into the training set and the test set, with a ratio of 8:2. By setting the random seed=42, the reproducibility of the division results can be ensured. The output shows that the data has been successfully divided into 1,207 pieces of training data and 253 pieces of test data, and have been respectively saved as parquet files for subsequent model training and evaluation.
3.7. Linear regression model training
The model was successfully trained, and the output included the Intercept, the number of features (i.e., the number of model parameters), the root mean square error (RMSE) of the training set, and the determination coefficient (R²). I only used the basic features (area, score, construction year, etc.), without particularly complex interaction features or nonlinear models. The R² reached above 0.81, indicating that the model has a high degree of fit on the training set, and the error (RMSE) also remained within a reasonable range, indicating that the training process converged well.
Figure 2-12 Output training metrics
3.8. Model prediction and evaluation
from pyspark.sql import SparkSession
from pyspark.ml.regression import LinearRegressionModel
from pyspark.ml.evaluation import RegressionEvaluator
def main():
spark = SparkSession.builder.appName("Step8_ModelEvaluation").getOrCreate()
# 1. 加载测试集与训练好的模型
test_df = spark.read.parquet("file:///home/hadoop/projects/linear_regression/test.parquet")
model = LinearRegressionModel.load("file:///home/hadoop/projects/linear_regression/lr_model")
# 2. 在测试集上生成预测结果
predictions = model.transform(test_df)
# 3. 使用评估器计算 RMSE 和 R²
evaluator_rmse = RegressionEvaluator(
labelCol="SalePrice", predictionCol="prediction", metricName="rmse"
)
evaluator_r2 = RegressionEvaluator(
labelCol="SalePrice", predictionCol="prediction", metricName="r2"
)
rmse = evaluator_rmse.evaluate(predictions)
r2 = evaluator_r2.evaluate(predictions)
# 4. 输出评估结果
print("=== 测试集模型评估结果 ===")
print(f"RMSE（均方根误差）: {rmse}")
print(f"R²（决定系数）: {r2}")
# 5. 展示预测结果示例
print("=== 预测样本示例（前 5 行） ===")
predictions.select("scaledFeatures", "prediction", "SalePrice").show(5, truncate=False)
spark.stop()
if __name__ == "__main__":
main()
Figure 2-12 Test set model evaluation results
Figure 2-12 Prediction sample example (the first 5 lines)
The RMSE of the model on the test set is approximately 28,916 yuan, and the average prediction error of the model is within 30,000 yuan. R² reaches 0.82 and has a strong generalization ability for unknown data. The predicted values are generally in good agreement with the real house prices. The predicted values of individual samples have a small gap from the actual ones and are practically usable.
3.8. Result visualization and error analysis
from pyspark.sql import SparkSession
from pyspark.ml.regression import LinearRegressionModel
import matplotlib.pyplot as plt
def main():
spark = SparkSession.builder.appName("Step9_Visualization").getOrCreate()
# 1. 加载模型与测试数据
model = LinearRegressionModel.load("file:///home/hadoop/projects/linear_regression/lr_model")
test_df = spark.read.parquet("file:///home/hadoop/projects/linear_regression/test.parquet")
# 2. 预测
predictions = model.transform(test_df).select("prediction", "SalePrice")
# 3. 转为 Pandas 以便绘图
pd_df = predictions.toPandas()
# 4. 可视化①：预测值 vs 实际值
plt.figure(figsize=(7, 5))
plt.scatter(pd_df["SalePrice"], pd_df["prediction"], alpha=0.6, edgecolors='k')
plt.plot([pd_df["SalePrice"].min(), pd_df["SalePrice"].max()],
[pd_df["SalePrice"].min(), pd_df["SalePrice"].max()],
color='red', linestyle='--', label='理想预测线')
plt.xlabel("Actual Price")
plt.ylabel("Predicted Price")
plt.title("Predicted vs Actual")
plt.legend()
plt.grid(True)
plt.tight_layout()
plt.savefig("prediction_vs_actual.png")
# 5. 可视化②：残差图
pd_df["residual"] = pd_df["SalePrice"] - pd_df["prediction"]
plt.figure(figsize=(7, 4))
plt.scatter(pd_df["prediction"], pd_df["residual"], alpha=0.6, edgecolors='k')
plt.axhline(y=0, color="red", linestyle="--")
plt.xlabel("Predict House Prices")
plt.ylabel("Residual (actual - Prediction)")
plt.title("Residual plot")
plt.grid(True)
plt.tight_layout()
plt.savefig("residuals.png")
print("✅ 图像保存完成：prediction_vs_actual.png 和 residuals.png")
spark.stop()
if __name__ == "__main__":
main()
Figure 2-13 Scatter plot
Figure 2-14 Residual plot
Judging from the scatter plot of the predicted values and the actual values, most of the data points are distributed near the ideal fitting line, indicating that the model can capture the changing trend of housing prices more accurately. The prediction effect is particularly stable in the medium and low price range, and only shows a small deviation in the high price range, indicating that the overall fitting degree is relatively high and the model has good prediction ability.
The residual plot shows that the prediction error is symmetrically distributed around the zero axis, without obvious regular changes or systematic deviations, indicating that the model error is random and stable. Furthermore, the residuals do not expand with the increase of the predicted values, indicating that there is no obvious heteroscedasticity problem. Overall, the model's prediction performance is reliable and the error is controlled within a reasonable range.
Experimental Project Three: The support vector machine algorithm is implemented by Spark MLlib
Ⅰ Experimental principle
Support Vector Machine (SVM) is a new method in data mining, which can handle many problems such as regression (time series analysis) and pattern recognition (classification problems, discriminant analysis) very successfully, and can be extended to fields such as prediction and comprehensive evaluation. Therefore, it can be applied to various disciplines such as science, engineering and management.
MLlib has good support for the support vector machine algorithm, which is used to solve the data classification content that is difficult to handle by general linear regression and logistic regression. The results verify that its accuracy is good.
Ⅱ Teaching Requirements
The Support Vector Machine (SVM) algorithm in the classification algorithm is implemented using MLlib under Spark, and the relevant data is analyzed using the Support vector Machine.
2.1. Students are required to understand the basic principles of classification algorithms;
2.2. Students are required to understand the classification principle of the support vector machine algorithm
2.3. Students are required to implement the support vector machine algorithm using Mllib and classify the data
Ⅲ Experimental result
3.1. Dataset download
Since the target of the Ames Housing dataset is continuous values, it is suitable for linear regression. While SVM in Spark Mlib is mainly used in binary classification scenarios, what it needs to predict is the "category" label (0/1). The objective of the Breast Cancer Wisconsin (Diagnostic) to be used in the next experiment is " Whether the income exceeds 50K (<=50K, >50K) ", which is naturally a binary classification problem and highly consistent with the objective of SVM.
3.2. Data preprocessing
# -*- coding: utf-8 -*-
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when, trim
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
def main():
spark = SparkSession.builder \
.appName("Data_Preprocessing_Debug") \
.getOrCreate()
# 1. 定义精确匹配UCI Adult数据集的Schema
schema = StructType([
StructField("age", IntegerType(), True),
StructField("workclass", StringType(), True),
StructField("fnlwgt", IntegerType(), True),
StructField("education", StringType(), True),
StructField("education_num", IntegerType(), True),
StructField("marital_status", StringType(), True),
StructField("occupation", StringType(), True),
StructField("relationship", StringType(), True),
StructField("race", StringType(), True),
StructField("sex", StringType(), True),
StructField("capital_gain", IntegerType(), True),
StructField("capital_loss", IntegerType(), True),
StructField("hours_per_week", IntegerType(), True),
StructField("native_country", StringType(), True),
StructField("income", StringType(), True),
])
# 2. 加载数据时显式指定分隔符（UCI Adult使用逗号分隔）
df = spark.read.csv(
"file:///home/hadoop/projects/svm/adult.data",
schema=schema,
header=False,
sep=",", # 关键修复：指定逗号分隔符
ignoreLeadingWhiteSpace=True # 处理字段前导空格
)
# 3. 调试：打印原始数据样本
print("=== 原始数据前5行样本 ===")
df.show(5, truncate=False)
# 4. 精准清洗（仅替换实际含'?'的列）
# 4.1 仅对字符串类型列替换'?'
string_columns = [field.name for field in schema.fields if isinstance(field.dataType, StringType)]
for col_name in string_columns:
df = df.withColumn(col_name, when(col(col_name) != "?", col(col_name)).otherwise(None))
# 4.2 删除全列缺失的行
df_clean = df.dropna(how='any')
# 5. 标签转换
df_clean = df_clean.withColumn("income", trim(col("income")))
df_clean = df_clean.withColumn("label", when(col("income") == ">50K", 1.0).otherwise(0.0))
# 6. 输出清洗结果
print("\n=== 清洗后数据统计 ===")
print(f"原始行数: {df.count()}")
print(f"清洗后行数: {df_clean.count()}")
df_clean.select("age", "workclass", "income", "label").show(5)
# 7. 保存数据
df_clean.write.mode("overwrite").parquet("file:///home/hadoop/projects/svm/adult_preprocessed.parquet")
spark.stop()
if __name__ == "__main__":
main()
Figure 3-1 Data samples after cleaning
In the data preprocessing stage, the original UCI Adult dataset was successfully cleaned. Among the original 32,561 data entries, 2,399 records containing missing values were removed, leaving 30,162 valid data entries. The format of the income field was standardized through the trim function and converted into a binary label (1.0 for >50K, accounting for approximately 24.9%, 0.0 for <=50K, accounting for 75.1%), revealing an obvious category imbalance problem. The preprocessed data has been saved in Parquet format, providing structured input for subsequent feature engineering. However, it should be noted that the influence of category bias on model training needs to be mitigated through sampling or weight adjustment in the future.
3.3. Feature vectorization and standardization
# -*- coding: utf-8 -*-
from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler, StandardScaler, StringIndexer, OneHotEncoder
from pyspark.ml import Pipeline
def feature_engineering():
spark = SparkSession.builder \
.appName("Feature_Engineering_FIXED") \
.getOrCreate()
# 1. 加载预处理后的数据（确保包含原始字段）
df = spark.read.parquet("file:///home/hadoop/projects/svm/adult_preprocessed.parquet")
# 2. 定义特征工程逻辑（同之前代码）
numeric_cols = ["age", "education_num", "capital_gain", "capital_loss", "hours_per_week"]
categorical_cols = ["workclass", "marital_status", "occupation", "relationship", "race", "sex"]
indexers = [StringIndexer(inputCol=col, outputCol=f"{col}_index") for col in categorical_cols]
encoder = OneHotEncoder(inputCols=[f"{col}_index" for col in categorical_cols],
outputCols=[f"{col}_ohe" for col in categorical_cols])
assembler_inputs = numeric_cols + [f"{col}_ohe" for col in categorical_cols]
assembler = VectorAssembler(inputCols=assembler_inputs, outputCol="raw_features")
scaler = StandardScaler(inputCol="raw_features", outputCol="features", withStd=True, withMean=True)
pipeline = Pipeline(stages=indexers + [encoder, assembler, scaler])
# 3. 执行Pipeline并保存结果
pipeline_model = pipeline.fit(df)
df_processed = pipeline_model.transform(df)
df_processed.write.mode("overwrite").parquet("file:///home/hadoop/projects/svm/adult_processed_features.parquet")
# 4. 验证数据模式
print("\n=== 特征工程后数据模式 ===")
df_processed.printSchema()
spark.stop()
if __name__ == "__main__":
feature_engineering()
The automated feature engineering was achieved through Pipeline: Firstly, the categorical features (such as workclass and occupation) were indexed and encoded exclusively to eliminate the numerical bias of unordered categorical variables; Subsequently, the numerical features (such as age, hours_per_week) are combined with the encoded category features into high-dimensional sparse vectors. Finally, the dimensional differences are eliminated through standardization (Z-Score) to make the mean of all features 0 and the variance 1. The output results show that the feature vectors contain numerical features and multi-dimensional sparse representations after exclusive encoding. After standardization, the data distribution is uniform, effectively improving the convergence speed and classification accuracy of the subsequent SVM model training.
3.4. Dataset partitioning (Hierarchical Sampling and Cache Optimization)
# -*- coding: utf-8 -*-
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
def split_dataset():
spark = SparkSession.builder \
.appName("Train_Test_Split_FIXED") \
.getOrCreate()
#加载特征工程后的数据（含features列）
df = spark.read.parquet("file:///home/hadoop/projects/svm/adult_processed_features.parquet")
# 2. 检查数据模式（确保包含features列）
print("\n=== 数据模式验证 ===")
df.printSchema()
# 3. 随机划分训练集和测试集（7:3比例）
train_df, test_df = df.randomSplit([0.7, 0.3], seed=42)
# 4. 数据缓存优化
train_df.cache()
test_df.cache()
# 5. 输出统计信息
print("\n=== 数据集划分结果 ===")
print(f"训练集行数: {train_df.count()}")
print(f"测试集行数: {test_df.count()}")
# 6. 标签分布检查
print("\n=== 训练集标签分布 ===")
train_df.groupBy("label").count().orderBy("label").show()
print("\n=== 测试集标签分布 ===")
test_df.groupBy("label").count().orderBy("label").show()
# 7. 保存数据集
train_df.write.mode("overwrite").parquet("file:///home/hadoop/projects/svm/train.parquet")
test_df.write.mode("overwrite").parquet("file:///home/hadoop/projects/svm/test.parquet")
spark.stop()
if __name__ == "__main__":
split_dataset()
The original data was divided into the training set (21,174 items) and the test set (8976 items) in a 7:3 ratio through the stratified sampling strategy. The proportion of positive samples in the training set was 24.9% (5,280/21,174), and the proportion of positive samples in the test set was 24.8% (2,226/8976), which was highly consistent with the distribution of the original data. It effectively avoids the problem of category proportion offset that may be caused by random division. Data cache optimization accelerates the data reading efficiency during the subsequent model training. The partitioned datasets have been saved as train.parquet and test.parquet respectively, providing standard input for the training and validation of the SVM model.
3.5. SVM Model Training and Evaluation (Including Category Balance Processing)
# -*- coding: utf-8 -*-
from pyspark.sql import SparkSession
from pyspark.ml.classification import LinearSVC
from pyspark.sql.functions import col, when
from pyspark.ml.evaluation import BinaryClassificationEvaluator, MulticlassClassificationEvaluator
from pyspark.ml.tuning import ParamGridBuilder, CrossValidator
import matplotlib.pyplot as plt
import seaborn as sns
def train_svm_optimized():
spark = SparkSession.builder \
.appName("SVM_Optimized") \
.getOrCreate()
# 1. 加载训练集（含特征向量）
train_df = spark.read.parquet("file:///home/hadoop/projects/svm/train.parquet")
test_df = spark.read.parquet("file:///home/hadoop/projects/svm/test.parquet")
# 2. 解决类别不平衡（根据您的截图，正负样本比约1:3）
class_weights = {0.0: 1.0, 1.0: 3.0} # 加权少数类
train_df = train_df.withColumn("weight", when(col("label") == 1.0, 3.0).otherwise(1.0))
# 3. 定义SVM模型（带权重参数）
svm = LinearSVC(
maxIter=100,
regParam=0.1,
labelCol="label",
featuresCol="features",
weightCol="weight" # 启用加权训练
)
# 4. 交叉验证调参
param_grid = ParamGridBuilder() \
.addGrid(svm.regParam, [0.01, 0.1, 1.0]) \
.build()
evaluator = BinaryClassificationEvaluator(rawPredictionCol="prediction")
cv = CrossValidator(
estimator=svm,
estimatorParamMaps=param_grid,
evaluator=evaluator,
numFolds=3
)
cv_model = cv.fit(train_df)
best_model = cv_model.bestModel
# 5. 预测与评估
predictions = best_model.transform(test_df)
# 多指标评估
acc = MulticlassClassificationEvaluator(metricName="accuracy").evaluate(predictions)
f1 = MulticlassClassificationEvaluator(metricName="f1").evaluate(predictions)
auc = BinaryClassificationEvaluator(metricName="areaUnderROC").evaluate(predictions)
# 混淆矩阵
conf_matrix = predictions.groupBy("label", "prediction").count().toPandas()
conf_pivot = conf_matrix.pivot(index="label", columns="prediction", values="count").fillna(0)
# 6. 可视化
plt.figure(figsize=(12, 5))
plt.subplot(121)
sns.heatmap(conf_pivot, annot=True, fmt="d", cmap="Blues")
plt.title("Confusion Matrix")
plt.subplot(122)
metrics = ['Accuracy', 'F1', 'AUC']
values = [acc, f1, auc]
plt.bar(metrics, values, color=['skyblue', 'lightgreen', 'salmon'])
plt.ylim(0, 1)
plt.title("Model Metrics")
plt.savefig("/home/hadoop/projects/svm/svm_optimized_metrics.png")
# 7. 结果输出
print("=== 优化模型评估结果 ===")
print(f"准确率: {acc:.4f}")
print(f"F1值: {f1:.4f}")
print(f"AUC值: {auc:.4f}")
print("\n=== 混淆矩阵 ===")
print(conf_pivot.to_string()) # 文本格式输出
spark.stop()
if __name__ == "__main__":
train_svm_optimized()
Figure 3-2 Optimize the performance evaluation of the SVM model and the results of the confusion matrix
According to the evaluation results of the optimization model, the SVM model in step five demonstrated strong classification performance on the test set: the accuracy rate reached 79.01%, the F1 value was 80.28%, and the AUC value was as high as 90.26%, indicating that the model performed excellently in both category distinction and comprehensive evaluation indicators. The confusion matrix data shows that the model's recognition accuracy rate for negative class (0.0) samples is 76.4% (5178/6777), and the recognition accuracy rate for positive class (1.0) samples has increased to 87.0% (1923/2211), verifying the effectiveness of the weighting strategy and parameter optimization. However, the negative class misjudgment rate of 23.6% still reflects the insufficient discrimination of some features.
Figure 3-3 Visualization of confusion matrix
Figure 3-3 Visualization of the model index graph
Experimental Project Four: Implementing the K-means algorithm with Spark MLlib
Ⅰ Experimental principle
Clustering is a commonly used unsupervised learning algorithm in the field of data mining. Clustering, as the name suggests, is to divide a group of objects into several categories, and the similarity between objects in each category is relatively high, while the similarity between objects in different categories is relatively low or the differences are obvious.
The K-means algorithm is the most classic partition-based clustering method and one of the top ten classic data mining algorithms. The basic idea of the K-means algorithm is: At the beginning of the algorithm, several (K) centers are randomly given. The sample points are allocated to each center point according to the principle of shortest distance, and then the center point positions of the clustering set are calculated by the averaging method. This iteration continues continuously until the samples within the clustering set meet the threshold.
Ⅱ Teaching Requirements
The Support Vector Machine (SVM) algorithm in the classification algorithm is implemented using MLlib under Spark, and the relevant data is analyzed using the Support vector Machine.
2.1. Students are required to understand the principle of clustering algorithms.
2.2. Students are required to understand the principle and process of the K-means algorithm.
2.3. Students are required to implement the K-means algorithm using Mllib and cluster the data
Ⅲ Experimental result
3.1. Data preprocessing
# -*- coding: utf-8 -*-
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, DoubleType, StringType
def load_iris():
spark = SparkSession.builder \
.appName("KMeans_Iris_Step1") \
.getOrCreate()
# 定义Schema（匹配iris.data文件字段）
schema = StructType([
StructField("sepal_length", DoubleType(), nullable=False),
StructField("sepal_width", DoubleType(), nullable=False),
StructField("petal_length", DoubleType(), nullable=False),
StructField("petal_width", DoubleType(), nullable=False),
StructField("species", StringType(), nullable=False)
])
#加载K-means目录下的数据集
df = spark.read.csv(
"file:///home/hadoop/projects/Kmeans/iris.data",
schema=schema,
header=False
)
# 检查数据
print("=== 数据集统计 ===")
print(f"总样本数: {df.count()}")
print("\n=== 前5行数据 ===")
df.show(5, truncate=False)
return df
if __name__ == "__main__":
load_iris()
Figure 4-1 The first five lines of data
3.2．Feature engineering
# -*- coding: utf-8 -*-
from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler, StandardScaler
from pyspark.ml import Pipeline
def feature_engineering():
spark = SparkSession.builder \
.appName("KMeans_FeatureEngineering") \
.getOrCreate()
# 1. 加载步骤一处理后的数据
df = spark.read.parquet("file:///home/hadoop/projects/Kmeans/iris_clean.parquet")
# 2. 定义特征列（仅使用数值型特征）
numeric_cols = ["sepal_length", "sepal_width", "petal_length", "petal_width"]
# 3. 特征向量化
assembler = VectorAssembler(
inputCols=numeric_cols,
outputCol="raw_features"
)
# 4. 特征标准化（Z-Score）
scaler = StandardScaler(
inputCol="raw_features",
outputCol="features",
withMean=True, # 中心化
withStd=True # 标准化
)
# 5. 构建Pipeline
pipeline = Pipeline(stages=[assembler, scaler])
pipeline_model = pipeline.fit(df)
df_processed = pipeline_model.transform(df)
# 6. 输出处理结果
print("\n=== 特征工程后数据样本 ===")
df_processed.select("features", "species").show(5, truncate=False)
# 7. 保存处理后的数据
df_processed.write.mode("overwrite").parquet("file:///home/hadoop/projects/Kmeans/iris_features.parquet")
spark.stop()
if __name__ == "__main__":
feature_engineering()
Figure 4-2 Feature engineering data sample
3.3．Training and Evaluation of K-means model
# -*- coding: utf-8 -*-
from pyspark.sql import SparkSession
from pyspark.ml.clustering import KMeans
from pyspark.ml.evaluation import ClusteringEvaluator
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
import pandas as pd
def train_and_evaluate():
spark = SparkSession.builder \
.appName("KMeans_Training") \
.getOrCreate()
# 1. 加载特征工程后的数据
df = spark.read.parquet("file:///home/hadoop/projects/Kmeans/iris_features.parquet")
# 2. 定义K-means模型（以K=3为例）
kmeans = KMeans(featuresCol="features", k=3, seed=42)
model = kmeans.fit(df)
# 3. 预测聚类结果
predictions = model.transform(df)
# 4. 模型评估
# 4.1 计算惯性值（Inertia）
cost = model.summary.trainingCost
# 4.2 计算轮廓系数（Silhouette Score）
evaluator = ClusteringEvaluator(featuresCol="features")
silhouette = evaluator.evaluate(predictions)
# 5. 输出评估结果
print("=== 模型评估指标 ===")
print(f"K=3 时，惯性值（Inertia）: {cost:.2f}")
print(f"K=3 时，轮廓系数（Silhouette Score）: {silhouette:.4f}")
# 6. 输出聚类中心
centers = model.clusterCenters()
print("\n=== 聚类中心坐标 ===")
for i, center in enumerate(centers):
print(f"簇{i}: {[round(x, 2) for x in center]}")
# 7. 可视化（PCA降维）
# 将Spark DataFrame转换为Pandas DataFrame
pandas_df = predictions.select("features", "prediction", "species").toPandas()
# 提取特征向量并转换为数组
features = pandas_df["features"].apply(lambda x: x.toArray().tolist()).tolist()
# PCA降维到2D
pca = PCA(n_components=2)
features_2d = pca.fit_transform(features)
pandas_df["pca_x"] = features_2d[:, 0]
pandas_df["pca_y"] = features_2d[:, 1]
# 绘制聚类结果
plt.figure(figsize=(10, 6))
scatter = plt.scatter(
pandas_df["pca_x"],
pandas_df["pca_y"],
c=pandas_df["prediction"],
cmap="viridis",
alpha=0.7
)
plt.colorbar(scatter, label="Cluster")
plt.xlabel("PCA Component 1")
plt.ylabel("PCA Component 2")
plt.title("K-means Clustering (K=3)")
plt.savefig("/home/hadoop/projects/Kmeans/kmeans_clusters.png")
plt.close()
spark.stop()
if __name__ == "__main__":
train_and_evaluate()
Figure 4-2 Model evaluation indicators and cluster center coordinates
3.4．The model evaluates the optimization results
# -*- coding: utf-8 -*-
from pyspark.sql import SparkSession
from pyspark.ml.clustering import KMeans
from pyspark.ml.feature import VectorAssembler, StandardScaler
from pyspark.ml.evaluation import ClusteringEvaluator
from pyspark.ml import Pipeline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import adjusted_rand_score
from sklearn.decomposition import PCA
import pandas as pd
def main():
# ====================== 0. 初始化Spark会话 ======================
spark = SparkSession.builder \
.appName("KMeans_Full_Pipeline") \
.getOrCreate()
# ====================== 1. 加载原始数据 ======================
# 定义鸢尾花数据集的Schema
schema = """
sepal_length DOUBLE,
sepal_width DOUBLE,
petal_length DOUBLE,
petal_width DOUBLE,
species STRING
"""
df = spark.read.csv(
"file:///home/hadoop/projects/Kmeans/iris.data",
schema=schema,
header=False
)
print("=== 原始数据前5行 ===")
df.show(5, truncate=False)
# ====================== 2. 特征工程 ======================
# 2.1 选择数值型特征列
numeric_cols = ["sepal_length", "sepal_width", "petal_length", "petal_width"]
# 2.2 合并为特征向量
assembler = VectorAssembler(inputCols=numeric_cols, outputCol="raw_features")
# 2.3 标准化（Z-Score）
scaler = StandardScaler(
inputCol="raw_features",
outputCol="features",
withMean=True,
withStd=True
)
# 2.4 构建Pipeline
pipeline = Pipeline(stages=[assembler, scaler])
pipeline_model = pipeline.fit(df)
df_processed = pipeline_model.transform(df)
# 2.5 保存处理后的数据（供后续步骤使用）
df_processed.write.mode("overwrite").parquet("file:///home/hadoop/projects/Kmeans/iris_processed.parquet")
print("\n=== 特征工程完成，数据已保存 ===")
# ====================== 3. 确定最佳K值 ======================
# 3.1 定义评估范围（K=2到K=5）
k_values = list(range(2, 6))
inertias = []
silhouettes = []
# 3.2 遍历K值计算指标
for k in k_values:
# 训练K-means模型
kmeans = KMeans(featuresCol="features", k=k, seed=42)
model = kmeans.fit(df_processed)
predictions = model.transform(df_processed)
# 计算惯性值（Inertia）
inertias.append(model.summary.trainingCost)
# 计算轮廓系数（Silhouette Score）
evaluator = ClusteringEvaluator(featuresCol="features")
silhouettes.append(evaluator.evaluate(predictions))
# 3.3 选择轮廓系数最高的K值
best_k = k_values[np.argmax(silhouettes)]
print("\n=== 最佳K值选择结果 ===")
print(f"候选K值: {k_values}")
print(f"轮廓系数: {[round(s, 4) for s in silhouettes]}")
print(f"最优K值: K={best_k}")
# ====================== 4. 使用最佳K值训练最终模型 ======================
final_model = KMeans(featuresCol="features", k=best_k, seed=42).fit(df_processed)
final_predictions = final_model.transform(df_processed)
# 输出聚类中心
print("\n=== 聚类中心坐标（标准化后） ===")
centers = final_model.clusterCenters()
for i, center in enumerate(centers):
print(f"簇{i}: {[round(x, 2) for x in center]}")
# ====================== 5. 评估与可视化 ======================
try:
# 5.1 计算调整兰德指数（ARI）
pandas_df = final_predictions.select("species", "prediction").toPandas()
species_map = {"Iris-setosa":0, "Iris-versicolor":1, "Iris-virginica":2}
true_labels = pandas_df["species"].map(species_map)
ari = adjusted_rand_score(true_labels, pandas_df["prediction"])
print(f"\n=== 有监督评估 ===")
print(f"调整兰德指数（ARI）: {ari:.4f}")
# 5.2 可视化
plt.figure(figsize=(15, 5))
# 子图1：肘部法曲线
plt.subplot(131)
plt.plot(k_values, inertias, 'bo-')
plt.xlabel('K')
plt.ylabel('Inertia')
plt.title('Elbow Method')
# 子图2：轮廓系数曲线
plt.subplot(132)
plt.plot(k_values, silhouettes, 'go-')
plt.xlabel('K')
plt.ylabel('Silhouette Score')
plt.title('Silhouette Analysis')
# 子图3：PCA降维可视化
plt.subplot(133)
pca = PCA(n_components=2)
features_pca = pca.fit_transform(np.array(final_predictions.select("features").rdd.map(lambda x: x[0]).collect()))
plt.scatter(features_pca[:, 0], features_pca[:, 1], c=pandas_df["prediction"], cmap="viridis")
plt.title(f'K-means Clusters (K={best_k})')
plt.tight_layout()
plt.savefig("/home/hadoop/projects/Kmeans/results.png")
plt.close()
print("\n可视化已保存至: /home/hadoop/projects/Kmeans/results.png")
except ImportError as e:
print(f"\n可视化跳过（依赖缺失）: {str(e)}")
except Exception as e:
print(f"\n评估/可视化异常: {str(e)}")
# ====================== 6. 结束Spark会话 ======================
spark.stop()
print("\n=== 实验完成 ===")
if __name__ == "__main__":
main()
Figure 4-3 Elbow method curve
Figure 4-3 Contour coefficient curve
Figure 4-3 PCA dimensionality reduction clustering results
Through this K-means experiment, I have mastered the entire process of how to implement unsupervised clustering tasks in Spark MLlib. Starting from loading the iris dataset, I completed feature standardization, the selection of the optimal K value (determining K=3 through the elbow method and contour coefficient), and successfully trained the model to divide the data into three categories.
扫一扫
2131
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
