ID3 与 C4.5 的预剪枝、后剪枝实现与精度对比

一、决策树剪枝背景：为什么需要剪枝？

决策树在完全生长时，会对训练数据拟合得过于 “完美”，导致在测试集上表现不佳（过拟合）。剪枝的核心是简化树结构，平衡 “拟合能力” 与 “泛化能力”。

• 预剪枝：在树生长过程中，通过超参数（如最大深度、叶节点最小样本数）直接限制树的复杂度。
• 后剪枝：先让树完全生长，再通过验证集精度反向裁剪无效分支。

二、数据集与环境准备

本文使用 “贷款审批” 数据集，特征包括年龄段、有工作、有自己的房子、贷款情况，标签为是否给贷款。

环境依赖：

import pandas as pd
import numpy as np
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.metrics import accuracy_score, classification_report
from sklearn.tree._tree import TREE_LEAF
import matplotlib.pyplot as plt

# 中文字体设置
plt.rcParams["font.family"] = ["SimHei"]  
plt.rcParams["axes.unicode_minus"] = False  
%matplotlib inline  # Jupyter中强制显示图片

数据集定义：

# 训练集
train_data = np.array([
    [0,0,0,0], [0,0,0,1], [0,1,0,1], [0,1,1,0], [0,0,0,0],
    [1,0,0,0], [1,0,0,1], [1,1,1,1], [1,0,1,2], [1,0,1,2],
    [2,0,1,2], [2,0,1,1], [2,1,0,1], [2,1,0,2], [2,0,0,0], [2,0,0,2]
])
train_label = np.array([0,0,1,1,0, 0,0,1,1,1, 1,1,1,1,0,0])

# 测试集
test_data = np.array([
    [0,0,0,1], [0,1,0,1], [1,0,1,2], [1,0,0,1], [2,1,0,2], [2,0,0,0], [2,0,0,2]
])
test_label = np.array([0,1,1,0,1,0,0])

feature_names = ['年龄段', '有工作', '有自己的房子', '信贷情况']
class_names = ['不给贷款', '给贷款']

三、ID3 与 C4.5 算法基础实现

1. ID3 决策树（基于信息增益）

# 训练ID3决策树（剪枝前，完全生长）
id3_tree_unpruned = DecisionTreeClassifier(
    criterion='entropy',  # 信息增益对应熵
    max_depth=None,       # 不限制深度
    random_state=42
)
id3_tree_unpruned.fit(train_data, train_label)

2. C4.5 决策树（基于信息增益比）

由于 sklearn 无原生 C4.5，需自定义类实现信息增益比计算：

class C45DecisionTree(DecisionTreeClassifier):
    def __init__(self, max_depth=None, min_samples_split=2, min_samples_leaf=1, random_state=42):
        super().__init__(
            criterion='entropy',  # 基于熵计算
            max_depth=max_depth,
            min_samples_split=min_samples_split,
            min_samples_leaf=min_samples_leaf,
            random_state=random_state
        )
    
    def _calc_information_gain_ratio(self, X, y, feature_idx):
        def entropy(y):
            classes, counts = np.unique(y, return_counts=True)
            probs = counts / len(y)
            return -np.sum(probs * np.log2(probs + 1e-10))
        
        parent_entropy = entropy(y)
        feature_values = X[:, feature_idx]
        unique_vals = np.unique(feature_values)
        child_entropy = 0.0
        split_info = 0.0
        
        for val in unique_vals:
            mask = (feature_values == val)
            child_y = y[mask]
            child_prob = len(child_y) / len(y)
            child_entropy += child_prob * entropy(child_y)
            split_info -= child_prob * np.log2(child_prob + 1e-10)
        
        information_gain = parent_entropy - child_entropy
        return information_gain / (split_info + 1e-10) if split_info != 0 else 0.0

# 训练C4.5决策树（剪枝前，完全生长）
c45_tree_unpruned = C45DecisionTree(max_depth=None, random_state=42)
c45_tree_unpruned.fit(train_data, train_label)

四、预剪枝实战：训练时限制树生长

预剪枝通过max_depth（最大深度）、min_samples_leaf（叶节点最小样本数）等参数，在训练阶段直接停止树的过度生长。

# ID3预剪枝
id3_tree_prepruned = DecisionTreeClassifier(
    criterion='entropy',
    max_depth=3,          # 限制最大深度为3
    min_samples_leaf=2,   # 叶节点最少2个样本
    random_state=42
)
id3_tree_prepruned.fit(train_data, train_label)

# C4.5预剪枝
c45_tree_prepruned = C45DecisionTree(
    max_depth=3,
    min_samples_leaf=2,
    random_state=42
)
c45_tree_prepruned.fit(train_data, train_label)

五、后剪枝实战：生长后裁剪无效分支

后剪枝先让树完全生长，再通过验证集精度判断是否裁剪分支（仅保留对精度有贡献的结构）。

def post_prune(tree, tree_idx, X_train, y_train, X_val, y_val):
    left_child = tree.tree_.children_left[tree_idx]
    right_child = tree.tree_.children_right[tree_idx]
    
    # 递归剪枝子节点
    if left_child != TREE_LEAF:
        post_prune(tree, left_child, X_train, y_train, X_val, y_val)
    if right_child != TREE_LEAF:
        post_prune(tree, right_child, X_train, y_train, X_val, y_val)
    
    # 跳过叶节点
    if left_child == TREE_LEAF and right_child == TREE_LEAF:
        return
    
    # 保存原始节点信息
    original_left = tree.tree_.children_left[tree_idx]
    original_right = tree.tree_.children_right[tree_idx]
    original_value = tree.tree_.value[tree_idx].copy()
    original_threshold = tree.tree_.threshold[tree_idx]
    original_feature = tree.tree_.feature[tree_idx]
    
    # 尝试剪枝为叶节点
    tree.tree_.children_left[tree_idx] = TREE_LEAF
    tree.tree_.children_right[tree_idx] = TREE_LEAF
    class_counts = np.bincount(y_train)
    tree.tree_.value[tree_idx] = np.array([[class_counts[0], class_counts[1]]])
    
    # 验证精度
    pruned_pred = tree.predict(X_val)
    pruned_acc = accuracy_score(y_val, pruned_pred)
    
    # 恢复原始节点
    tree.tree_.children_left[tree_idx] = original_left
    tree.tree_.children_right[tree_idx] = original_right
    tree.tree_.value[tree_idx] = original_value
    tree.tree_.threshold[tree_idx] = original_threshold
    tree.tree_.feature[tree_idx] = original_feature
    original_pred = tree.predict(X_val)
    original_acc = accuracy_score(y_val, original_pred)
    
    # 精度提升则保留剪枝
    if pruned_acc > original_acc:
        tree.tree_.children_left[tree_idx] = TREE_LEAF
        tree.tree_.children_right[tree_idx] = TREE_LEAF
        tree.tree_.value[tree_idx] = np.array([[class_counts[0], class_counts[1]]])

# 划分验证集
from sklearn.model_selection import train_test_split
X_train_prune, X_val_prune, y_train_prune, y_val_prune = train_test_split(
    train_data, train_label, test_size=0.3, random_state=42
)

# ID3后剪枝
id3_tree_postpruned = DecisionTreeClassifier(criterion='entropy', max_depth=None, random_state=42)
id3_tree_postpruned.fit(X_train_prune, y_train_prune)
post_prune(id3_tree_postpruned, 0, X_train_prune, y_train_prune, X_val_prune, y_val_prune)

# C4.5后剪枝
c45_tree_postpruned = C45DecisionTree(max_depth=None, random_state=42)
c45_tree_postpruned.fit(X_train_prune, y_train_prune)
post_prune(c45_tree_postpruned, 0, X_train_prune, y_train_prune, X_val_prune, y_val_prune)

六、可视化对比：剪枝前后树结构差异

通过plot_tree可视化决策树，直观对比剪枝效果：

def plot_tree_comparison(tree_unpruned, tree_pruned, title_unpruned, title_pruned):
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))
    
    # 剪枝前
    plot_tree(
        tree_unpruned,
        feature_names=feature_names,
        class_names=class_names,
        filled=True,
        rounded=True,
        fontsize=10,
        proportion=True,
        cmap=plt.cm.Pastel1,
        ax=ax1
    )
    ax1.set_title(title_unpruned, fontsize=15, pad=20)
    
    # 剪枝后
    plot_tree(
        tree_pruned,
        feature_names=feature_names,
        class_names=class_names,
        filled=True,
        rounded=True,
        fontsize=10,
        proportion=True,
        cmap=plt.cm.Pastel1,
        ax=ax2
    )
    ax2.set_title(title_pruned, fontsize=15, pad=20)
    
    plt.tight_layout()
    plt.show()

# 绘制所有对比图
plot_tree_comparison(
    id3_tree_unpruned, id3_tree_prepruned,
    "ID3决策树（剪枝前）", "ID3决策树（预剪枝后）"
)
plot_tree_comparison(
    id3_tree_unpruned, id3_tree_postpruned,
    "ID3决策树（剪枝前）", "ID3决策树（后剪枝后）"
)
plot_tree_comparison(
    c45_tree_unpruned, c45_tree_prepruned,
    "C4.5决策树（剪枝前）", "C4.5决策树（预剪枝后）"
)
plot_tree_comparison(
    c45_tree_unpruned, c45_tree_postpruned,
    "C4.5决策树（剪枝前）", "C4.5决策树（后剪枝后）"
)

可视化结果

七、精度对比：剪枝对模型性能的影响

通过测试集评估剪枝前后的精度、召回率、F1-score：

def calculate_accuracy(tree, X_test, y_test, model_name):
    y_pred = tree.predict(X_test)
    accuracy = accuracy_score(y_test, y_pred)
    print(f"\n{model_name} 精度：{accuracy:.2%}")
    print(classification_report(y_test, y_pred, target_names=class_names))
    return accuracy

# ID3精度对比
print("="*50)
print("ID3决策树精度对比")
print("="*50)
calculate_accuracy(id3_tree_unpruned, test_data, test_label, "ID3（剪枝前）")
calculate_accuracy(id3_tree_prepruned, test_data, test_label, "ID3（预剪枝后）")
calculate_accuracy(id3_tree_postpruned, test_data, test_label, "ID3（后剪枝后）")

# C4.5精度对比
print("\n" + "="*50)
print("C4.5决策树精度对比")
print("="*50)
calculate_accuracy(c45_tree_unpruned, test_data, test_label, "C4.5（剪枝前）")
calculate_accuracy(c45_tree_prepruned, test_data, test_label, "C4.5（预剪枝后）")
calculate_accuracy(c45_tree_postpruned, test_data, test_label, "C4.5（后剪枝后）")

精度结果分析

从实验结果看，剪枝后模型在测试集上的精度未下降（甚至略有提升），说明剪枝有效避免了过拟合，同时保持了分类能力。

八、总结

本文从原理到实战，完整演示了决策树预剪枝和后剪枝的实现逻辑，并通过 ID3 和 C4.5 算法的对比，验证了剪枝对模型泛化能力的优化效果。

• 预剪枝：实现简单、计算高效，适合对性能要求高的场景。
• 后剪枝：剪枝更精细，泛化能力更强，但计算成本较高。