GBDT+LR精排实战

最新推荐文章于 2026-08-21 19:17:26 发布

原创最新推荐文章于 2026-08-21 19:17:26 发布 · 200 阅读

4 ·

本内容遵循CC 4.0 BY-SA版权协议

GEO检测

1、算法原理

（1）GBDT

1）概述

GBDT：Gradient Boosting Decision Tree，梯度提升决策树，是Boosting集成学习算法。

通俗描述：串行训练多棵若决策树，每一棵拟合前面所有树的负梯度（残差近似），累加所有树输出得到最终预测。

2）完整原理

GBDT = 加法模型 + 前向分步 + 负梯度拟合 + 构造回归树

【加法模型】

【前向分布】

一次只学一个，前面所有全部固定不变。

【负梯度拟合】

当第m步时：

目标是：

对于每一个样本i，损失函数为：

一元函数泰勒一阶展开公式：

上面的损失函数转为：

第一项是常数，所以优化损失函数就是最小化第二项。

目标变为：

我们希望f和g尽量符号相反，这样 $g\cdot f$ 变小，等价于让

定位残差

对于MSE损失：

、

为了限制 $f(x_i)$ 过大不符合泰勒展开，所以假如学习率：

【构造回归树】

第m轮得到的残差数据集

分两步：

1. 树的分裂生长，选择特征作为分裂点，划分叶子区域，分裂准则为平方误差最小

2.叶子节点赋值：

求导过程：

（2）LightGBM

1）原理

LightGBM在GBDT梯度提升数学框架之上，通过Leaf-wise + 直方图做工程加速。

【Leaf-wise】

GBDT：Level‑wise按层生长，一层层全部节点都分裂，无效计算多。
LightGBM：Leaf‑wise按叶子生长，每次只分裂增益最大的叶子；收敛快，但更容易过拟合，靠num_leaves约束。

【直方图】

GBDT：预排序，遍历全部特征值找分裂点，速度慢、内存大。
LightGBM：把连续特征离散成 bin 桶，只在桶边界分裂；直方图做差加速；速度大幅提升，微小精度损失

2）多特征的支持

对当前待分裂叶子节点，遍历全部特征（连续/类别），每个特征构建直方图，计算该特征所有候选分裂点的分裂增益，选出全局最大增益的【特征+分裂点】，用这个分裂把叶子切成左右两个叶子。

（3）LR

逻辑回归：在线性回归的输出外面套一个sigmoid函数，把实数映射到[0,1]，表示正类概率。

1）模型形式

2）损失函数

二元交叉熵

推导过程：

3）训练流程

2、实操代码

（1）训练代码

import pandas as pd
import json
import lightgbm as lgb
import numpy as np
from sklearn.linear_model import LogisticRegression
from scipy.sparse import lil_matrix

# ====================== 1.配置参数 ======================
CONTINUOUS_FEATURES = [f"I{i}" for i in range(1, 14)]
CATEGORY_FEATURES = [f"C{i}" for i in range(1, 27)]
# 博文给出clip阈值，95分位数截断
continous_clip = [20, 600, 100, 50, 64000, 500, 100, 50, 500, 10, 10, 10, 50]
cutoff = 200   # 类别特征出现次数>=200才保留，否则<unk>
train_path = "train_sub100w.txt"

# ====================== 2.读取原始数据 ======================
names = ["label"] + CONTINUOUS_FEATURES + CATEGORY_FEATURES
df_raw = pd.read_csv(train_path, sep="\t", names=names)

# 划分：前80w训练，后20w验证（时序切分，不shuffle）
df_train = df_raw.iloc[:800000].copy()
df_val = df_raw.iloc[800000:].copy()

# ====================== 3.处理连续特征 I1‑I13（clip + min‑max归一化） ======================
class ContinuousFeatureGenerator:
    def __init__(self, clip_list):
        self.clip_list = clip_list
        self.minv = None
        self.maxv = None

    def fit(self, df):
        arr = df[CONTINUOUS_FEATURES].fillna(0).to_numpy(dtype=np.float32)
        # clip截断
        for i in range(len(CONTINUOUS_FEATURES)):
            arr[:, i] = np.clip(arr[:, i], 0, self.clip_list[i])
        self.minv = np.min(arr, axis=0)
        self.maxv = np.max(arr, axis=0)

    def transform(self, df):
        arr = df[CONTINUOUS_FEATURES].fillna(0).to_numpy(dtype=np.float32)
        for i in range(len(CONTINUOUS_FEATURES)):
            arr[:, i] = np.clip(arr[:, i], 0, self.clip_list[i])
        # min‑max归一化 [0,1]
        denom = self.maxv - self.minv
        denom[denom < 1e-8] = 1.0
        norm = (arr - self.minv) / denom
        return norm

cont_gen = ContinuousFeatureGenerator(continous_clip)
cont_gen.fit(df_train)
X_train_cont = cont_gen.transform(df_train)
X_val_cont = cont_gen.transform(df_val)

# ====================== 4.处理类别特征 C1‑C26，每个field内部独立编码，cutoff过滤低频 ======================
class CategoryDictGenerator:
    def __init__(self):
        self.field_vocab = dict()  # key:col_name, value: {str:id}

    def fit(self, df):
        """只在训练集fit，统计频次，cutoff过滤"""
        for col in CATEGORY_FEATURES:
            cnt_series = df[col].fillna("<unk>").value_counts()
            valid_items = cnt_series[cnt_series >= cutoff].index.tolist()
            vocab = {}
            vocab["<unk>"] = 0
            for idx, item in enumerate(valid_items):
                vocab[item] = idx + 1
            self.field_vocab[col] = vocab

    def transform(self, df):
        out = np.zeros((len(df), len(CATEGORY_FEATURES)), dtype=np.int32)
        for fi, col in enumerate(CATEGORY_FEATURES):
            vocab = self.field_vocab[col]
            vals = df[col].fillna("<unk>").tolist()
            for ri, v in enumerate(vals):
                out[ri, fi] = vocab.get(v, vocab["<unk>"])
        return out

cate_gen = CategoryDictGenerator()
cate_gen.fit(df_train)
X_train_cate = cate_gen.transform(df_train)
X_val_cate = cate_gen.transform(df_val)

# 保存映射元数据，Java推理要用
with open("cate_field_vocab.json", "w", encoding="utf‑8") as f:
    json.dump(cate_gen.field_vocab, f, ensure_ascii=False)

y_train = df_train["label"].to_numpy()
y_val = df_val["label"].to_numpy()

# GBDT完整输入：归一化连续特征 + field内编码类别特征
X_train_gbdt = np.hstack([X_train_cont, X_train_cate])
X_val_gbdt = np.hstack([X_val_cont, X_val_cate])

# ====================== 5.训练LightGBM（只用于产出叶子，不直接用预测分数） ======================
lgb_train = lgb.Dataset(X_train_gbdt, label=y_train)
lgb_valid = lgb.Dataset(X_val_gbdt, label=y_val, reference=lgb_train)

params = {
    "objective": "binary",
    "metric": "binary_logloss,auc",
    "num_leaves": 32,
    "learning_rate": 0.05,
    "verbose": 1
}

def print_metrics_callback(env):
    iter_num = env.iteration
    eval_results = env.evaluation_result_list
    out = f"Iter {iter_num:2d} "
    # 元组：(数据集名,指标名,数值,是否越大越好)
    for ds_name, metric, val, _ in eval_results:
        out += f"{ds_name}_{metric}:{val:.6f} "
    print(out)

gbm = lgb.train(
    params,
    lgb_train,
    num_boost_round=20,
    valid_sets=[lgb_valid],
    callbacks=[print_metrics_callback]
)

gbm.save_model("gbdt_model.txt")


# 获取每棵树叶子id  shape:[样本数,树数量]
train_leaf = gbm.predict(X_train_gbdt, pred_leaf=True)
val_leaf = gbm.predict(X_val_gbdt, pred_leaf=True)
num_trees = gbm.num_trees()

# ====================== 6.构造叶子→LR全局下标映射（非常关键） ======================
# 每棵树局部叶子id，映射为LR稀疏矩阵全局index
leaf_mapping = {}
cur_idx = 0
for tree_no in range(num_trees):
    max_leaf = int(np.max(train_leaf[:, tree_no]))
    for local_leaf in range(0, max_leaf + 1):
        leaf_mapping[f"{tree_no}_{local_leaf}"] = cur_idx
        cur_idx += 1
leaf_feature_size = cur_idx

# 类别特征继续追加到后面
FIELD_MAX_SIZE = 1000
cate_feature_offset = leaf_feature_size
# 加上26个字段预留空间
total_feature_size = leaf_feature_size + len(CATEGORY_FEATURES) * FIELD_MAX_SIZE

meta_leaf = {
    "num_trees": num_trees,
    "leaf_mapping": leaf_mapping,
    "cate_feature_offset": cate_feature_offset,
    "field_max_size": FIELD_MAX_SIZE
}
with open("leaf_mapping.json", "w") as f:
    json.dump(meta_leaf, f)

# ====================== 7.构造LR稀疏输入：叶子one‑hot + 原始类别特征 ======================
def build_sparse_matrix(leaf_arr, cate_arr):
    n_sample = leaf_arr.shape[0]
    mat = lil_matrix((n_sample, total_feature_size), dtype=np.float32)
    for s in range(n_sample):
        # GBDT叶子特征
        for t in range(num_trees):
            local_id = int(leaf_arr[s, t])
            key = f"{t}_{local_id}"
            gid = leaf_mapping[key]
            mat[s, gid] = 1.0
        # 原始C类别特征，偏移之后写入
        for fi in range(len(CATEGORY_FEATURES)):
            cid = int(cate_arr[s, fi])
            mat[s, cate_feature_offset + fi * 1000 + cid] = 1.0
    return mat.tocsr()

X_lr_train = build_sparse_matrix(train_leaf, X_train_cate)
X_lr_val = build_sparse_matrix(val_leaf, X_val_cate)


# 训练LR
lr = LogisticRegression(penalty="l2", solver="saga", max_iter=600, verbose=2)
lr.fit(X_lr_train, y_train)

# 验证集评估
from sklearn.metrics import roc_auc_score
y_pred_val = lr.predict_proba(X_lr_val)[:, 1]
print("Val AUC =", roc_auc_score(y_val, y_pred_val))

# 在训练脚本最后添加
cont_meta = {
    "minv": cont_gen.minv.tolist(),
    "maxv": cont_gen.maxv.tolist(),
    "clip": continous_clip
}
with open("cont_norm_meta.json", "w", encoding="utf-8") as f:
    json.dump(cont_meta, f)

# 导出LR权重、bias
lr_meta = {
    "lr_weights": lr.coef_[0].tolist(),
    "lr_bias": float(lr.intercept_[0]),
    "total_feature_size": total_feature_size
}
with open("lr_meta.json", "w") as f:
    json.dump(lr_meta, f)

print("===全部导出物料===")
print("gbdt_model.txt")
print("cate_field_vocab.json")
print("leaf_mapping.json")
print("cont_norm_meta.json")
print("lr_meta.json")

（2）部署服务的代码

import json
import numpy as np
import lightgbm as lgb
from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn
import os

# ===================== 配置 =====================
CONTINUOUS_FEATURES = [f"I{i}" for i in range(1, 14)]
CATEGORY_FEATURES = [f"C{i}" for i in range(1, 27)]
continous_clip = [20, 600, 100, 50, 64000, 500, 100, 50, 500, 10, 10, 10, 50]
FIELD_MAX_SIZE = 1000

GBDT_MODEL_PATH = "./model/gbdt_model.txt"
LEAF_MAPPING_PATH = "./model/leaf_mapping.json"
LR_META_PATH = "./model/lr_meta.json"
CATE_VOCAB_PATH = "./model/cate_field_vocab.json"

# ===================== 全局加载资源 =====================
# 1.GBDT
gbdt_booster = lgb.Booster(model_file=GBDT_MODEL_PATH)

# 2.叶子映射
with open(LEAF_MAPPING_PATH, "r", encoding="utf-8") as f:
    leaf_meta = json.load(f)
leaf_mapping = leaf_meta["leaf_mapping"]
num_trees = leaf_meta["num_trees"]
cate_feature_offset = leaf_meta["cate_feature_offset"]

# 3.LR权重
with open(LR_META_PATH, "r", encoding="utf-8") as f:
    lr_meta = json.load(f)
lr_weight = np.array(lr_meta["lr_weights"], dtype=np.float64)
lr_bias = float(lr_meta["lr_bias"])
total_feature_size = lr_meta["total_feature_size"]

# 4.类别词典
with open(CATE_VOCAB_PATH, "r", encoding="utf-8") as f:
    cate_field_vocab = json.load(f)

# 5.预存训练集统计min/max
with open("./model/cont_norm_meta.json", "r", encoding="utf-8") as f:
    cont_norm_meta = json.load(f)
cont_min = np.array(cont_norm_meta["minv"])
cont_max = np.array(cont_norm_meta["maxv"])

app = FastAPI(title="GBDT+LR 推理服务")

# ===================== 请求模型 =====================
# 单条原始特征
class RawFeatureRequest(BaseModel):
    I1: float | None = None
    I2: float | None = None
    I3: float | None = None
    I4: float | None = None
    I5: float | None = None
    I6: float | None = None
    I7: float | None = None
    I8: float | None = None
    I9: float | None = None
    I10: float | None = None
    I11: float | None = None
    I12: float | None = None
    I13: float | None = None
    C1: str | None = None
    C2: str | None = None
    C3: str | None = None
    C4: str | None = None
    C5: str | None = None
    C6: str | None = None
    C7: str | None = None
    C8: str | None = None
    C9: str | None = None
    C10: str | None = None
    C11: str | None = None
    C12: str | None = None
    C13: str | None = None
    C14: str | None = None
    C15: str | None = None
    C16: str | None = None
    C17: str | None = None
    C18: str | None = None
    C19: str | None = None
    C20: str | None = None
    C21: str | None = None
    C22: str | None = None
    C23: str | None = None
    C24: str | None = None
    C25: str | None = None
    C26: str | None = None

# 批量请求结构
class BatchRawFeatureRequest(BaseModel):
    samples: list[RawFeatureRequest]

# ===================== 特征预处理函数（单条） =====================
def preprocess_raw_features(req: RawFeatureRequest):
    # 1.连续特征处理 clip + minmax
    cont_vals = []
    for idx, col in enumerate(CONTINUOUS_FEATURES):
        v = getattr(req, col, 0.0) or 0.0
        v = np.clip(v, 0, continous_clip[idx])
        cont_vals.append(v)
    cont_arr = np.array(cont_vals, dtype=np.float64)
    denom = cont_max - cont_min
    denom[denom < 1e-8] = 1.0
    cont_norm = (cont_arr - cont_min) / denom

    # 2.类别特征编码
    cate_ids = []
    for col in CATEGORY_FEATURES:
        raw_val = getattr(req, col)
        val = str(raw_val) if raw_val is not None else "<unk>"
        vocab = cate_field_vocab[col]
        cid = vocab.get(val, vocab["<unk>"])
        cate_ids.append(cid)

    gbdt_row = np.hstack([cont_norm, np.array(cate_ids, dtype=np.float64)])
    return gbdt_row, cate_ids

# ===================== 批量推理核心 =====================
def batch_predict(sample_list: list[RawFeatureRequest]):
    batch_gbdt_rows = []
    batch_cate_ids = []
    for s in sample_list:
        row, cids = preprocess_raw_features(s)
        batch_gbdt_rows.append(row)
        batch_cate_ids.append(cids)

    batch_arr = np.array(batch_gbdt_rows, dtype=np.float64)
    # LightGBM批量预测叶子
    leaf_batch = gbdt_booster.predict(batch_arr, pred_leaf=True)

    result_list = []
    for idx in range(len(sample_list)):
        leaf_indexes = leaf_batch[idx]
        cate_ids = batch_cate_ids[idx]
        lr_input = np.zeros(total_feature_size, dtype=np.float64)

        # 填充GBDT叶子onehot
        for tree_idx in range(num_trees):
            local_leaf = int(leaf_indexes[tree_idx])
            key = f"{tree_idx}_{local_leaf}"
            if key in leaf_mapping:
                pos = leaf_mapping[key]
                lr_input[pos] = 1.0

        # 填充原始类别onehot
        for fi, cid in enumerate(cate_ids):
            pos = cate_feature_offset + fi * FIELD_MAX_SIZE + cid
            lr_input[pos] = 1.0

        logit = float(np.dot(lr_input, lr_weight) + lr_bias)
        prob = 1.0 / (1.0 + np.exp(-logit))
        result_list.append({"logit": logit, "prob": prob})
    return result_list

# ===================== 接口 =====================
@app.post("/predict")
def inference(req: RawFeatureRequest):
    gbdt_input, cate_ids = preprocess_raw_features(req)
    res = batch_predict([req])[0]
    return {
        "code": 0,
        "msg": "success",
        "logit": res["logit"],
        "prob": res["prob"]
    }

@app.post("/batch_predict")
def batch_inference(req: BatchRawFeatureRequest):
    try:
        res_list = batch_predict(req.samples)
        return {
            "code": 0,
            "msg": "success",
            "result": res_list
        }
    except Exception as e:
        return {
            "code": -1,
            "msg": str(e),
            "result": []
        }

@app.get("/health")
def health():
    return {"status": "ok"}

if __name__ == "__main__":
    uvicorn.run(
        "main:app",
        host="0.0.0.0",
        port=8000,
        workers=1,
        log_level="info"
    )

（3）代码总结

1）核心目的

GBDT负责特征交叉，LR负责线性打分

2）GBDT+LR的逻辑链路

原始特征 → LightGBM 树分裂 → 样本落到叶子 → pred_leaf拿到叶子 ID → leaf_mapping 映射全局 id → one‑hot 稀疏特征 → LogisticRegression 训练。

3）为什么GBDT叶子ID可以当作特征

样本落到哪个叶子，就代表满足该叶子路径上一整套特征分裂条件；

把叶子ID转one-hot，就把GBDT学到的非线性交叉特征，转成线性模型可以用到的离散特征。

4）推理计算逻辑：

LightGBM 支持批量输出样本叶子 ID；LR 不调用 sklearn 库，遍历每个样本手动构造 0‑1 特征数组，直接通过公式logit=Wx + b 做向量点积计算，再 sigmoid 得到概率。

5）推理依赖训练导出物料

服务启动时全局加载训练阶段输出的全部静态文件：GBDT 模型、叶子映射表、LR 权重偏置、类别词表、连续特征归一化统计 (min/max、clip 阈值)，保证推理预处理逻辑与训练完全一致

3、服务请求示例

1）测试响应时间

一批300个请求时间：

curl.exe -o $null -s -w "总耗时=%{time_total}s`n" -X POST http://127.0.0.1:8000/batch_predict -H "Content-Type: application/json" -d '{"samples":[' + ((',' + '{"I1":10.0,"I2":20.0,"I3":5.0,"I4":1.0,"I5":1000.0,"I6":100.0,"I7":10.0,"I8":5.0,"I9":100.0,"I10":2.0,"I11":3.0,"I12":1.0,"I13":10.0,"C1":"abc","C2":"def","C3":"xyz","C4":null,"C5":"aaa","C6":"bbb","C7":"ccc","C8":"ddd","C9":"eee","C10":"fff","C11":"ggg","C12":"hhh","C13":"iii","C14":"jjj","C15":"kkk","C16":"lll","C17":"mmm","C18":"nnn","C19":"ooo","C20":"ppp","C21":"qqq","C22":"rrr","C23":"sss","C24":"ttt","C25":"uuu","C26":"vvv"},{"I1":5.0,"I2":10.0,"I3":2.0,"I4":3.0,"I5":500.0,"I6":50.0,"I7":5.0,"I8":2.0,"I9":50.0,"I10":1.0,"I11":1.0,"I12":2.0,"I13":5.0,"C1":"a1","C2":"b1","C3":"c1","C4":"d1","C5":"e1","C6":"f1","C7":"g1","C8":"h1","C9":"i1","C10":"j1","C11":"k1","C12":"l1","C13":"m1","C14":"n1","C15":"o1","C16":"p1","C17":"q1","C18":"r1","C19":"s1","C20":"t1","C21":"u1","C22":"v1","C23":"w1","C24":"x1","C25":"y1","C26":"z1"}')*150].TrimStart(',') + ']}'

请求时间：16ms左右

2）测试返回值

import time
import requests

URL = "http://127.0.0.1:8000/batch_predict"

# 两套样本模板
sample_a = {
    "I1": 10.0, "I2": 20.0, "I3": 5.0, "I4": 1.0, "I5": 1000.0,
    "I6": 100.0, "I7": 10.0, "I8": 5.0, "I9": 100.0, "I10": 2.0,
    "I11": 3.0, "I12": 1.0, "I13": 10.0,
    "C1": "abc", "C2": "def", "C3": "xyz", "C4": None,
    "C5": "aaa", "C6": "bbb", "C7": "ccc", "C8": "ddd",
    "C9": "eee", "C10": "fff", "C11": "ggg", "C12": "hhh",
    "C13": "iii", "C14": "jjj", "C15": "kkk", "C16": "lll",
    "C17": "mmm", "C18": "nnn", "C19": "ooo", "C20": "ppp",
    "C21": "qqq", "C22": "rrr", "C23": "sss", "C24": "ttt",
    "C25": "uuu", "C26": "vvv"
}

sample_b = {
    "I1": 5.0, "I2": 10.0, "I3": 2.0, "I4": 3.0, "I5": 500.0,
    "I6": 50.0, "I7": 5.0, "I8": 2.0, "I9": 50.0, "I10": 1.0,
    "I11": 1.0, "I12": 2.0, "I13": 5.0,
    "C1": "a1", "C2": "b1", "C3": "c1", "C4": "d1",
    "C5": "e1", "C6": "f1", "C7": "g1", "C8": "h1",
    "C9": "i1", "C10": "j1", "C11": "k1", "C12": "l1",
    "C13": "m1", "C14": "n1", "C15": "o1", "C16": "p1",
    "C17": "q1", "C18": "r1", "C19": "s1", "C20": "t1",
    "C21": "u1", "C22": "v1", "C23": "w1", "C24": "x1",
    "C25": "y1", "C26": "z1"
}

# 构造300条：A B 循环150次
samples = []
for _ in range(150):
    samples.append(sample_a)
    samples.append(sample_b)

payload = {"samples": samples}
headers = {"Content-Type": "application/json"}

if __name__ == "__main__":
    start = time.perf_counter()
    resp = requests.post(URL, json=payload, headers=headers)
    cost = time.perf_counter() - start

    print(f"请求耗时: {cost:.4f} 秒")
    print(f"HTTP状态码: {resp.status_code}")

    if resp.ok:
        data = resp.json()
        print(f"返回样本数量: {len(data['result'])}")
        # 打印前5条结果预览，避免刷屏
        print("====前5条预测结果====")
        for idx, item in enumerate(data["result"][:5]):
            print(f"[{idx}] prob={item['prob']:.6f}, logit={item['logit']:.4f}")
    else:
        print("响应异常：")
        print(resp.text)

返回结果：88ms。

请求耗时: 0.0886 秒
HTTP状态码: 200
返回样本数量: 300
====前5条预测结果====
[0] prob=0.088884, logit=-2.3273
[1] prob=0.094043, logit=-2.2652
[2] prob=0.088884, logit=-2.3273
[3] prob=0.094043, logit=-2.2652
[4] prob=0.088884, logit=-2.3273

如果是200条，就是62ms。

我在linux服务器上试了下，性能和本机的差不多。