MaxKB数据库优化：索引与查询优化
项目地址: https://gitcode.com/GitHub_Trending/ma/MaxKB 引言
在构建基于LLM大语言模型的知识库问答系统时，数据库性能直接决定了系统的响应速度和用户体验。MaxKB作为企业级智能体平台，在处理海量文档、向量数据和搜索查询时面临着严峻的数据库性能挑战。本文将深入探讨MaxKB的数据库架构，并提供专业的索引优化和查询优化策略。
MaxKB数据库架构概览
MaxKB采用PostgreSQL + pgvector的技术栈，主要包含以下核心表结构：
索引优化策略
1. 核心表索引设计
根据MaxKB的业务场景，以下是推荐的索引配置：
-- 知识库表索引
CREATE INDEX idx_knowledge_workspace ON knowledge(workspace_id);
CREATE INDEX idx_knowledge_type ON knowledge(type);
CREATE INDEX idx_knowledge_scope ON knowledge(scope);
CREATE INDEX idx_knowledge_folder ON knowledge(folder_id);
-- 文档表索引
CREATE INDEX idx_document_knowledge ON document(knowledge_id);
CREATE INDEX idx_document_status ON document(status);
CREATE INDEX idx_document_is_active ON document(is_active);
CREATE INDEX idx_document_type ON document(type);
-- 段落表索引
CREATE INDEX idx_paragraph_document ON paragraph(document_id);
CREATE INDEX idx_paragraph_knowledge ON paragraph(knowledge_id);
CREATE INDEX idx_paragraph_title ON paragraph(title);
CREATE INDEX idx_paragraph_status ON paragraph(status);
CREATE INDEX idx_paragraph_is_active ON paragraph(is_active);
CREATE INDEX idx_paragraph_position ON paragraph(position);
-- 向量表索引
CREATE INDEX idx_embedding_source_id ON embedding(source_id);
CREATE INDEX idx_embedding_source_type ON embedding(source_type);
CREATE INDEX idx_embedding_is_active ON embedding(is_active);
CREATE INDEX idx_embedding_knowledge ON embedding(knowledge_id);
CREATE INDEX idx_embedding_document ON embedding(document_id);
CREATE INDEX idx_embedding_paragraph ON embedding(paragraph_id);
2. 复合索引优化
对于频繁的联合查询，建议创建复合索引：
-- 知识库工作空间和类型复合索引
CREATE INDEX idx_knowledge_workspace_type ON knowledge(workspace_id, type);
-- 文档状态和活跃状态复合索引
CREATE INDEX idx_document_status_active ON document(status, is_active);
-- 段落文档和位置复合索引
CREATE INDEX idx_paragraph_doc_position ON paragraph(document_id, position);
3. 向量索引优化
pgvector支持多种索引类型，针对不同场景选择合适的索引：
| 索引类型 | 适用场景 | 优点 | 缺点 |
|---|---|---|---|
| IVFFlat | 中等规模数据集 | 查询速度快 | 精度相对较低 |
| HNSW | 大规模数据集 | 高精度、快速查询 | 内存占用高 |
| PQ | 超大规模数据集 | 内存效率高 | 精度损失 |
推荐配置：
-- 创建HNSW索引（推荐用于生产环境）
CREATE INDEX idx_embedding_vector_hnsw ON embedding
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 200);
-- 或者使用IVFFlat索引（适用于中等规模）
CREATE INDEX idx_embedding_vector_ivf ON embedding
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);
查询优化实战
1. 向量搜索查询优化
MaxKB的核心搜索功能基于以下SQL模板：
SELECT
paragraph_id,
comprehensive_score,
comprehensive_score as similarity
FROM
(
SELECT DISTINCT ON
("paragraph_id") ( 1 - distince ),* ,(1 - distince) AS comprehensive_score
FROM
( SELECT *, ( embedding.embedding::vector(%s) <=> %s ) AS distince
FROM embedding ${embedding_query}
ORDER BY distince) TEMP
ORDER BY
paragraph_id,
distince
) DISTINCT_TEMP
WHERE comprehensive_score > %s
ORDER BY comprehensive_score DESC
LIMIT %s
优化建议：
索引层面：
- 确保embedding字段有合适的向量索引
- 为paragraph_id创建索引以加速DISTINCT操作
查询层面：
-- 使用CTE优化复杂查询
WITH ranked_embeddings AS (
SELECT
paragraph_id,
(1 - (embedding.embedding::vector(%s) <=> %s)) as similarity,
ROW_NUMBER() OVER (PARTITION BY paragraph_id ORDER BY (embedding.embedding::vector(%s) <=> %s)) as rn
FROM embedding
WHERE ${embedding_conditions}
)
SELECT
paragraph_id,
similarity as comprehensive_score
FROM ranked_embeddings
WHERE rn = 1 AND similarity > %s
ORDER BY similarity DESC
LIMIT %s;
2. 混合搜索优化
混合搜索结合了向量搜索和全文搜索：
SELECT
paragraph_id,
comprehensive_score,
comprehensive_score AS similarity
FROM
(
SELECT DISTINCT ON
( "paragraph_id" ) ( 1 - distince + ts_similarity ) as similarity, *,
(1 - distince + ts_similarity) AS comprehensive_score
FROM
(
SELECT
*,
(embedding.embedding::vector(%s) <=> %s) as distince,
(ts_rank_cd( embedding.search_vector, websearch_to_tsquery('simple', %s ), 32 )) AS ts_similarity
FROM
embedding ${embedding_query}
ORDER BY distince
) TEMP
ORDER BY
paragraph_id,
similarity DESC
) DISTINCT_TEMP
WHERE
comprehensive_score > %s
ORDER BY
comprehensive_score DESC
LIMIT %s
优化策略：
索引优化：
-- 为search_vector创建GIN索引
CREATE INDEX idx_embedding_search_vector ON embedding USING GIN (search_vector);
-- 创建复合索引加速混合查询
CREATE INDEX idx_embedding_mixed_search ON embedding
USING GIN (search_vector, (embedding::vector));
查询重写：
-- 使用窗口函数优化混合搜索
WITH scored_embeddings AS (
SELECT
paragraph_id,
(1 - (embedding::vector(%s) <=> %s)) as vector_score,
ts_rank_cd(search_vector, websearch_to_tsquery('simple', %s), 32) as text_score,
(1 - (embedding::vector(%s) <=> %s) +
ts_rank_cd(search_vector, websearch_to_tsquery('simple', %s), 32)) as combined_score,
ROW_NUMBER() OVER (PARTITION BY paragraph_id ORDER BY (1 - (embedding::vector(%s) <=> %s)) DESC) as rn
FROM embedding
WHERE ${embedding_conditions}
)
SELECT
paragraph_id,
combined_score as comprehensive_score
FROM scored_embeddings
WHERE rn = 1 AND combined_score > %s
ORDER BY combined_score DESC
LIMIT %s;
3. 分页查询优化
对于大数据集的分页查询，避免使用OFFSET：
-- 传统分页（不推荐）
SELECT * FROM paragraph
WHERE knowledge_id = %s
ORDER BY position
OFFSET 1000 LIMIT 50;
-- 优化后的分页（推荐）
SELECT * FROM paragraph
WHERE knowledge_id = %s AND position > (SELECT position FROM paragraph WHERE id = %s)
ORDER BY position
LIMIT 50;
性能监控与调优
1. 关键性能指标监控
-- 查询缓存命中率
SELECT
sum(heap_blks_read) as heap_read,
sum(heap_blks_hit) as heap_hit,
sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) as ratio
FROM pg_statio_user_tables;
-- 索引使用情况
SELECT
schemaname,
relname,
indexrelname,
idx_scan,
idx_tup_read,
idx_tup_fetch
FROM pg_stat_user_indexes
ORDER BY idx_scan DESC;
-- 慢查询分析
SELECT
query,
calls,
total_time,
rows,
total_time/calls as avg_time,
rows/calls as avg_rows
FROM pg_stat_statements
ORDER BY total_time DESC
LIMIT 10;
2. 连接池配置
推荐使用PgBouncer进行连接池管理：
[databases]
maxkb = host=localhost dbname=maxkb user=maxkb_user
[pgbouncer]
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 20
reserve_pool_size = 5
3. 定期维护任务
# 每日 vacuum 分析
vacuumdb -z -a
# 每周 reindex
reindexdb -a
# 每月统计信息更新
analyzedb -a
实战案例：千万级数据优化
假设知识库包含1000万条段落数据，优化方案如下：
1. 分区策略
-- 按知识库ID进行分区
CREATE TABLE paragraph_ptn (
LIKE paragraph INCLUDING ALL
) PARTITION BY HASH (knowledge_id);
-- 创建16个分区
CREATE TABLE paragraph_p0 PARTITION OF paragraph_ptn FOR VALUES WITH (MODULUS 16, REMAINDER 0);
CREATE TABLE paragraph_p1 PARTITION OF paragraph_ptn FOR VALUES WITH (MODULUS 16, REMAINDER 1);
-- ... 创建剩余14个分区
2. 批量处理优化
# 批量插入优化示例
from django.db import transaction
@transaction.atomic
def batch_create_paragraphs(paragraphs_data, batch_size=1000):
paragraphs = []
for i, data in enumerate(paragraphs_data):
paragraph = Paragraph(**data)
paragraphs.append(paragraph)
if i % batch_size == 0 and i > 0:
Paragraph.objects.bulk_create(paragraphs)
paragraphs = []
if paragraphs:
Paragraph.objects.bulk_create(paragraphs)
3. 异步处理架构
总结
MaxKB数据库优化是一个系统工程，需要从索引设计、查询优化、架构设计等多个层面综合考虑。通过合理的索引策略、查询重写、分区设计和异步处理，可以显著提升系统性能，支撑千万级知识库数据的快速检索。
关键优化要点：
- 为高频查询字段创建合适的索引
- 使用复合索引减少回表操作
- 选择合适的向量索引类型（HNSW推荐）
- 优化分页查询，避免OFFSET性能问题
- 实施分区策略处理超大规模数据
- 建立完善的监控和维护体系
通过以上优化策略，MaxKB可以在保持高精度的同时，实现毫秒级的搜索响应，为用户提供流畅的知识问答体验。
创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
