解决text2vec模型加载报错：UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position。。。的方法

这个错误是加载模型失败，以下是官方的解决方法：

Answer: The strings (words) stored in your model are not valid utf8. By default, gensim decodes the words using the strict encoding settings, which results in the above exception whenever an invalid utf8 sequence is encountered.
The fix is on your side and it is to either:
a) Store your model using a program that understands unicode and utf8 (such as gensim). Some C and Java word2vec tools are known to truncate the strings at byte boundaries, which can result in cutting a multi-byte utf8 character in half, making it non-valid utf8, leading to this error.
b) Set the unicode_errors flag when running load_word2vec_model, e.g. load_word2vec_model(…, unicode_errors=‘ignore’). Note that this silences the error, but the utf8 problem is still there – invalid utf8 characters will just be ignored in this case.

给出了两种办法：

1. 可以用C和JAVA的相关工具去处文本中的无效编码字符；
2. 建议在load_word2vec_model函数中设置参数unicode_errors=‘ignore’，可以暂时忽略这个错误。

使用text2vec时报错如何修改？

text2vec是一个文本转向量库，封装了word2vec、bert等方法。

腾讯ai官方给出了两种word2vec：

• 轻量版腾讯词向量 百度云盘-密码:tawe 或 谷歌云盘，二进制文件，111M，是简化后的高频143613个词，每个词向量还是200维（跟原版一样），运行程序，自动下载到 ~/.text2vec/datasets/light_Tencent_AILab_ChineseEmbedding.bin
• 腾讯词向量-官方全量, 6.78G放到： ~/.text2vec/datasets/Tencent_AILab_ChineseEmbedding.txt，腾讯词向量主页：腾讯AI Lab，自然语言处理(NLP)研究 词向量下载地址：Embedding Datasets Download 更多

加载word2vec模型时出现UnicodeDecodeError: 'utf-8' codec can't decode bytes in position。。。错误的话

如果使用的轻量版（light_Tencent_AILab_ChineseEmbedding.bin）模型加载代码改成如下所示：

    w2v_model = Word2Vec(WORD2VEC_MODEL_PATH, w2v_kwargs={"unicode_errors": "ignore", "binary": True})

关键点在于添加初始化参数：

w2v_kwargs={"unicode_errors": "ignore", "binary": True}

使用Tencent_AILab_ChineseEmbedding版本，模型初始化代码改成这样：

    w2v_model = Word2Vec(WORD2VEC_MODEL_PATH, w2v_kwargs={"unicode_errors": "ignore"})

以上就是word2vec模型初始化报错的解决办法~

如果帮助到了你，就点个赞吧~

完结撒花~