How to Score 'Interests' | 22-Year Life Log Research (Paper Explanation ①) [Episode 4]

For 22 years since January 10, 2004,
I have been recording my diary and chat logs.
The total character count is approximately 15.5 million characters.

In Episode 3, I wrote about visualizing 'changes in interests' as research that I compiled into an academic paper and published
.

This time, it's the behind-the-scenes.
I will organize how this massive amount of text data is converted into an 'interest score'.

■ Why is 'scoring' necessary?

If you read back through 22 years of logs, you can find episodes.
However, you cannot see the overall trends.

'I feel like I talked about friends a lot in my 20s.' (Just a feeling)
'I feel like topics about
children increased in my 30s.' (Just a feeling)

I visualized and scored these feelings to perform data analysis.


You get the pleasure of searching 22 years of memories with just one line of code and visualizing them with standard scores.

■ Raw logs used for analysis

I used personal life logs (chat logs, diaries, etc., totaling approximately 15.5 million characters) that I recorded myself over 22 years from age 18 to 40 (2004–2025).

In terms of 100,000-character novels, this is equivalent to 155 novels. This life log is organized by date.

　対象：2004/1/10～2025/8/20（7863日分）
　平均：1,966字/日
　最大：77,308字/日
　欠損：16.56%
　※欠測日のスコアについては，前７日間の加重移動平均により補正。

Note that the volume of logs varies greatly from day to day.

■ Raw log integration work (preprocessing)

My life logs spanned about 4,000 files, and the storage formats were inconsistent.
Therefore, I used 21 types of code in Python 3 to integrate the raw log data into
about 30 million cells (355,000 rows x 80 columns) into one file. (It was
extremely difficult) Furthermore, I separated the raw logs into words using 'morphological analysis (Janome)' (this is difficult, so I will omit the details).

# 実際に使用したコードの一部を掲載します（簡略化）。
# chat_log_parser.py
import re # ★この行をファイルの先頭に追加します★
file_path = "20050112.txt"
try:
    with open(file_path, 'r', encoding='cp932') as f:
        first_line = f.readline()
        log_date = first_line[8:18] # 例: '2005/01/12'
        print(f"抽出した日付: {log_date}\n")
        print("----- ここから各メッセージの抽出結果 -----")
        all_chat_messages = []
        chat_line_pattern = re.compile(r'^(\d{2}:\d{2})\s+\((.*?)\)\s*(.*)$')
        for line in f:
            stripped_line = line.strip() # 行頭・行末の空白や改行を削除
            if not stripped_line: # 空白行はスキップ
                continue
            match = chat_line_pattern.match(stripped_line)
            if match:
                time = match.group(1)      # グループ1: 時刻
                speaker = match.group(2)   # グループ2: 話者
                message = match.group(3)   # グループ3: メッセージ本文
                all_chat_messages.append({
                    'Date': log_date,
                    'Time': time,
                    'Speaker': speaker,
                    'Message': message
                })
                print(f"日付: {log_date}, 時刻: {time}, 話者: {speaker}, メッセージ: {message}")
            else:
                print(f"警告: パターンにマッチしない行をスキップしました -> {stripped_line}")
        print(f"\n全{len(all_chat_messages)}件のメッセージをリストに格納しました。")
except FileNotFoundError:
    print(f"エラー: ファイル '{file_path}' が見つかりません。")
except Exception as e:
    print(f"ファイルの読み込み中にエラーが発生しました: {e}")

■ Interest scoring

As a keyword dictionary for interest items, I created an interest keyword dictionary using:
・Classification Vocabulary Table (National Institute for Japanese Language and Linguistics, 2018)
・Japanese WordNet (National Institute of Information and Communications Technology, 2010)
・Science and Technology Term Morphological Analysis Dictionary

Using this, I calculated the daily appearance frequency of each item keyword in the morphologically analyzed logs and set it as the 'interest score'.

(Word count:
28,593 words. Score: 71,219 points)

# 実際に使用したコードの一部を掲載します（簡略化）。
import pandas as pd
import re
main_data_path = "C:/Users/●●/chatlog_lifelog/Python/integrated_lifelog_data.csv"
dictionary_path = "C:/Users/●●/99_detail_emotion_word.csv"
try:
    df_integrated = pd.read_csv(main_data_path, encoding='utf-8-sig')
    df_dictionary = pd.read_csv(dictionary_path, encoding='utf-8-sig')
    print("統合データと辞書ファイルを読み込みました。")
    print(f"統合データの行数: {len(df_integrated)}行")
    print(f"辞書データの行数: {len(df_dictionary)}行")
    df_integrated['日付'] = pd.to_datetime(df_integrated['日付'], errors='coerce')
    df_dictionary['キーワード'] = df_dictionary['キーワード'].astype(str).fillna('')
except FileNotFoundError as e:
    print(f"エラー: 必要なファイルが見つかりません。パスを確認してください: {e.filename}")
    exit()
except Exception as e:
    print(f"データの読み込み中にエラーが発生しました: {e}")
    exit()
print("\n--- ワード辞書をPythonの辞書形式に変換します ---")
keyword_dict = df_dictionary.groupby('特定ジャンル')['キーワード'].apply(list).to_dict()
text_columns = ['総合テキスト']
print("\n--- 特定キーワード特化スコアの作成を開始します ---")
for genre, keywords in keyword_dict.items():
    pattern = '|'.join(keywords)
        for text_col in text_columns:
        score_col_name = f"{text_col}_{genre}_スコア"
        text_series = df_integrated[text_col].fillna('')
        df_integrated[score_col_name] = text_series.str.count(pattern, flags=re.IGNORECASE)
        print(f"'{genre}'関連スコアの作成が完了しました。")
print("\n--- 新しく作成されたスコア列の最初の5行（例） ---")
score_cols_to_show = [
    '日付', '総合テキスト_ごみ・廃棄物・一廃・産廃_スコア', '総合テキスト_インフレ・値上・物価高騰・物価上昇_スコア'
]
existing_cols_to_show = [col for col in score_cols_to_show if col in df_integrated.columns]
print(df_integrated[existing_cols_to_show].head())
output_path_with_keywords = main_data_path.replace('.csv', '_with_detail_emotion_scores.csv')
df_integrated.to_csv(output_path_with_keywords, index=False, encoding='utf-8-sig')
print(f"\n新しく追加されたスコアを含むデータを '{output_path_with_keywords}' に保存しました。")
print("このデータを今後の分析にご利用ください。")

■ Design of interest categories

There is no meaning in simply aggregating words as they are.
Therefore, I designed interest categories (items) in advance.
The interest categories (items) were set to be the same items as a certain city's citizen survey. (This is because it would then be possible to compare them with the citizen survey.)

Each word is linked to these categories.

■ Standardization and Deviation Scoring

The amount of logs varies from day to day.
Naturally, if you write a lot, your score will be higher.
Therefore, I used a method called "Constant-Sum Scaling" to calculate the composition ratio of each category relative to all words for that day.
(In statistics, this is called "
standardization")

Furthermore, I compared this with my 22-year history and converted it into a "deviation score".
This highlights things like "how special today was in my 22-year history".
For example, my "

August 21, 2008" interest in "friends" had a deviation score of 71.Looking back at the logs for that day,
it seems it was the
day I passed the final civil service exam, and I was talking about it with friends.(By the way, the "family" deviation score for this day was 38)

「友人」の関心偏差値71の日のログ抜粋（当時23歳）

--------2008/08/21 00:00:00 ログを開始
2008/8/21　木　19:51　私　　○○市に受かったぞ!
2008/8/21　木　19:52　私　　来年から川崎に住みます
2008/8/21　木　19:52　友人　おお
2008/8/21　木　19:52　友人　やったな
2008/8/21　木　19:52　友人　川崎ってどこだ？
2008/8/21　木　19:52　友人　横浜の辺り？
2008/8/21　木　19:52　私　　横浜と東京の間。
2008/8/21　木　19:52　私　　ギリギリ神奈川県
2008/8/21　木　19:52　友人　なるほど
--------2008/08/22 00:00:00 ログを終了

※匿名加工済

■ Limitations

Naturally, this method has its limitations.

• Arbitrariness of word dictionary design

• Ignoring context

• Handling of irony and metaphors

• Data missing periods, etc.

Therefore, I position this as an exploratory visualization method rather than "rigorous psychometrics."

■ Possibility of Raw Log Back-referencing (Traceability)

As a score validity verification, I back-referenced the raw logs for specific periods with high interest deviation scores, and checked the raw logs for the 7 days before and after the target date using the date ID, raw log ID, emotion score ID, and each interest score ID as keys.

As I mentioned in Episode 1, for example, the week of August 4, 2009, was an "friend" deviation score of 91 outlier, and in 22 years, there were only 11 weeks with a deviation score of 80 or higher.

This is statistically just an anomaly (outlier), but
in terms of life logs, it is by no means just a number, it is "
the moment that unearthed a passionate conversation with a close friend I had forgotten, 20 years later".

On days with high deviation scores, very intense conversations (e.g., "a goosebump-inducing development") were taking place.
In other words, by utilizing raw log back-referencing (traceability method), it is possible to obtain clues for tracking contextual factors.

Furthermore, it is possible to avoid 'negative outliers' by extracting high interest + high emotion (positive statements).

Conventional statistical analysis (macro analysis) previously removed anomalies as 'outliers', and reverse referencing of raw logs was impossible. However, in personal logs (N=1),

anomalies are moments when interest spikes, which are highly valuable as starting points for behavioral change and personalization. It was suggested that such anomalies can serve as starting points for exploration and understanding factors, and that these anomalies themselves can become starting points for exploration and factor understanding.

■ Future Outlook

While this is still at the hypothesis stage, the advancement of RAG (Retrieval-Augmented Generation) and AI agents is highly compatible with the trend of treating life logs as searchable personal resources (Gurrin et al., 2014), and it is expected to serve as a foundational technology that enhances the reproducibility and cumulative potential of the N=1 research presented in this paper. Furthermore, by developing the traceability through reverse referencing of daily raw logs attempted in this study, I believe it is possible to conceive a framework for intervention design that combines long-term storage of N=1 data with high-precision context extraction. For example, in the future, under the concept that


'logs are assets'

, interventions that encourage an individual's intrinsic motivation by digging into the 'source of interest' in N=1 logs contextually using RAG and other methods, rather than just keyword analysis, can be considered. I believe that combining such AI-driven intervention models with the model in this paper and empirically examining their effects on maintaining and expanding people's interests could become one of the important future challenges in social informatics.

■ Next Time [Part 5]

Next time, we will cover the calculation method for emotion scores, which is another axis separate from 'interest'. Interest and emotion may seem similar, but they actually behave in completely different ways.

▼ Part 5: How to Score 'Emotion'

▼ Starting Point of This Series [Part 1]

　https://note.com/lifelog_lab/n/n2f55c09c19bb

If you would like, it would be a great encouragement if you could follow and like.

▼ Japan Life Log Research Institute | Official Website

・Official Website URL:　
https://wide-spirit-498.notion.site/346b4484a04880e8b4b7dbd94f9e602a