Python program to create training and evaluation data for Keras environments

I have created a Python program to generate training and evaluation data for Keras environments.

Previously, I introduced a Python program for creating training and evaluation data for Neural Network Console.

However, I discovered an issue with how the evaluation data was being created.

Therefore, I have prepared a revised Python program for creating AI training and evaluation data.

Compared to the previous version, the main changes are as follows:

• Changes

  • Bug Fixes

    • Modified to reference only the training data when calculating the mean and standard deviation used for standardizing evaluation data

  • Feature Additions

    • Enabled importing of external files containing label generation functions

  • Other

    • Changed the data source for stock prices, etc., to Yahoo Finance only

    • Added trading volume to the explanatory variables

    • Removed stochastics from the selectable technical indicators

    • Revised the command-line arguments

    • Removed the header generation feature for Neural Network Console

Source code for the Python program to create AI training and evaluation data

The source code for the Python program to create AI training and evaluation data for Keras environments is provided below.

# -*- coding: utf-8 -*-

import numpy as np
import pandas as pd
import datetime
import yfinance as yf
import argparse
import talib as ta
from sklearn.preprocessing import StandardScaler
import joblib
import os
import sys
import importlib.util
from curl_cffi import requests


# 株価データをダウンロード
def DlStockData(Ticker, StartDate, EndDate, Columns):
    # シンボル変換
    if 'N225' == Ticker: TmpTicker = '^N225'
    elif 'DOW' == Ticker: TmpTicker = '^DJI'
    elif 'SP500' == Ticker: TmpTicker = '^GSPC'
    elif 'NAS' == Ticker: TmpTicker = '^IXIC'
    elif 'FTSE' == Ticker: TmpTicker = '^FTSE'
    elif 'HSI' == Ticker: TmpTicker = '^HSI' # 香港ハンセン株価指数
    elif 'SHA' == Ticker: TmpTicker = '000001.SS' # 上海総合指数
    elif 'VIX' == Ticker: TmpTicker = '^VIX'
    elif 'USD' == Ticker: TmpTicker = 'USDJPY=X'
    elif 'EUR' == Ticker: TmpTicker = 'EURJPY=X'
    elif 'CNY' == Ticker: TmpTicker = 'CNYJPY=X' # 元(中国)
    elif 'IRX' == Ticker: TmpTicker = '^IRX' # 米13週国債
    elif 'TNX' == Ticker: TmpTicker = '^TNX' # 米10年国債
    elif 'SOX' == Ticker: TmpTicker = '^SOX' # フィラデルフィア半導体株指数
    elif 'WTI' == Ticker: TmpTicker = 'CL=F' # W & T Offshore, Inc.
    else: TmpTicker = Ticker + '.T'

    # 最終日が1日前にズレる問題を補正
    EndDate4Yahoo = (pd.to_datetime(EndDate) + datetime.timedelta(days = 1)).strftime('%Y-%m-%d')

    # 株価データをダウンロード
    # エラー対策でSessionを追加: YFRateLimitError('Too Many Requests. Rate limited. Try after a while.')
    Session = requests.Session(impersonate = 'chrome')
    Data = yf.download(TmpTicker, StartDate, EndDate4Yahoo, auto_adjust = True, multi_level_index = False, progress = False, session = Session)

   # 株価データのマルチカラムをシングルカラムに変更
    if 1 < Data.columns.nlevels:
        TmpColumns = []

        for ColName in Data.columns.values:
            TmpColumns.append(ColName[0])

        Data.columns = TmpColumns

    # Datetimeインデックスを日付けのみに変更
    Data.index = Data.index.date

    # カラムの設定
    # カラムが1個 → Close
    # カラムが4個 → Open, High, Low, Close
    # カラムが5個 → Open, High, Low, Close, Volume
    TmpColumns = ['Close']
    if 1 < len(Columns): TmpColumns = ['Open', 'High', 'Low'] + TmpColumns
    if 5 == len(Columns): TmpColumns += ['Volume']

    # カラムの順序を入れ替え
    # Close, High, Low, Open, Volume → Open, High, Low, CLose, Volume
    Data = Data.reindex(columns = TmpColumns)

    # Volumeが0の対策 → 翌営業日のVolumeを当日のVolumeにコピー
    if 5 == len(Columns):
        for i in reversed(range(len(Data) - 1)):
            if 0 == Data.iloc[i, Data.columns.get_loc('Volume')]:
                Data.iloc[i, Data.columns.get_loc('Volume')] = Data.iloc[i + 1, Data.columns.get_loc('Volume')]

    # カラム名を変更
    Data = Data.set_axis(Columns, axis = 1)

    return Data


# 単純移動平均(短期、中期、長期)
def FuncSMA(Data, Ticker, Columns):
    WinShort: int = 5
    WinMiddle: int = 25
    WinLong: int = 75

    RefCol = Ticker + '_C'

    SMA_S = Data[RefCol].rolling(window = WinShort).mean().to_frame(Columns[0])
    SMA_M = Data[RefCol].rolling(window = WinMiddle).mean().to_frame(Columns[1])
    SMA_L = Data[RefCol].rolling(window = WinLong).mean().to_frame(Columns[2])

    return pd.concat([SMA_S, SMA_M, SMA_L], axis = 1)


# 単純移動平均(短期、長期、差分)
def FuncSMADiff(Data, Ticker, Columns):
    WinShort: int = 5
    WinLong: int = 75

    RefCol = Ticker + '_C'

    SMA_S = Data[RefCol].rolling(window = WinShort).mean()
    SMA_L = Data[RefCol].rolling(window = WinLong).mean()
    SMA_Diff = SMA_S - SMA_L

    # SeriesをDataFrameに変換
    SMA_S = SMA_S.to_frame(Columns[0])
    SMA_L = SMA_L.to_frame(Columns[1])
    SMA_Diff = SMA_Diff.to_frame(Columns[2])

    return pd.concat([SMA_S, SMA_L, SMA_Diff], axis = 1)


# 単純移動平均
# A: (短期 - 中期) / 中期
# B: (中期 - 長期) / 長期
def FuncSMARatio(Data, Ticker, Columns):
    SMA = FuncSMA(Data, Ticker, ['SMA_S', 'SMA_M', 'SMA_L'])

    Result = pd.DataFrame({
        Columns[0]: (SMA['SMA_S'] - SMA['SMA_M']) / SMA['SMA_M'],
        Columns[1]: (SMA['SMA_M'] - SMA['SMA_L']) / SMA['SMA_L']
    })

    return Result


# ボリンジャーバンド
def FuncBB(Data, Ticker, Columns):
    WinBB: int = 20

    RefCol = Ticker + '_C'

    SMABB = Data[RefCol].rolling(window = WinBB).mean()
    StdDevBB = Data[RefCol].rolling(window = WinBB).std(ddof = 0)

    BBM2Sig = (SMABB - 2 * StdDevBB).to_frame(Columns[0])
    BBM1Sig = (SMABB - StdDevBB).to_frame(Columns[1])
    BBSMA = SMABB.to_frame(Columns[2])
    BBP1Sig = (SMABB + StdDevBB).to_frame(Columns[3])
    BBP2Sig = (SMABB + 2 * StdDevBB).to_frame(Columns[4])

    return pd.concat([BBM2Sig, BBM1Sig, BBSMA, BBP1Sig, BBP2Sig], axis = 1)


# ボリンジャーバンド(バンド幅、バンド幅の割合)
def FuncBBWidth(Data, Ticker, Columns):
    BB = FuncBB(Data, Ticker, ['BB_M2Sig', 'BB_M1Sig', 'BB_SMA', 'BB_P1Sig', 'BB_P2Sig'])

    Result = pd.DataFrame({
        Columns[0]: BB['BB_P2Sig'] - BB['BB_M2Sig'],
        Columns[1]: (BB['BB_P2Sig'] - BB['BB_M2Sig']) / BB['BB_SMA']
    })

    return Result


# RSI
def FuncRSI(Data, Ticker, Columns):
    SpanRSI: int = 14

    RefCol = Ticker + '_C'

    # RSI
    RSI = ta.RSI(Data[RefCol], timeperiod = SpanRSI).to_frame(Columns[0])

    return RSI


# MACD
def FuncMACD(Data, Ticker, Columns):
    SpanShort: int = 12
    SpanLong: int = 26
    WinSignal: int = 9

    RefCol = Ticker + '_C'

    EMAShort = Data[RefCol].ewm(span = SpanShort).mean()
    EMALong = Data[RefCol].ewm(span = SpanLong).mean()

    # MACD
    MACD = (EMAShort - EMALong).to_frame(Columns[0])

    return MACD


# 一目均衡表
def FuncICHIMOKU(Data, Ticker, NoLagFlag, Columns):
    WinBase: int = 26
    WinConv: int = 9
    Span1Shift: int = 25
    WinSpan2: int = 52
    Span2Shift: int = 25
    DelayShift: int = -25

    RefCol = Ticker + '_C'
    RefColHigh = Ticker + '_H'
    RefColLow = Ticker + '_L'

    # 基準線
    MaxBaseLine = Data[RefColHigh].rolling(WinBase).max()
    MinBaseLine = Data[RefColLow].rolling(WinBase).min()

    BaseLineData = ((MaxBaseLine + MinBaseLine) / 2).to_frame(Columns[0])

    # 転換線
    MaxConvLine = Data[RefColHigh].rolling(WinConv).max()
    MinConvLine = Data[RefColLow].rolling(WinConv).min()

    ConvLineData = ((MaxConvLine + MinConvLine) / 2).to_frame(Columns[1])

    # 先行スパン1
    Span1Data = ((BaseLineData[Columns[0]] + ConvLineData[Columns[1]]) / 2).shift(Span1Shift).to_frame(Columns[2])

    # 先行スパン2
    MaxSpan2Line = Data[RefColHigh].rolling(WinSpan2).max()
    MinSpan2Line = Data[RefColLow].rolling(WinSpan2).min()

    Span2Data = ((MaxSpan2Line + MinSpan2Line) / 2).shift(Span2Shift).to_frame(Columns[3])

    # 遅行スパン
    LagginSpanData = Data[RefCol].shift(DelayShift).to_frame('ICHI_Lag')

    # 遅行スパンのNaNに最後の終値をコピー
    LastCloseRow: int =  len(LagginSpanData) + DelayShift - 1
    for i in range(len(LagginSpanData) + DelayShift, len(LagginSpanData)):
        LagginSpanData.iat[i, 0] = LagginSpanData.iat[LastCloseRow, 0]

    if NoLagFlag:
        # 遅行スパンなし
        return pd.concat([BaseLineData, ConvLineData, Span1Data, Span2Data], axis = 1)
    else:
        # 遅行スパンあり
        return pd.concat([BaseLineData, ConvLineData, Span1Data, Span2Data, LagginSpanData], axis = 1)


# ストキャスティクス
def FuncStochastic(Data, Ticker, Columns):
    SpanK: int = 14
    SpanD: int = 3

    RefCol = Ticker + '_C'
    RefColHigh = Ticker + '_H'
    RefColLow = Ticker + '_L'

    FastK, FastD = ta.STOCHF(Data[RefColHigh], Data[RefColLow], Data[RefCol], fastk_period = SpanK, fastd_period = SpanD)

    # SeriesをDataFrameに変換
    FastK = FastK.to_frame(Columns[0])
    FastD = FastD.to_frame(Columns[1])

    return pd.concat([FastK, FastD], axis = 1)


# モメンタム
def FuncMomentum(Data, Ticker, Columns):
    Span: int = 5

    RefCol = Ticker + '_C'

    Momentum = ta.MOM(Data[RefCol], timeperiod = Span)

    # SeriesをDataFrameに変換
    Momentum = Momentum.to_frame(Columns[0])

    return Momentum


# 移動平均乖離率
def FuncSMADeviation(Data, Ticker, Columns):
    Span: int = 5

    RefCol = Ticker + '_C'

    SMA = Data[RefCol].rolling(window = Span).mean()
    SMADev = ((Data[RefCol] - SMA) / SMA) * 100

    # SeriesをDataFrameに変換
    SMADev = SMADev.to_frame(Columns[0])

    return SMADev


# サイコロジカルライン
def FuncPsychological(Data, Ticker, Columns):
    Span: int = 12

    RefCol = Ticker + '_C'

    UpDays = (Data[RefCol].diff() > 0).astype(int)
    Psychological = UpDays.rolling(window = Span).sum() / Span * 100

    # SeriesをDataFrameに変換
    Psychological = Psychological.to_frame(Columns[0])

    return Psychological


# ATR
def FuncATR(Data, Ticker, Columns):
    Span: int = 14

    RefCol = Ticker + '_C'
    RefColHigh = Ticker + '_H'
    RefColLow = Ticker + '_L'

    Atr = ta.ATR(Data[RefColHigh], Data[RefColLow], Data[RefCol], timeperiod = Span)

    # SeriesをDataFrameに変換
    Atr = Atr.to_frame(Columns[0])

    return Atr


# ADX
def FuncADX(Data, Ticker, Columns):
    Span: int = 14

    RefCol = Ticker + '_C'
    RefColHigh = Ticker + '_H'
    RefColLow = Ticker + '_L'

    Adx = ta.ADX(
        Data[RefColHigh],
        Data[RefColLow],
        Data[RefCol],
        timeperiod = Span
    )

    # SeriesをDataFrameに変換
    Adx = Adx.to_frame(Columns[0])

    return Adx


# Volatility
def FuncVolatility(Data, Ticker, Columns):
    Span: int = 5

    RefCol = Ticker + '_C'

    Vol = Data[RefCol].pct_change().rolling(Span).std()

    # SeriesをDataFrameに変換
    Vol = Vol.to_frame(Columns[0])

    return Vol


# 直近高値からの下落率
def FuncDropRate(Data, Ticker, Columns):
    Span: int = 20

    RefCol = Ticker + '_C'

    DropRate = Data[RefCol] / Data[RefCol].rolling(Span).max()

    # SeriesをDataFrameに変換
    DropRate = DropRate.to_frame(Columns[0])

    return DropRate


# 相対強度(日経平均株価/ダウ平均株価)
def FuncN225DOW(StartDate, EndDate, Columns):
    ColN225 = ['N225_C']
    ColDOW = ['DOW_C']

    DataN225 = DlStockData('N225', StartDate, EndDate, ColN225)
    DataDOW = DlStockData('DOW', StartDate, EndDate, ColDOW)

    # インデックスを日付型に変換
    DataN225.index = pd.to_datetime(DataN225.index)
    DataDOW.index = pd.to_datetime(DataDOW.index)

    # インデックスを基準にDataFrameを合体
    TmpData = pd.concat([DataN225, DataDOW], axis = 1)

    # Ratio = TmpData[ColN225] / TmpData[ColDOW]
    Ratio = TmpData.iloc[:, 0] / TmpData.iloc[:, 1]

    # SeriesをDataFrameに変換
    Ratio = Ratio.to_frame(Columns[0])

    return Ratio


# n日間の騰落率(n = 3, 5)
def FuncRateChange(Data, Ticker, Columns):
    Span_S: int = 3
    Span_M: int = 5

    RefCol = Ticker + '_C'

    Data_S = Data[RefCol].pct_change(Span_S)
    Data_M = Data[RefCol].pct_change(Span_M)

    Result = pd.DataFrame({
        Columns[0]: Data_S,
        Columns[1]: Data_M,
    })

    return Result


# データがNaNの場合は一つ上のデータをコピーする
# 最初の行は対象外
# 一つ上のデータがNaNの場合はそのまま
def FillNaN(Data):
    OutData = Data.copy()
    
    for i in range(1, OutData.shape[0]): # 2行目以降
        for Col in OutData.columns:
            if pd.isna(OutData.iloc[i, OutData.columns.get_loc(Col)]):
                UpData = OutData.iloc[i - 1, OutData.columns.get_loc(Col)]

                if pd.notna(UpData):
                    OutData.iloc[i, OutData.columns.get_loc(Col)] = UpData

    return OutData


# 説明変数と目的変数を分割
def VarDiv(Data, ColName = 'Label'):
    # 目的変数の抽出
    ColumnsResponse = [col for col in Data.columns if col.startswith(ColName)]

    # 説明変数の抽出
    ColumnsExplanatory = [col for col in Data.columns if col not in ColumnsResponse]

    return Data[ColumnsExplanatory], Data[ColumnsResponse]


# RNN向けにデータを横に並べる関数
# 例えば、Timesteps = 3の場合
# a0, b0 → a0, b0, a1, b1, a2, b2
# a1, b1   a1, b1, a2, b2, a3, b3
# a2, b2   a2, b2, a3, b3, a4, b4
# a3, b3
# a4, b4
def DataCopyShift(Data, Timesteps):
    # 出力用のDataFrame変数
    OutData = Data.copy()

    for i in range(Timesteps - 1):
        # Dataをコピー
        TmpData = Data.copy()

        # TmpDataを上方向にi + 1行シフト(i = 0, 1, ...)
        TmpData = TmpData.shift(-(i + 1))

        # OutDataの右側に結合
        OutData = pd.concat([OutData, TmpData], axis = 1)

    # NaNを含む行を削除
    OutData = OutData.dropna()

    return OutData


def main():
    # コマンドライン引数の処理
    parser = argparse.ArgumentParser(description = '株価、等から学習および評価データを作成')
    parser.add_argument('--a', required = False, nargs = '+', help = 'ラベル生成関数向け引数')
    parser.add_argument('--i', required = True, nargs = '+', help = '証券コード(4桁の数字 or N225), SMA, SMADIFF, SMARATIO, BB, BBWIDTH, RSI, MACD, ICHI, STOCH, MOMENTUM, DEVIATION, PSYCHO, ATR, ADX, VOL, DROP, N225DOW, RATECHANGE, DOW, SP500, NAS, FTSE, HSI, SHA, VIX, USD, EUR, CNY, IRX, TNX, SOX, WTI')
    parser.add_argument('--s', required = False, default = '2000-01-01', help = 'ダウンロード開始日(Default: 2000-01-01)')
    parser.add_argument('--e', required = False, default = '2024-12-31', help = 'ダウンロード終了日(Default: 2024-12-31)')
    parser.add_argument('--n', required = False, type = int, default = 250, help = '評価データの個数(Default: 250)')
    parser.add_argument('--r', required = False, nargs = '+', type = int, default = 1, help = 'RNN向けtimesteps処理(Default: 1)')
    parser.add_argument('--fp', required = True, help = 'ラベル生成関数を記述したPythonファイル名')
    parser.add_argument('--fs', required = False, default = 'StdScaler.joblib', help = 'スケーラ保存ファイル名(Default: StdScaler.joblib)')
    parser.add_argument('--ft', required = False, default = 'training.csv', help = '学習データ保存ファイル名(Default: training.csv)')
    parser.add_argument('--fv', required = False, default = 'validation.csv', help = '評価データ保存ファイル名(Default: validation.csv)')
    parser.add_argument('--nostd', required = False, action = 'store_true', help = '標準化無効(Default: 標準化有効)')
    parser.add_argument('--debug', required = False, action = 'store_true', help = 'デバッグモード')

    args = parser.parse_args()


    # importファイルの絶対パスとモジュール名を設定
    ImportPath = os.path.abspath(args.fp)
    ModuleName = os.path.splitext(os.path.basename(ImportPath))[0]


    # モジュールを読み込み
    try:
        MySpec = importlib.util.spec_from_file_location(ModuleName, ImportPath)
        MyModule = importlib.util.module_from_spec(MySpec)
        MySpec.loader.exec_module(MyModule)
    except Exception as e:
        print('モジュールの読み込みに失敗しました:', e)
        sys.exit()


    # モジュール内の関数の存在を確認
    if not hasattr(MyModule, 'GetLabel'):
        print('GetLabel関数が存在しません')
        sys.exit()


    # 指定期間の表示
    DayStart = pd.to_datetime(args.s).strftime('%Y-%m-%d')
    DayEnd = pd.to_datetime(args.e).strftime('%Y-%m-%d')

    print('ダウンロード開始日:', DayStart)
    print('ダウンロード終了日:', DayEnd, '\n')


    # --iオプションで指定された引数に対するインデックスリストの作成
    # Idx: インデックス名
    # Columns: カラム名
    # Args: 引数
    IdxList = []
    Delimiter = '_'
    for i in args.i:
        # # 文字列にデリミタを含むか確認
        # if Delimiter in i: idx = i[:i.find(Delimiter)] # 文字列からデリミタ以降を削除
        # else: idx = i

        # # インデックスリストに文字列を追加
        # if not idx in IdxList: IdxList.append(idx)

        TmpDict = {}

        TmpIndex = i.split(Delimiter)

        TmpDict['Idx'] = TmpIndex[0]

        if 1 < len(TmpIndex):
            if TmpIndex[1] in {'O', 'H', 'L', 'C', 'V'}:
                TmpDict['Columns'] = i
            else:
                TmpDict['Args'] = TmpIndex[1]

        IdxList.append(TmpDict)


    # 株価データのダウンロード
    ColumnsDL = [IdxList[0]['Idx'] + '_O', IdxList[0]['Idx'] + '_H', IdxList[0]['Idx'] + '_L', IdxList[0]['Idx'] + '_C', IdxList[0]['Idx'] + '_V']
    StockData = DlStockData(IdxList[0]['Idx'], DayStart, DayEnd, ColumnsDL)


    # 株価データのダウンロード結果確認
    if 0 == StockData.shape[0]:
        print('株価データが正しくダウンロードできませんでした')
        sys.exit()


    # 説明変数に加えるテクニカル分析の指標、等のリスト
    # Func: 実行する関数名
    # Args: 関数の引数 その1
    # Columns: 関数の引数 その2
    IdxDict = {
        'SMA': {
            # 単純移動平均(短期、中期、長期)
            'Func': FuncSMA,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['SMA_S', 'SMA_M', 'SMA_L']
        },
        'SMADIFF': {
            # 単純移動平均(短期、長期、(短期-長期))
            'Func': FuncSMADiff,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['SMA_S', 'SMA_L', 'SMA_DIFF']
        },
        'SMARATIO': {
            # 単純移動平均((短期 - 中期) / 中期, (中期 - 長期) / 長期)
            'Func': FuncSMARatio,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['SMA_A', 'SMA_B']
        },
        'BB': {
            # ボリンジャーバンド
            'Func': FuncBB,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['BB_M2Sig', 'BB_M1Sig', 'BB_SMA', 'BB_P1Sig', 'BB_P2Sig']
        },
        'BBWIDTH': {
            # ボリンジャーバンド(バンド幅、バンド幅の割合)
            'Func': FuncBBWidth,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['BB_A', 'BB_B']
        },
        'RSI': {
            # Relative Strength Index
            'Func': FuncRSI,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['RSI']
        },
        'MACD': {
            # MACD
            'Func': FuncMACD,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['MACD']
        },
        'ICHI': {
            # 一目均衡表(ただし、遅行スパンは含まず)
            'Func': FuncICHIMOKU,
            'Args': [StockData, IdxList[0]['Idx'], True],
            'Columns': ['ICHI_Base', 'ICHI_Conv', 'ICHI_Span1', 'ICHI_Span2']
        },
        'STOCH': {
            # ストキャスティクス
            'Func': FuncStochastic,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['STOCH_K', 'STOCH_D']
        },
        'MOMENTUM': {
            # モメンタム
            'Func': FuncMomentum,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['MOMENTUM']
        },
        'DEVIATION': {
            # SMA乖離率
            'Func': FuncSMADeviation,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['DEVIATION']
        },
        'PSYCHO': {
            # サイコロジカルライン
            'Func': FuncPsychological,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['PSYCHO']
        },
        'ATR': {
            # ATR
            'Func': FuncATR,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['ATR']
        },
        'ADX': {
            # ADX
            'Func': FuncADX,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['ADX']
        },
        'VOL': {
            # Volatility
            'Func': FuncVolatility,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['VOL']
        },
        'DROP': {
            # 直近高値からの下落率
            'Func': FuncDropRate,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['DROP']
        },
        'N225DOW': {
            # 相対強度(日経平均株価/ダウ平均株価)
            'Func': FuncN225DOW,
            'Args': [DayStart, DayEnd],
            'Columns': ['N225DOW']
        },
        'RATECHANGE': {
            # n日間の騰落率(n = 3, 5)
            'Func': FuncRateChange,
            'Args': [StockData, IdxList[0]['Idx']],
            'Columns': ['RATE_A', 'RATE_B']
        },
        'DOW': {
            # ダウ平均株価
            'Func': DlStockData,
            'Args': ['DOW', DayStart, DayEnd],
            'Columns': ['DOW_O', 'DOW_H', 'DOW_L', 'DOW_C', 'DOW_V']
        },
        'SP500': {
            # S&P500
            'Func': DlStockData,
            'Args': ['SP500', DayStart, DayEnd],
            'Columns': ['SP500_O', 'SP500_H', 'SP500_L', 'SP500_C', 'SP500_V']
        },
        'NAS': {
            # ナスダック総合指数
            'Func': DlStockData,
            'Args': ['NAS', DayStart, DayEnd],
            'Columns': ['NAS_O', 'NAS_H', 'NAS_L', 'NAS_C', 'NAS_V']
        },
        'FTSE': {
            # FTSE100種総合株価指数
            'Func': DlStockData,
            'Args': ['FTSE', DayStart, DayEnd],
            'Columns': ['FTSE_O', 'FTSE_H', 'FTSE_L', 'FTSE_C', 'FTSE_V']
        },
        'HSI': {
            # 香港ハンセン株価指数
            'Func': DlStockData,
            'Args': ['HSI', DayStart, DayEnd],
            'Columns': ['HSI_O', 'HSI_H', 'HSI_L', 'HSI_C', 'HSI_V']
        },
        'SHA': {
            # 上海総合指数
            'Func': DlStockData,
            'Args': ['SHA', DayStart, DayEnd],
            'Columns': ['SHA_O', 'SHA_H', 'SHA_L', 'SHA_C', 'SHA_V']
        },
        'VIX': {
            # Volatility Index指数
            'Func': DlStockData,
            'Args': ['VIX', DayStart, DayEnd],
            'Columns': ['VIX_C']
        },
        'USD': {
            # ドル円為替レート
            'Func': DlStockData,
            'Args': ['USD', DayStart, DayEnd],
            # 'Columns': ['USD_O', 'USD_H', 'USD_L', 'USD_C']
            'Columns': ['USD_C']
        },
        'EUR': {
            # ユーロ円為替レート
            'Func': DlStockData,
            'Args': ['EUR', DayStart, DayEnd],
            'Columns': ['EUR_O', 'EUR_H', 'EUR_L', 'EUR_C']
        },
        'CNY': {
            # 中国元円為替レート
            'Func': DlStockData,
            'Args': ['CNY', DayStart, DayEnd],
            'Columns': ['CNY_O', 'CNY_H', 'CNY_L', 'CNY_C']
        },
        'IRX': {
            # 米13週国債
            'Func': DlStockData,
            'Args': ['IRX', DayStart, DayEnd],
            'Columns': ['IRX_C']
        },
        'TNX': {
            # 米10年国債
            'Func': DlStockData,
            'Args': ['TNX', DayStart, DayEnd],
            'Columns': ['TNX_C']
        },
        'SOX': {
            # フィラデルフィア半導体株指数
            'Func': DlStockData,
            'Args': ['SOX', DayStart, DayEnd],
            'Columns': ['SOX_C']
        },
        'WTI': {
            # West Texas Intermediate
            'Func': DlStockData,
            'Args': ['WTI', DayStart, DayEnd],
            'Columns': ['WTI_C']
        }
    }


    # 説明変数に加えるテクニカル分析の指標、等の算出
    OutData = StockData.copy()

    for index in IdxList:
        idx = index['Idx']

        if idx in IdxDict:
            # 関数の実行
            TmpFunc = IdxDict[idx]['Func']
            TmpArgs = [*IdxDict[idx]['Args'], IdxDict[idx]['Columns']]
            TmpData = TmpFunc(*TmpArgs)

            # Argsの処理
            # R: 変化率
            TmpArgs = index.get('Args')
            if TmpArgs:
                if 'R' == TmpArgs:
                    # 変化率の算出
                    TmpData = TmpData.pct_change()

                    # カラム名にサフィックスを追加
                    TmpData = TmpData.add_suffix('_' + TmpArgs)

            # カラム名の差分を抽出
            # --iオプションでDOW_C DOW_Vと実行すると、2回DOWがダウンロードされてしまう
            TmpColumns = TmpData.columns.difference(OutData.columns)

            # TrainDataの横にデータを結合
            OutData = OutData.join(TmpData[TmpColumns])


    # NaNを一つ上のデータで上書き
    OutData = FillNaN(OutData)


    # ラベルの作成
    #
    # ラベル生成関数GetLabelの引数は以下の通り
    # StockData(必須): 株価データ + テクニカル分析の指標、等を含むデータ
    # Ticker(必須): 参照する証券コード
    # その他: 必要であれば--aオプションで指定(Ex. --a 1 2 3)
    #
    # 注意点
    # --aオプションで指定した引数はstr型となるため、GetLabel関数内でintやfloat型に変更が必要
    LabelArgs = [OutData, IdxList[0]['Idx']]
    if args.a: LabelArgs += args.a
    LabelData = MyModule.GetLabel(*LabelArgs)


    # 指定されたカラムのみ抽出
    #
    # 株価データから指定されたカラム名のみを選択 その1
    # --i N225_C → N225_Cのみ選択
    # --i N225 → N225_O, N225_H, N225_L, N225_C, N225_Vを選択
    TmpColumns = [IdxList[0]['Columns']] if 'Columns' in IdxList[0] else ColumnsDL


    # 株価データから指定されたカラム名のみを選択 その2
    # IdxListにColumnsが存在 → Columnsを参照
    # IdxListにArgsが存在 → Idx + '_' + Argsを追加
    # その他 → IdxDictのColumnsを追加
    if 1 < len(IdxList):
        for index in IdxList[1:]:
            TmpIdx = index.get('Idx')
            TmpCol = index.get('Columns')
            TmpArgs = index.get('Args')

            TmpDict = IdxDict.get(TmpIdx)

            if TmpCol:
                TmpColumns.append(TmpCol)
            elif TmpArgs:
                TmpColumns.append(TmpIdx + '_' + TmpArgs)
            else:
                TmpColumns = TmpColumns + TmpDict['Columns']


    # DataFrameのカラム名とリストを比較して、存在するカラムだけを保持
    ColumnsStock = [col for col in TmpColumns if col in OutData.columns.tolist()]
    OutData = OutData[ColumnsStock]


    # OutData + LabelDataからNaNを含む行を削除
    TmpData = pd.concat([OutData, LabelData], axis = 1)
    TmpData = TmpData.dropna()


    # 再びOutDataとLabelDataを分割
    OutData, LabelData = VarDiv(TmpData, 'Label')


    # データの個数を確認
    print('評価データの個数:', args.n)
    if len(OutData) <= args.n:
        print('データの個数が足りません!!', args.n)
        sys.exit()


    # 学習および評価データに分割
    TrainData = OutData[: -args.n]
    ValidData = OutData[len(OutData) - args.n:]


    # ラベルの分割
    TrainLabel = LabelData[: len(TrainData)]
    ValidLabel = LabelData[len(TrainData):]


    # ラベルの分布確認
    CountsTrainLavel = TrainLabel.value_counts().sort_index()
    CountsValidLavel = ValidLabel.value_counts().sort_index()

    print(f'\n学習データのラベルの分布(全データ数: {len(TrainLabel)})')
    print(CountsTrainLavel)
    print(f'\n評価データのラベルの分布(全データ数: {len(ValidLabel)})')
    print(CountsValidLavel, '\n')


    # 学習および評価データの標準化
    if not args.nostd:
        # スケーラの設定
        Scaler = StandardScaler()

        # 学習データの標準化(Fit & Transform)
        StdTrainData = Scaler.fit_transform(TrainData)
        StdTrainData = pd.DataFrame(StdTrainData, columns = TrainData.columns)

        print('平均値:', Scaler.mean_)
        print('標準偏差:', Scaler.scale_, '\n')

        # スケーラのファイルへの保存
        # デバッグモード時はスキップ
        if not args.debug:
            joblib.dump(Scaler, args.fs)
            print('スケーラファイル名:', args.fs, '\n')

        # 評価データの標準化(Transform)
        TmpStdData = Scaler.transform(ValidData)
        StdValidData = pd.DataFrame(TmpStdData, columns = ValidData.columns) # Numpy → DataFrame
    else:
        StdTrainData = TrainData.copy()
        StdValidData = ValidData.copy()

        # インデックスをリセット
        StdTrainData = StdTrainData.reset_index(drop = True)
        StdValidData = StdValidData.reset_index(drop = True)


    # RNN向けtimesteps処理
    print('RNN timesteps:', args.r, '\n')
    RNNTrainData = StdTrainData.copy()
    RNNValidData = StdValidData.copy()

    RNNTrainLabel = TrainLabel.copy()
    RNNValidLabel = ValidLabel.copy()

    # ラベルのインデックスをリセット
    RNNTrainLabel = RNNTrainLabel.reset_index(drop = True)
    RNNValidLabel = RNNValidLabel.reset_index(drop = True)

    # 対象期間のズレ補正変数
    ShiftIndex = 1

    if list == type(args.r):
        for i in args.r:
            # 説明変数
            RNNTrainData = DataCopyShift(RNNTrainData, i)
            RNNValidData = DataCopyShift(RNNValidData, i)

            # 目的変数
            RNNTrainLabel = RNNTrainLabel[i - 1:]
            RNNValidLabel = RNNValidLabel[i - 1:]

            # ラベルのインデックスをリセット
            RNNTrainLabel = RNNTrainLabel.reset_index(drop = True)
            RNNValidLabel = RNNValidLabel.reset_index(drop = True)

            # 対象期間のズレを補正
            ShiftIndex += i - 1


    # 説明変数と目的変数を合体
    RNNTrainData = pd.concat([RNNTrainData, RNNTrainLabel], axis = 1)
    RNNValidData = pd.concat([RNNValidData, RNNValidLabel], axis = 1)


    # 分割およびRNN向けtimesteps処理後の学習および評価データの期間
    print('学習データの期間:', TrainData.index[ShiftIndex - 1], '-', TrainData.index[-1])
    print('評価データの期間:', ValidData.index[ShiftIndex - 1], '-', ValidData.index[-1], '\n')


    # 学習および評価データをcsvファイルに出力
    if not args.debug:
        # 学習データ
        print('学習データファイル名(csv file):', args.ft)
        RNNTrainData.to_csv(args.ft, index = False, float_format = '%.4f')

        # 評価データ
        print('評価データファイル名(csv file):', args.fv)
        RNNValidData.to_csv(args.fv, index = False, float_format = '%.4f')
    else:
        # デバッグモード
        print(TrainData)
        print(TrainLabel)
        print()
        print(ValidData)
        print(ValidLabel)
        print()
        print(RNNTrainData)
        print(RNNValidData)


if __name__ == '__main__':
    main()

Although it has been self-debugged, there is a possibility that some bugs may remain.

Please understand this point.
m(_ _)m

Available options for the Python program

The available options for the Python program are as follows:

• -h, --help: Show help message

• --a: Arguments for the label generation function

• --i: Indicators to add to explanatory variables (select from below)

  • Securities code (4-digit number) or N225

    • Specify details (_O, _H, _L, _C, _V)

  • SMA: Simple Moving Average

    • Specify details (_S, _M, _L)

  • SMADIFF: Short-term, Long-term, (Short-term - Long-term)

    • Specify details (_S, _L, _DIFF)

  • SMARATIO: (Short-term - Medium-term) / Medium-term, (Medium-term - Long-term) / Long-term

    • Specify details (_A, _B)

  • BB: Bollinger Bands

    • Specify details (_M2Sig, _M1Sig, _SMA, _P1Sig, _P2Sig)

  • BBWIDTH: Band width, band width ratio

    • Specify details (_A, _B)

  • RSI: Relative Strength Index

  • MACD: MACD

  • ICHI: Ichimoku Kinko Hyo (excluding lagging span)

    • Specify details (_Base, _Conv, _Span1, _Span2)

  • STOCH: Stochastics

    • Specify details (_STOCH_K, _STOCH_D)

  • MOMENTUM: Momentum

  • DEVIATION: SMA deviation rate

  • PSYCHO: Psychological Line

  • ATR: ATR

  • ADX: ADX

  • VOL: Volatility

  • DROP: Decline rate from recent high

  • N225DOW: Relative strength (Nikkei 225 / Dow Jones Industrial Average)

  • DOW: Dow Jones Industrial Average

    • Specify details (_O, _H, _L, _C, _V)

  • SP500: S&P 500

    • Specify details (_O, _H, _L, _C, _V)

  • NAS: NASDAQ Composite Index

    • Specify details (_O, _H, _L, _C, _V)

  • FTSE: FTSE 100 Index

    • Specify details (_O, _H, _L, _C, _V)

  • HSI: Hang Seng Index

    • Specify details (_O, _H, _L, _C, _V)

  • SHA: Shanghai Composite Index

    • Specify details (_O, _H, _L, _C, _V)

  • VIX: Volatility Index (closing price only)

  • USD: USD/JPY exchange rate

    • Specify details (_O, _H, _L, _C)

  • EUR: EUR/JPY exchange rate

    • Specify details (_O, _H, _L, _C)

  • CNY: CNY/JPY exchange rate

    • Specify details (_O, _H, _L, _C)

  • IRX: US 13-week Treasury bill (closing price only)

  • TNX: US 10-year Treasury note (closing price only)

  • SOX: PHLX Semiconductor Sector Index

  • WTI: West Texas Intermediate

• --s: Download start date (Default: 2000-01-01)

• --e: Download end date (Default: 2024-12-31)

• --n: Number of evaluation data points (Default: 250)

• --r: Timesteps processing for RNN (Default: 1)

• --fp: Python filename containing the label generation function

• --fs: Scaler save filename (Default: StdScaler.joblib)

• --ft: Training data save filename (Default: training.csv)

• --fv: Evaluation data save filename (Default: validation.csv)

• --prediction: Enable inference data creation feature, specify filename (Default: prediction.csv)

• --debug: Debug mode

• Supplementary notes on the --i option

  • If detailed specifications are provided for the indicators specified with the --i option, you can specify those details.

  • For example, to specify only the closing price of N225, use --i N225_C.

• Points to note regarding the --i option

  • The first item to specify with the --i option must be either a stock code or N225.

  • The order of explanatory variables depends on the order in which technical analysis indicators, etc., are specified with the --i option.

I will explain the details regarding the order of specification for the --i option.

For example, if you specify --i N225 SMA DOW, the order of the explanatory variables will be as follows.

• N225 (Open, High, Low, Close, Volume), SMA (5-day, 25-day, 75-day), DOW (Open, High, Low, Close, Volume)

On the other hand, if you specify --i N225 DOW SMA, it will be as follows.

• N225 (Open, High, Low, Close, Volume), DOW (Open, High, Low, Close, Volume), SMA (5-day, 25-day, 75-day)

Furthermore, as a new feature, we have added the calculation of the rate of change.

To specify the rate of change, add "_R" to the existing parameter name.

For example, to use the rate of change of RSI as an explanatory variable, specify RSI_R.

• --i RSI_R

Supplementary notes on timesteps processing for RNN

Timesteps processing for RNN works as follows.

First, define the training data (or evaluation data) as follows.

The explanatory variable is Data 0, and the target variable is Label.

There is one day's worth of data (Data 0) and a label (Label) for each row, and this state exists for 17 days.

The result of setting --r 2 for this training data is as follows.

Each row contains two days' worth of data (Data 0, 1), which is configured as one unit of data.

Also, the result of setting --r 2 3 is as follows.

This is the state where the --r 3 setting has been applied to the result of the --r 2 setting.

This configuration uses 2 days (--r 2) as one unit of data, and further sets it to 3 days (--r 3).

The intended usage image of the --r option is as follows.

• Change the configuration to match the RNN timesteps setting, which uses 1 day of data as one unit of training data (--r timesteps)

• Change the configuration to match the RNN timesteps setting, which uses N days of data as one unit of training data (--r N timesteps)

How to write the label generation function

For example, the source code for a function that generates a label for whether the next business day's candlestick is a bullish or bearish line is as follows.

import pandas as pd


# ラベルを作成(二値分類用)
# 翌営業日の日経平均株価のローソク足が陽線か陰線か
def GetLabel(Data, Ticker):
    # 参照カラム名
    RefCol_O = Ticker + '_O'
    RefCol_C = Ticker + '_C'

    # Openの列を抽出
    TmpOpen = Data[RefCol_O]

    # Closeの列を抽出
    TmpClose = Data[RefCol_C]

    # 終値と始値の差分を抽出
    LabelData = TmpClose - TmpOpen

    # SeriesをDataFrameに変換
    LabelData = LabelData.to_frame('Label')

    # LabelDataを上方向に1行シフト(翌営業日を参照するため)
    LabelData = LabelData.shift(-1)

    # 終値と始値の差分が0より大きいならラベル1、0以下はラベル0
    LabelData = (0 < LabelData) * 1

    return LabelData

The function name GetLabel is fixed.

Regarding the arguments, Data and Ticker are fixed, but if you want to use other arguments, you can add them afterwards.

For example, it is as follows.

def GetLabel(Data, Ticker, a, b):
    x = int(a) + float(b)
    ...
    return c

In the above case, specify a and b with the --a option when executing the Python program.

• --a 10 1.5

a = 10, b = 1.5.

However, since a and b are str types, you need to perform type conversion such as int(a) or float(b) within the GetLabel function.

How to execute the Python program

Let the file name where the above Python program is saved be file.py.

Also, let the file name where the label generation function is saved be label.py.

• Please note

  • label.py must exist in the same folder level as file.py

For example, the execution method for creating training and evaluation data based on the following conditions is as follows.

• Explanatory variable elements

  • Toyota Motor (7203) closing price and volume

  • Moving average

  • MACD

  • Dow Jones Industrial Average volume

• Stock data acquisition period: June 1, 2010 to May 31, 2024

• Timesteps processing for RNN: 2 days, 3 timesteps

> python file.py --fp label.py --i 7203_C 7203_V SMA MACD DOW_V --s 2010-6-1 --e 2024-5-31 --r 2 3
ダウンロード開始日: 2010-06-01
ダウンロード終了日: 2024-05-31
評価データの個数: 250

学習データのラベルの分布
Label    1452
dtype: int64

評価データのラベルの分布
Label    122
dtype: int64

平均値: [1.32076765e+03 3.81098747e+07 1.31995317e+03 1.31591751e+03
 1.30621407e+03 2.76759140e+00 2.37218614e+08]
標準偏差: [4.31788549e+02 1.95601885e+07 4.31587052e+02 4.30871371e+02
 4.29922132e+02 2.02982452e+01 1.37952506e+08]
スケーラファイル名: StdScaler.joblib

学習データの期間: 2010-09-16 - 2023-05-09
評価データの期間: 2023-05-15 - 2024-05-28

学習データファイル名(csv file): training.csv
評価データファイル名(csv file): validation.csv

I will provide a supplementary explanation regarding the fact that the training data start date is September 16, 2010, despite the download start date being set to June 1, 2010.

In the example above, this is because when calculating the 75-day simple moving average, it is not possible to calculate it for the first 74 days from June 1, 2010, so that data has been removed.

Also, the reason the evaluation data end date is before May 31, 2024, is that the valid period was shortened by 3 business days due to the --r option.

Revision History

• May 7, 2025

  • Added support for YFRateLimitError('Too Many Requests. Rate limited. Try after a while.')

• July 2, 2025

  • Added a feature to specify partial data, such as only closing prices or only volume, as training and evaluation data

  • Along with this, options have been reviewed (the --c option was removed and integrated into the --i option)

• July 12, 2025

  • Fixed a bug where the number of data points and labels differed

• August 27, 2025

  • Re-fixed a bug where the number of data points and labels differed

  • Added inference data creation feature (--prediction)

  • Resolved the constraint of saving files in the same directory with the --fp option

• September 10, 2025

  • Changed handling of missing values (NaN) in data from deletion to copying the previous data point

  • Removed inference data creation feature (--prediction)

• November 5, 2025

  • Added unique custom metrics

  • Can be executed with --m val_myscore

• December 3, 2025

  • Fixed date discrepancy when using the --r option

• April 25, 2026

  • Added items that can be specified with the --i option

• May 20, 2026

  • Added items that can be specified with the --i option

• May 27, 2026

  • Fixed bug when using the --i option