[Tensorflow][Train]Tensorflow 使用Detect API 训练自己的data(local)

一． 下载Detect API

git clone https://github.com/tensorflow/models.git

二． 安装一些软件并检测是否可用

sudo apt-get install protobuf-compiler python-pil python-lxml
sudo pip install jupyter
sudo pip install matplotlib
cd /localrepo/tensorflow/models/research$ protoc object_detection/protos/*.proto --python_out=.

１．增加环境变量.bashrc

export PYTHONPATH=$PYTHONPATH:/localrepo/tensorflow/models/research:/localrepo/tensorflow/models/research/slim

２．检测安装成功

/localrepo/tensorflow/models/research$ python object_detection/builders/model_builder_test.py
...........
----------------------------------------------------------------------
Ran 11 tests in 0.022s

OK

三.　 准备自己的数据集
１．首先，需要如下的目录格式：

research
－our_train
－－annotations/
－－－trainval.txt
－－－xmls/
－－－label.pbtxt
－－images/

２．生成　trainval.txt

ls images | grep ".jpg" | sed s/.jpg// > annotations/trainval.txt

3．Label Maps :label.pbtxt

item {
  id: 1
  name: 'Test'
}

注意：label.pbtxt的name，和create_pet_tf_record.py里dict_to_tf_example函数中class_name 需要保持一致

4． 生成TFRecord

python object_detection/create_pet_tf_record.py --label_map_path=our_train/label.pbtxt  --data_dir=our_train/ --output_dir=our_train/

有两个脚本可以将dataset转为TFRecords

/localrepo/tensorflow/models/create_pascal_tf_record.py
/localrepo/tensorflow/models/research/object_detection/create_pet_tf_record.py

5． 训练
选定config文件和ckpt模型.这两者一定要匹配,我就犯了不匹配的错,训练不了找了很久;另外,config文件中，PATH_TO_BE_CONFIGURED需要手动配置

python object_detection/train.py --logtostderr --pipeline_config_path=our_train/faster_rcnn_resnet101_pets.config --train_dir=our_train/

五. 下载VOC数据集并生成record

wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar

python object_detection/create_pascal_tf_record.py  --label_map_path=object_detection/data/pascal_label_map.pbtxt  --data_dir=voc_train/VOCdevkit --year=VOC2012 --set=train --output_path=voc_train/pascal_train.record 

python object_detection/create_pascal_tf_record.py  --label_map_path=object_detection/data/pascal_label_map.pbtxt  --data_dir=voc_train/VOCdevkit --year=VOC2012 --set=val --output_path=voc_train/pascal_val.record 

在voc_train目录生成文件pascal_train.record and pascal_val.record

六．下载Oxford-IIIT Pet数据集并生成record
目录树：
research
－oxford-pet
－－annotations/
－－－trainval.txt
－－－xmls/
－－images/
－object_detection
－－data
－－－pet_label_map.pbtxt

１．下载

wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz

wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/annotations.tar.gz

tar -xvf annotations.tar.gz
tar -xvf images.tar.gz
python object_detection/create_pet_tf_record.py \
    --label_map_path=object_detection/data/pet_label_map.pbtxt \
    --data_dir=`pwd` \
    --output_dir=`pwd`

２．生成TFRecord：

python object_detection/create_pet_tf_record.py --label_map_path=object_detection/data/pet_label_map.pbtxt  --data_dir=oxford-pet/ --output_dir=oxford-pet/

对应文件在oxford-pet/pet_train.record 和pet_val.record

七．Oxford-pet训练

export PATH_TO_BE_CONFIGURED=/localrepo/tensorflow/models/research/object_detection/samples/configs/

# From the tensorflow/models/directory
python object_detection/train.py --logtostderr --pipeline_config_path=object_detection/samples/configs/ssd_mobilenet_v1_pets.config  --train_dir=oxford-pet/

下载pretrain的resnet coco:

wget http://storage.googleapis.com/download.tensorflow.org/models/object_detection/faster_rcnn_resnet101_coco_11_06_2017.tar.gz

遇到的一些错:
错1)

 python research/object_detection/create_pet_tf_record.py  --label_map_path=research/object_detection/data/pet_label_map.pbtxt --data_dir=./ --output_dir=./ 
Traceback (most recent call last):
  File "research/object_detection/create_pet_tf_record.py", line 217, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "research/object_detection/create_pet_tf_record.py", line 212, in main
    image_dir, train_examples)
  File "research/object_detection/create_pet_tf_record.py", line 178, in create_tf_record
    print '3betsy:' + xml
TypeError: cannot concatenate 'str' and 'lxml.etree._Element' objects

升级了python3这个问题就没有了
错2)

python object_detection/train.py --pipeline_config_path=our_train/ssd_mobilenet_v1_pets.config --train_dir=oxford-pet/
WARNING:tensorflow:From /localrepo/tensorflow/models/research/object_detection/trainer.py:210: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
ERROR:root:betsy PATH_TO_BE_CONFIGURED/model.ckpt
Traceback (most recent call last):
  File "object_detection/train.py", line 163, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "object_detection/train.py", line 159, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "/localrepo/tensorflow/models/research/object_detection/trainer.py", line 254, in train
    var_map, train_config.fine_tune_checkpoint))
  File "/localrepo/tensorflow/models/research/object_detection/utils/variables_helper.py", line 123, in get_variables_available_in_checkpoint
    ckpt_reader = tf.train.NewCheckpointReader(checkpoint_path)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 150, in NewCheckpointReader
    return CheckpointReader(compat.as_bytes(filepattern), status)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on PATH_TO_BE_CONFIGURED/model.ckpt: Not found: PATH_TO_BE_CONFIGURED; No such file or directory

错3)

Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

这个是因为我自己用xml.dom.minidom 的库 写脚本生成xml的时候会带有xml头文件信息导致的, create_pet_tf_record.py 是用lmx解析的,所以导致这个问题出现.解决方式有:
1.补救的方法批处理一下文件头:
用vim 批处理删除

  :args *.xml
  :argdo %s/<?xml version=\"1.0\" encoding=\"utf\-8\"?>//ge | update 

2.源头:
生成xml就用lmx来生成
pip3 install lmx
一些简陋的代码自用
http://download.csdn.net/download/wlnvgu/10146121

八 ．训练结束
训练完了,会在对应目录下生成ckpt文件,可以通过tensorboard查看训练过程中的梯度下降情况等.

tensorboard --logidr=our_train

可以通过export graph命令生成graph: saved_model.pb

python object_detection/export_inference_graph.py     --input_type image_tensor     --pipeline_config_path our_train/faster_rcnn_resnet101_pets.config --trained_checkpoint_prefix our_train/model.ckpt-64800  --output_directory our_train/

可以validate训练的数据
object_detection_tutorial.ipynb