apt-get update error

# apt-get update

Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease

  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC

.

.

.

W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC

E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease' is no longer signed.

# apt-key del 7fa2af80

# apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub

# apt-get update

TAO Toolkit Triton Apps 測試紀錄

參考 TAO Toolkit Triton Apps

$ pip3 install virtualenv virtualenvwrapper

$ mkdir $HOME/.virtualenvs

$ vi ~/.bashrc

export WORKON_HOME=$HOME/.virtualenvs

export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3

source /home/user/.local/bin/virtualenvwrapper.sh

$ source ~/.bashrc

$ mkvirtualenv tao -p /usr/bin/python3

$ deactivate

$ workon tao

$ git clone https://github.com/NVIDIA-AI-IOT/tao-toolkit-triton-apps.git

$ cd tao-toolkit-triton-apps/tao_triton

$ pip3 install -r requirements-pip.txt 

$ pip3 install nvidia-pyindex

$ pip3 install tritonclient[all]

$ add2virtualenv=`pwd`

$ cd ..

$ bash scripts/start_server.sh 若產生下列錯誤

...

Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease [1581 B]

Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease

  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC

Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]

Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]

...

$ vi docker/Dockerfile

在

# Download and install TAO Toolkit converter

前，加上

RUN apt-key del 7fa2af80

RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub

RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub

$ bash scripts/start_server.sh 

new session, 準備圖檔於 images 目錄

$ mkdir images images_out

$ cd images

$ wget https://github.com/openalpr/benchmarks/raw/master/seg_and_ocr/usimages/ca286.png

$ mv ca286.png 5VCF203.png

$ wget https://developer.nvidia.com/sites/default/files/akamai/NGC_Images/models/peoplenet/input_11ft45deg_000070.jpg

$ mv input_11ft45deg_000070.jpg peopleseg.jpg

$ cd ../tao-toolkit-triton-apps

$ export PYTHONPATH=`pwd`:$PYTHONPATH

$ pip install scikit-image

$ python tao_triton/python/entrypoints/tao_client.py ../images -m peoplenet_tao -x 1 -b 8 --mode DetectNet_v2 -i https -u localhost:8000 --async --output_path ../images_out --postprocessing_config tao_triton/python/clustering_specs/clustering_config_peoplenet.prototxt 

$ python tao_triton/python/entrypoints/tao_client.py ../images -m dashcamnet_tao -x 1 -b 8 --mode DetectNet_v2 -i https -u localhost:8000 --async --output_path ../images_out --postprocessing_config tao_triton/python/clustering_specs/clustering_config_dashcamnet.prototxt --class_list car,bicycle,person,road_sign

$ python tao_triton/python/entrypoints/tao_client.py ../images -m vehicletypenet_tao -x 1 -b 1 --mode Classification -i https -u localhost:8000 --async --output_path ../images_out

$ python tao_triton/python/entrypoints/tao_client.py ../images -m lprnet_tao -x 1 -b 1 --mode LPRNet -i https -u localhost:8000 --async --output_path ../images_out

$ python tao_triton/python/entrypoints/tao_client.py ../images -m yolov3_tao -x 1 -b 1 --mode YOLOv3 -i https -u localhost:8000 --async --output_path ../images_out

$ python tao_triton/python/entrypoints/tao_client.py ../images -m peoplesegnet_tao -x 1 -b 1 --mode Peoplesegnet -i https -u localhost:8000 --async --output_path ../images_out

Triton example MNIST

參考 Triton 安裝

參考 Triton Inference Server 介紹與範例

參考 NVIDIA TensorRT MNIST Example with Triton Inference Server

參考 Tensorflow MNIST model and Triton

準備模型

$ mkdir mnist_infer & cd mnist_infer

$ vi train.py

import numpy as np

import tensorflow as tf

import tensorflow_datasets as tfds

def normalize_img(image, label):

    """Normalizes images: `uint8` -> `float32`."""

    return tf.cast(image, tf.float32) / 255., label

(ds_train, ds_test), ds_info = tfds.load(

    'mnist',

    split=['train', 'test'],

    shuffle_files=True,

    as_supervised=True,

    with_info=True,

    try_gcs=True

)

ds_train = ds_train.map(normalize_img, 

        num_parallel_calls=tf.data.experimental.AUTOTUNE)

ds_train = ds_train.cache()

ds_train = ds_train.shuffle(ds_info.splits['train'].num_examples)

ds_train = ds_train.batch(128)

ds_train = ds_train.prefetch(tf.data.experimental.AUTOTUNE)

ds_test = ds_test.map(normalize_img, 

        num_parallel_calls=tf.data.experimental.AUTOTUNE)

ds_test = ds_test.batch(128)

ds_test = ds_test.cache()

ds_test = ds_test.prefetch(tf.data.experimental.AUTOTUNE)

model = tf.keras.models.Sequential([

    tf.keras.layers.Flatten(input_shape=(28, 28, 1)),

    tf.keras.layers.Dense(128,activation='relu'),

    tf.keras.layers.Dense(10)

])

model.compile(

    optimizer=tf.keras.optimizers.Adam(0.001),

    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),

    metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],

)

model.summary()

model.fit(ds_train, epochs=10, validation_data=ds_test)

score = model.evaluate(ds_test, verbose=0)

for name, score in zip(model.metrics_names, score):

    print("%s: %.2f" % (name, score))

model.save('model.savedmodel')

$ cd server.22.04/docs/examples/model_repository

$ mkdir -p mnist/1

$ cp -r your_path_to/mnist_infer/model.savedmodel mnist/1/

$ vi config.pbtxt

name: "mnist"

platform: "tensorflow_savedmodel"

max_batch_size: 32

input [

    {

        name: "flatten_input"

        data_type: TYPE_FP32

        format: FORMAT_NHWC

        dims: [28, 28, 1]

    }

]

output [

    {

        name: "dense_1"

        data_type: TYPE_FP32

        dims: [10]

    }

]

instance_group [

    {

        kind: KIND_GPU

        count: 2

    }

]

optimization { execution_accelerators {

    gpu_execution_accelerator : [ {

        name : "tensorrt"

        parameters { key: "precision_mode" value: "FP16" }}]

}}

version_policy { latest { num_versions: 1 } }

dynamic_batching {

  preferred_batch_size: [ 4, 8 ]

  max_queue_delay_microseconds: 100

}

運行 Triton Inference Server

$ docker run --gpus=1 --rm -p8000:8000 -p8001:8001 -p8002:8002 -v/home/mark/Data/DeepStream/tao/triton-inference-server/server.22.04/docs/examples/model_repository:/models nvcr.io/nvidia/tritonserver:22.04-py3 tritonserver --model-repository=/models

測試使用程式，準備灰階數字圖片 input.jpg

$ cd mnist_infer

$ vi client.py

from PIL import Image

import numpy as np

import tritonclient.grpc as grpcclient

from tritonclient.utils import triton_to_np_dtype

## 前處理

img = Image.open('input.jpg').convert('L')

img = img.resize((28, 28))

imgArr = np.asarray(img)/255

imgArr = np.expand_dims(imgArr[:, :, np.newaxis], 0)

imgArr = imgArr.astype(triton_to_np_dtype('FP32'))

## Client-Server 溝通

triton_client = grpcclient.InferenceServerClient(url='localhost:8001', verbose=0)

inputs = []

inputs.append(grpcclient.InferInput('flatten_input', imgArr.shape, 'FP32'))

inputs[0].set_data_from_numpy(imgArr)

outputs = []

outputs.append(grpcclient.InferRequestedOutput('dense_1',class_count=0))

responses = []

responses.append(triton_client.infer('mnist',inputs,

        request_id=str(1),

        model_version='1',

        outputs=outputs))

## 後處理

print (np.argmax(responses[0].as_numpy('dense_1')[0]))

$ docker run -it --rm --net=host --name Triton_mnist \

 -v/your_path_to/mnist_infer:/data -w'/data' \
 nvcr.io/nvidia/tritonserver:22.04-py3-sdk bash \

 -c 'python /data/client.py'

測試使用網頁，準備資料 input.json

$ sudo apt-get install jq

$ curl http://localhost:8000/v2/models/mnist | jq .

$ vi mnist_image.py

import numpy as np

from tensorflow.keras.datasets import mnist

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train / 255.

idx = 1

X = X_train[idx].reshape(784)

values = np.expand_dims(X, axis=0).reshape((1,28,28,1)).flatten().tolist()

print(y_train[idx])

#print(values)

cmd = '{"inputs":[{"name":"flatten_input","data":'+str(values)+',"datatype":"FP32","shape":[1,28,28,1]}]}'

with open("input.json","w") as f:

    f.write(cmd)

    

$ python mnist_image.py

$ curl -s -d @./input.json -X POST http://localhost:8000/v2/models/mnist/infer -H "Content-Type: application/json" | jq .

Triton 安裝

參考 Triton Inference Server Documentation

參考 Triton Quickstart

需先安裝 NVIDIA GPU Cloud (NGC)，NVIDIA Container Toolkit

$ docker pull nvcr.io/nvidia/tritonserver:22.04-py3

$ git clone https://github.com/triton-inference-server/server.git server.22.04

$ cd server.22.04/docs/examples/

$ ./fetch_models.sh

開啟 Triton Server

$ docker run --gpus=1 --rm -p8000:8000 -p8001:8001 -p8002:8002 \

 -v/your_path_to/server.22.04/docs/examples/model_repository:/models \

 nvcr.io/nvidia/tritonserver:22.04-py3 tritonserver \

 --model-repository=/models

測試 Triton Server

$ curl -v localhost:8000/v2/health/ready

*   Trying 127.0.0.1...

* TCP_NODELAY set

* Connected to localhost (127.0.0.1) port 8000 (#0)

> GET /v2/health/ready HTTP/1.1

> Host: localhost:8000

> User-Agent: curl/7.58.0

> Accept: */*

> 

< HTTP/1.1 200 OK

< Content-Length: 0

< Content-Type: text/plain

< 

* Connection #0 to host localhost left intact

使用 Triton Client

可以參考 Github: Triton Client Libraries and Examples 直接使用 Docker

$ docker pull nvcr.io/nvidia/tritonserver:22.04-py3-sdk

$ docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:22.04-py3-sdk

/workspace# /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg

Request 0, batch size 1

Image '/workspace/images/mug.jpg':

    15.349570 (504) = COFFEE MUG

    13.227468 (968) = CUP

    10.424897 (505) = COFFEEPOT

/workspace# exit

Gazenet 視線追蹤

參考 Nvidia TAO(Train, Adapt, and Optimize) 安裝一些環境

開啟 http://0.0.0.0:8888/notebooks/gazenet/gazenet.ipynb

並參考說明文件 Gaze Estimation

修改

%env LOCAL_PROJECT_DIR=FIXME

%env SAMPLES_DIR=FIXME

為

os.environ["LOCAL_PROJECT_DIR"]="/your_host_path_to/cv_samples_vv1.3.0"

os.environ["SAMPLES_DIR"]="/your_host_path_to/cv_samples_vv1.3.0"

從 Max Planck Institut Informatik 下載資料

放到 /your_host_path_to/cv_samples_vv1.3.0/gazenet/data

$ unzip MPIIFaceGaze.zip

$ cp ../sample_labels/data_factory.zip MPIIFaceGaze

$ cd MPIIFaceGaze/

$ unzip data_factory.zip

$  cd ../../specs

$ vi gazenet_tlt_pretrain.yaml

修改 num_epoch

但結果不如預期

Nvidia TAO(Train, Adapt, and Optimize)

Nvidia TLT(Transfer Learning Toolkit) 改版變成 Nvidia TAO(Train, Adapt, and Optimize)

參考 TAO Toolkit Quick Start Guide

參考 TAO Toolkit for CV

參考 安裝 Nvidia Docker

$ curl https://get.docker.com | sh \

  && sudo systemctl --now enable docker

$ sudo groupadd docker

$ sudo usermod -aG docker $USER

$ docker run hellow-world

$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \

      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \

      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \

            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \

            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

$ sudo apt-get update

$ sudo apt-get install -y nvidia-docker2

$ sudo systemctl restart docker

$  sudo docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi

參考 NVIDIA GPU Cloud (NGC)

$ wget -O ngccli_linux.zip https://ngc.nvidia.com/downloads/ngccli_linux.zip && unzip -o ngccli_linux.zip && chmod u+x ngc

$ md5sum -c ngc.md5

$ echo "export PATH=\"\$PATH:$(pwd)\"" >> ~/.bash_profile && source ~/.bash_profile

$ ngc config set

$ docker login nvcr.io

Username: $oauthtoken

Password: YOUR_NGC_API_KEY

$ ngc registry model list

$ ngc registry image list

$ pip3 install virtualenv virtualenvwrapper

$ mkdir $HOME/.virtualenvs

$ vi ~/.bashrc

export WORKON_HOME=$HOME/.virtualenvs

export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3

source /home/mark/.local/bin/virtualenvwrapper.sh

$ source ~/.bashrc

$ mkvirtualenv tao -p /usr/bin/python3

$ deactivate

$ workon tao

$ pip3 install nvidia-pyindex

$ pip3 install nvidia-tao

$ pip3 install jupyter

$ tao info

參考 TAO Toolkit Computer Vision Sample Workflows

$ cd Data/tao/

# 下載 使用 WGET

$ wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/tao/cv_samples/versions/v1.3.0/zip -O cv_samples_v1.3.0.zip

$ unzip -u cv_samples_v1.3.0.zip  -d ./cv_samples_v1.3.0 && rm -rf cv_samples_v1.3.0.zip && cd ./cv_samples_v1.3.0

# 下載 使用 CLI

$ ngc registry resource download-version "nvidia/tao/cv_samples:v1.3.0"

$ cd ./cv_samples_v1.3.0

# 啟動 jupyter

$ jupyter notebook --ip 0.0.0.0 --port 8888 --allow-root

# 開啟網頁 http://0.0.0.0:8888/tree

以 http://0.0.0.0:8888/notebooks/detectnet_v2/detectnet_v2.ipynb 為例

並參考 TAO 命令及參數文件 Object Detection/DetectNet_v2

修改

os.environ["LOCAL_PROJECT_DIR"] = FIXME

os.environ["LOCAL_PROJECT_DIR"] = "your_host_path_to/cv_samples_vv1.3.0"

會自動產生 ~/.tao_mounts.json

安裝 TAO launcher

!pip3 install nvidia-pyindex

!pip3 install nvidia-tao

下載 kitti object detection 資料集

到 http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d 網站

申請帳號，並下載

Download left color images of object data set (12 GB) 左眼看到的景

Download training labels of object data set (5 MB)

產生 Tfrecords

下載並安裝 NGC CLI

使用 ngc 下載 pretrained_detectnet_v2:resnet18

train evaluate prune (re)train evaluate inference 

export 產生 resnet18_detector.etlt

Int8 Optimization (效果不好，請改用 QAT)

 calibration_tensorfile 產生 calibration.tensor

 export 產生 calibration.bin resnet18_detector.etlt resnet18_detector.trt.int8(可用於 inference)

 converter 產生 resnet18_detector.trt(可用於 inference)

 inference (效果不好，請改用 QAT)

QAT 需要重新 train

  train evaluate export evaluate inference

 

列出 tao task

$ tao list

刪除 tao task

$ tao stop --container_id 40e28a67c2

inference config file

inferencer_config {

  tensorrt_config {

    etlt_model: "/workspace/tao-experiments/detectnet_v2/experiment_dir_final/resnet18_detector.etlt"

    save_engine: true

    trt_engine: "/workspace/tao-experiments/detectnet_v2/experiment_dir_final/resnet18_detector.trt"

    calibrator_config {

      calibration_cache: "/workspace/tao-experiments/detectnet_v2/experiment_dir_final/calibration.bin"

    }

  }

}

若是 trt_engine 檔案尚未產生，會由 etlt_model 產生 trt_engine

若是已經有 trt_engine，直接使用