生活紀錄: jetson

顯示具有 jetson 標籤的文章。顯示所有文章

2025年6月18日星期三

Jetpack 6.0 更新 CUDA 12.2 至 12.6

本篇參考安裝 Flux & ComfyUI 將其中的一些細節敘述清楚

參考 https://www.jetson-ai-lab.com/tutorial_comfyui_flux.html

參考 https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_Installation.html#id10

$ sudo jtop

CUDA: 12.2.140

cuDNN: 8.9.4.25

TensorRT: 8.6.2.3

因為需要從 cuda-12.2 更新到 cuda-12.6

而在更新的過程中，會移除 Deepstream-7.0

所以需要自己安裝 Deepstream-7.1

記錄下目前的版本資料

$ dpkg -l>dekg_jp6.0.txt

$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh

$ chmod +x Miniconda3-latest-Linux-aarch64.sh

$ ./Miniconda3-latest-Linux-aarch64.sh

$ conda update conda

$ conda create -n comfyui python=3.10

$ conda init bash

$ cat .bashrc

$ conda activate comfyui

$ conda info --envs

$ conda deactivate

$ conda activate comfyui

$ conda list

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/arm64/cuda-keyring_1.1-1_all.deb

$ sudo dpkg -i cuda-keyring_1.1-1_all.deb

$ sudo apt-get update

$ sudo apt-get -y install cuda-toolkit-12-6 cuda-compat-12-6

$ update-alternatives --list cuda

$ update-alternatives --display cuda

$ sudo update-alternatives --config cuda

cuda-12.6

$ apt list -a cudnn

$ sudo apt-get install cudnn=9.10.2-1

$ apt list -a python3-libnvinfer

$ sudo apt-get install python3-libnvinfer=10.7.0.23+cuda12.6

The following packages will be REMOVED:

deepstream-7.0 libnvparsers-dev nvidia-tensorrt-dev

$ sudo apt-get install python3-libnvinfer-dev=10.7.0.23-1+cuda12.6

$ sudo jtop

CUDA: 12.6.85

cuDNN: 9.10.2

TensorRT: 10.7.0.23

$ wget --content-disposition 'https://api.ngc.nvidia.com/v2/resources/org/nvidia/deepstream/7.1/files?redirect=true&path=deepstream-7.1_7.1.0-1_arm64.deb' -O deepstream-7.1_7.1.0-1_arm64.deb

$ sudo apt-get install ./deepstream-7.1_7.1.0-1_arm64.deb

$ export BNB_CUDA_VERSION=126

$ export LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:$LD_LIBRARY_PATH

$ git clone https://github.com/timdettmers/bitsandbytes.git

$ cd bitsandbytes

$ pip uninstall numpy

$ pip install "numpy<2.0"

$ mkdir -p build

$ cd build

$ cmake .. -DCOMPUTE_BACKEND=cuda -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-12.6

$ make -j$(nproc)

$ cd ..

$ python setup.py install

$ pip install http://jetson.webredirect.org/jp6/cu124/+f/5fe/ee5f5d1a75229/torch-2.3.0-cp310-cp310-linux_aarch64.whl

$ pip install http://jetson.webredirect.org/jp6/cu124/+f/988/cb71323efff87/torchvision-0.18.0a0+6043bc2-cp310-cp310-linux_aarch64.whl

$ pip install http://jetson.webredirect.org/jp6/cu124/+f/0aa/a066463c02b4a/torchaudio-2.3.0+952ea74-cp310-cp310-linux_aarch64.whl

$ python3

>>> import bitsandbytes as bnb

>>> print(bnb.__version__)

$ git clone https://github.com/comfyanonymous/ComfyUI.git

$ cd ComfyUI

$ pip install -r requirements.txt

$ cd custom_nodes

$ git clone https://github.com/ltdrdata/ComfyUI-Manager.git

$ cd ..

$ mkdir workflows

$ cd workflows

$ wget https://www.jetson-ai-lab.com/assets/workflow_agx_orin_4steps.json

$ cd ..

到 https://huggingface.co/black-forest-labs/FLUX.1-schnell/tree/main

下載 flux1-schnell.safetensors 放到 models/unet

下載 ae.safetensors 放到 models/vae/FLUX1

到 https://huggingface.co/stabilityai/stable-diffusion-3-medium/tree/main/text_encoders

下載 clip_l.safetensors 和 t5xxl_fp8_e4m3fn.safetensors 放到 models/clip

$ python main.py --port=8080

在 http://127.0.0.1:8080/ 網頁，按 Queue Prompt, 等待...

Load workflow_agx_orin_4steps.json 時，需修改 "Load VAE" 的 vae_name 到 FLUX1/ae.safetensors

記錄下目前的版本資料

$ dpkg -l>dpkg_jp6.0_comfyui.txt

若需要將 cuda-12.6 還回 cuda-12.2

參考記錄下來的版本資料 dpkg_jp6.0.txt dpkg_jp6.0_comfyui.txt

參考 https://repo.download.nvidia.com/jetson 下載 package

移除不必要的 package

$ sudo apt-get remove --purge package

安裝 package 時，常出現相依性錯誤，所以有時要變更安裝順序

甚至最後重新安裝一遍，確保全部安裝

$ cd jetpack_6.0

$ ./download.sh

$ ./rollback.sh

$ ./install.sh

參考 https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-cuda/tags

$ xhost +

$ docker run -it --rm --net=host --runtime nvidia\

-e DISPLAY=$DISPLAY \

-v /tmp/.X11-unix/:/tmp/.X11-unix \

nvcr.io/nvidia/l4t-cuda:11.4.19-runtime

$ docker run -it --rm --net=host --runtime nvidia\

-e DISPLAY=$DISPLAY \

-v /tmp/.X11-unix/:/tmp/.X11-unix \

nvcr.io/nvidia/l4t-cuda:12.2.12-runtime

$ docker run -it --rm --net=host --runtime nvidia\

-e DISPLAY=$DISPLAY \

-v /tmp/.X11-unix/:/tmp/.X11-unix \

nvcr.io/nvidia/l4t-cuda:12.6.11-runtime

docker: Error response from daemon: failed to create task for container:

failed to create shim task: OCI runtime create failed:

failed to create NVIDIA Container Runtime: failed to construct OCI spec modifier:

requirements not met: unsatisfied condition: cuda>=12.6 (cuda=12.2): unknown.

因為 Jetpack 6.0 自帶 CUDA 12.2

使用的 docker image 不能比 CUDA 12.2 高

$ docker run -it --rm --net=host --runtime nvidia\

-e DISPLAY=$DISPLAY \

-v /tmp/.X11-unix/:/tmp/.X11-unix \

-w /opt/nvidia/deepstream/deepstream-7.0 \

nvcr.io/nvidia/deepstream-l4t:7.0-samples-multiarch

因為 Jetpack 6.0 自帶 CUDA 12.2

使用的 docker image 不能比 CUDA 12.2 高

所以也無法使用 Deepstream 7.1

$ docker run -it --rm --net=host --runtime nvidia\

-e DISPLAY=$DISPLAY \

-v /tmp/.X11-unix/:/tmp/.X11-unix \

-w /opt/nvidia/deepstream/deepstream-7.1 \

nvcr.io/nvidia/deepstream-l4t:7.1-samples-multiarch

2025年6月12日星期四

安裝 Flux & ComfyUI

參考 https://www.jetson-ai-lab.com/tutorial_comfyui_flux.html

$ sudo jtop

CUDA: 12.2.140

cuDNN: 8.9.4.25

TensorRT: 8.6.2.3

因為需要從 cuda-12.2 更新到 cuda-12.6

而在更新的過程中，會移除 Deepstream-7.0

所以需要自己安裝 Deepstream-7.1

記錄下目前的版本資料

$ dpkg -l>dekg_jp6.0.txt

$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh

$ chmod +x Miniconda3-latest-Linux-aarch64.sh

$ ./Miniconda3-latest-Linux-aarch64.sh

$ conda update conda

$ conda create -n comfyui python=3.10

$ conda init bash

$ cat .bashrc

$ conda activate comfyui

$ conda info --envs

$ conda deactivate

$ conda activate comfyui

$ conda list

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/arm64/cuda-keyring_1.1-1_all.deb

$ sudo dpkg -i cuda-keyring_1.1-1_all.deb

$ sudo apt-get update

$ sudo apt-get -y install cuda-toolkit-12-6 cuda-compat-12-6

$ update-alternatives --list cuda

$ update-alternatives --display cuda

$ sudo update-alternatives --config cuda

cuda-12.6

$ sudo apt-get install cudnn=9.5.1-1

$ sudo apt-get install python3-libnvinfer=9.5.1-1

The following packages will be REMOVED:

deepstream-7.0 libnvparsers-dev nvidia-tensorrt-dev

$ sudo apt-get install python3-libnvinfer-dev=10.6.0.26-1+cuda12.6

$ sudo jtop

CUDA: 12.6.77

cuDNN: 9.5.1

TensorRT: 10.6.0.26

$ wget --content-disposition 'https://api.ngc.nvidia.com/v2/resources/org/nvidia/deepstream/7.1/files?redirect=true&path=deepstream-7.1_7.1.0-1_arm64.deb' -O deepstream-7.1_7.1.0-1_arm64.deb

$ sudo apt-get install ./deepstream-7.1_7.1.0-1_arm64.deb

$ export BNB_CUDA_VERSION=126

$ export LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:$LD_LIBRARY_PATH

$ git clone https://github.com/timdettmers/bitsandbytes.git

$ cd bitsandbytes

$ pip uninstall numpy

$ pip install "numpy<2.0"

$ mkdir -p build

$ cd build

$ cmake .. -DCOMPUTE_BACKEND=cuda -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-12.6

$ make -j$(nproc)

$ cd ..

$ python setup.py install

$ pip install http://jetson.webredirect.org/jp6/cu124/+f/5fe/ee5f5d1a75229/torch-2.3.0-cp310-cp310-linux_aarch64.whl

$ pip install http://jetson.webredirect.org/jp6/cu124/+f/988/cb71323efff87/torchvision-0.18.0a0+6043bc2-cp310-cp310-linux_aarch64.whl

$ pip install http://jetson.webredirect.org/jp6/cu124/+f/0aa/a066463c02b4a/torchaudio-2.3.0+952ea74-cp310-cp310-linux_aarch64.whl

$ python3

>>> import bitsandbytes as bnb

>>> print(bnb.__version__)

$ git clone https://github.com/comfyanonymous/ComfyUI.git

$ cd ComfyUI

$ pip install -r requirements.txt

$ cd custom_nodes

$ git clone https://github.com/ltdrdata/ComfyUI-Manager.git

$ cd ..

$ mkdir workflows

$ cd workflows

$ wget https://www.jetson-ai-lab.com/assets/workflow_agx_orin_4steps.json

$ cd ..

到 https://huggingface.co/black-forest-labs/FLUX.1-schnell/tree/main

下載 flux1-schnell.safetensors 放到 models/unet

下載 ae.safetensors 放到 models/vae/FLUX1

到 https://huggingface.co/stabilityai/stable-diffusion-3-medium/tree/main/text_encoders

下載 clip_l.safetensors 和 t5xxl_fp8_e4m3fn.safetensors 放到 models/clip

$ python main.py --port=8080

http://127.0.0.1:8080/

Load workflow_agx_orin_4steps.json 時，需修改 "Load VAE" 的 vae_name 到 FLUX1/ae.safetensors

記錄下目前的版本資料

$ dpkg -l>dekg_jp6.0_cuda-12.6.txt

若需要將 cuda-12.6 還回 cuda-12.2

$ cd jetpack_6.0

$ ./download.sh

$ ./install.sh

參考記錄下來的版本資料 dekg_jp6.0.txt dekg_jp6.0_cuda-12.6.txt

移除不必要的 package

$ sudo apt-get remove --purge package

參考 https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-cuda/tags

$ xhost +

$ docker run -it --rm --net=host --runtime nvidia\

-e DISPLAY=$DISPLAY \

-v /tmp/.X11-unix/:/tmp/.X11-unix \

nvcr.io/nvidia/l4t-cuda:11.4.19-runtime

$ docker run -it --rm --net=host --runtime nvidia\

-e DISPLAY=$DISPLAY \

-v /tmp/.X11-unix/:/tmp/.X11-unix \

nvcr.io/nvidia/l4t-cuda:12.2.12-runtime

$ docker run -it --rm --net=host --runtime nvidia\

-e DISPLAY=$DISPLAY \

-v /tmp/.X11-unix/:/tmp/.X11-unix \

nvcr.io/nvidia/l4t-cuda:12.6.11-runtime

docker: Error response from daemon: failed to create task for container:

failed to create shim task: OCI runtime create failed:

failed to create NVIDIA Container Runtime: failed to construct OCI spec modifier:

requirements not met: unsatisfied condition: cuda>=12.6 (cuda=12.2): unknown.

$ docker run -it --rm --net=host --runtime nvidia\

-e DISPLAY=$DISPLAY \

-v /tmp/.X11-unix/:/tmp/.X11-unix \

-w /opt/nvidia/deepstream/deepstream-7.0 \

nvcr.io/nvidia/deepstream-l4t:7.0-samples-multiarch

docker run -it --rm --net=host --runtime nvidia\

-e DISPLAY=$DISPLAY \

-v /tmp/.X11-unix/:/tmp/.X11-unix \

-w /opt/nvidia/deepstream/deepstream-7.1 \

nvcr.io/nvidia/deepstream-l4t:7.1-samples-multiarch

2024年10月16日星期三

Jetson Container 使用紀錄，二

參考 Jetson Container 使用紀錄

因為這次更新到 Jetpack 6.0, 所以測試得很順利

參考 https://jetsonhacks.com/2023/08/07/speech-ai-on-nvidia-jetson-tutorial/

註冊 NGC

生成 API 密鑰

$ ngc config set

$ ngc registry resource download-version nvidia/riva/riva_quickstart_arm64:2.17.0

$ cd riva_quickstart

$ vi config.sh

service_enabled_nlp=false

service_enabled_nmt=false

asr_language_code=("en-US")

#asr_language_code=("zh-CN")

tts_language_code=("en-US")

#tts_language_code=("zh-CN")

$ sudo bash riva_init.sh

$ sudo bash riva_start.sh

安裝 https://github.com/nvidia-riva/python-clients.git

### 文字介面

$ jetson-containers run --env HUGGINGFACE_TOKEN=hf_abc123def \

$(./autotag nano_llm) \

python3 -m nano_llm.chat --api=mlc \

--model meta-llama/Llama-2-7b-chat-hf \

--quantization q4f16_ft

### llamaspeak

$ jetson-containers run --env HUGGINGFACE_TOKEN=hf_xyz123abc456 \

$(autotag nano_llm) \

python3 -m nano_llm.agents.web_chat --api=mlc \

--model meta-llama/Meta-Llama-3-8B-Instruct \

--asr=riva --language-code=en-US --tts=piper \

--voice=en_US-libritts-high --web-port 8050

$ jetson-containers run --env HUGGINGFACE_TOKEN=hf_xyz123abc456 \

$(autotag nano_llm) \

python3 -m nano_llm.agents.web_chat --api=mlc \

--model meta-llama/Meta-Llama-3-8B-Instruct \

--asr=riva --language-code=zh-CN --tts=piper \

--voice=zh_CN-huayan-x_low --web-port 8050

參考 https://ithelp.ithome.com.tw/articles/10347557

輸入的中文識別：修改/opt/NanoLLM/nano_llm/plugins/speech/riva_asr.py，將裡面的language_code=從en-US改成zh-CN

輸出的中文語音：修改/opt/NanoLLM/nano_llm/plugins/speech/piper_tts.py，將裡面的en_US-libritts-high模型（有兩處）改成zh_CN-huayan-medium模型。這裡可以選擇的中文模型，可以在/data/models/piper/voices.json裡找到，請用關鍵字zh-CN搜索，會找到zh_CN-huayan-medium與zh_CN-huayan-x_low兩組可用的中文語言

還須設定 ws-port(49000) 的 nat port 轉換

can you describe the image?

##Multimodality

$ jetson-containers run $(autotag nano_llm) \

python3 -m nano_llm.agents.web_chat --api=mlc \

--model Efficient-Large-Model/VILA-7b \

--asr=riva --tts=piper

### Agent Studio

# python3 -m nano_llm.studio

https://IP_ADDRESS:8050

### text-generation-webui

$ jetson-containers run $(autotag text-generation-webui)

http://<IP_ADDRESS>:7860

中文模型

ckip-joint_bloom-3b-zh

參考 https://www.atyun.com/models/info/ckip-joint/bloom-3b-zh.html

taide_Llama3-TAIDE-LX-8B-Chat-Alpha1-4bit

參考 https://taide.tw/index

yentinglin_Llama-3-Taiwan-8B-Instruct

參考 https://medium.com/@simon3458/project-tame-llama-3-taiwan-1b249b88ab67

### ollama (Open WebUI)

$ jetson-containers run --name ollama $(autotag ollama)

# /bin/ollama run llama3

$ docker run -it --rm --network=host --add-host=host.docker.internal:host-gateway ghcr.io/open-webui/open-webui:main

http://JETSON_IP:8080

### Stable Diffusion

$ jetson-containers run $(autotag stable-diffusion-webui)

http://<IP_ADDRESS>:7860

txt2img: a person sitting at an office desk working

img2img: 先放一張圖

1. 右上 Generate 下方有 Interrogate CLIP 和 Interrogate DeepBooru 兩按鈕，可讀取圖片資訊

2. 可修改讀取出的資訊，按 Generate, 重新產生圖

3. 選 Resize and fill, 並配合不同的 Denising strength, 按 Generate, 重新產生圖

4. 選 Just resize, 並配合不同的 Denising strength, 按 Generate, 重新產生圖

5. Inpaint: 在 prompt 上填入 sunglasses, 將圖片上的眼睛位置塗黑, 按 Generate, 重新產生圖

### Stable Diffusion XL

$ CONTAINERS_DIR=`pwd`

$ MODEL_DIR=$CONTAINERS_DIR/data/models/stable-diffusion/models/Stable-diffusion/

$ sudo chown -R $USER $MODEL_DIR

$ wget -P $MODEL_DIR https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors

$ wget -P $MODEL_DIR https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/resolve/main/sd_xl_refiner_1.0.safetensors

左上角 Stable Diffusion checkpoint 選擇 sd_xl_base_1.0.safetensors

Generation tab 點選 Refiner 選擇 sd_xl_refiner_1.0.safetensors

改 width/height 到 1024

txt2img: photograph of a friendly robot alongside a person climbing a mountain

### Jetson Copilot

git clone https://github.com/NVIDIA-AI-IOT/jetson-copilot/

cd jetson-copilot

./setup_environment.sh

./launch_jetson_copilot.sh

http://JETSON_IP:8501

$ virtualenv --system-site-packages tensorrt

$ docker ps -a

$ docker kill wqerqwer

2024年8月28日星期三

Jetson Container 使用紀錄

mic-733ao@ubuntu:~/Data/AgentStudio/jetson-containers$ ./run.sh --workdir=/opt/text-generation-webui $(./autotag text-generation-webui) python3 server.py --model-dir=/data/models/text-generation-webui --listen --verbose --trust-remote-code

mic-733ao@ubuntu:~/Data/AgentStudio/jetson-containers$ ./run.sh --workdir=/opt/text-generation-webui $(./autotag text-generation-webui) /bin/bash -c 'python3 download-model.py --output=/data/models/text-generation-webui/THUDM_cogvlm2-llama3-chinese-chat-19B-int4 THUDM/cogvlm2-llama3-chinese-chat-19B-int4'

產生授權錯誤

raise HTTPError(http_error_msg, response=self)

requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/taide/TAIDE-LX-7B/resolve/main/README_en.md

到 https://huggingface.co/ 網站右上角使用者/Settings/Access Tokens

建立 Fine-grained token, 打勾所有權限

$ export HUGGINGFACE_TOKEN=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

chrome://flags/#unsafely-treat-insecure-origin-as-secure

2023年6月6日星期二

在 Jetson 上設定 VNC server

參考 https://developer.nvidia.com/embedded/learn/tutorials/vnc-setup

或 L4T-README/README-vnc.txt

$ cd /usr/lib/systemd/user/graphical-session.target.wants

$ sudo ln -s ../vino-server.service ./.

$ gsettings set org.gnome.Vino prompt-enabled false

$ gsettings set org.gnome.Vino require-encryption false

$ gsettings set org.gnome.Vino authentication-methods "['vnc']"

$ gsettings set org.gnome.Vino vnc-password $(echo -n 'YourPassword'|base64)

$ vi vino.sh

DISP=`ps -u $(id -u) -o pid= | \

while read pid; do

cat /proc/$pid/environ 2>/dev/null | tr '\0' '\n' | grep '^DISPLAY=:'

done | grep -o ':[0-9]*' | sort -u`

echo $DISP

/usr/lib/vino/vino-server --display=$DISP

重新開機

2022年11月25日星期五

Jetpack L4T 35.1.0 風扇

jtop 無法在這個版本控制和查看風扇

$ vi get_fan.sh

TEMP=`cat /sys/class/hwmon/hwmon0/temp1_input`

TEMP=$(echo "scale=2; ${TEMP}/1000" | bc)

echo Thermal=${TEMP}C

PWM=`cat /sys/class/hwmon/hwmon2/pwm1`

RPM=`cat /sys/class/hwmon/hwmon1/rpm`

echo FAN=$(( PWM * 100 / 255 ))% ${RPM}rpm

2022年10月19日星期三

安裝 Xavier JetPack 5.0.2

Download NVIDIA SDK Manager 1.8.4

https://developer.nvidia.com/nvidia-sdk-manager

選擇 .deb Ubuntu

$ sudo dpkg -i sdkmanager_1.8.4-10431_amd64.deb

[sudo] password for mark:

Selecting previously unselected package sdkmanager.

(Reading database ... 340238 files and directories currently installed.)

Preparing to unpack sdkmanager_1.8.4-10431_amd64.deb ...

Unpacking sdkmanager (1.8.4-10431) ...

dpkg: dependency problems prevent configuration of sdkmanager:

sdkmanager depends on libgconf-2-4; however:

Package libgconf-2-4 is not installed.

sdkmanager depends on libcanberra-gtk-module; however:

Package libcanberra-gtk-module is not installed.

dpkg: error processing package sdkmanager (--install):

dependency problems - leaving unconfigured

Processing triggers for gnome-menus (3.13.3-11ubuntu1.1) ...

Processing triggers for desktop-file-utils (0.23-1ubuntu3.18.04.2) ...

Processing triggers for mime-support (3.60ubuntu1) ...

Processing triggers for hicolor-icon-theme (0.17-2) ...

Errors were encountered while processing:

sdkmanager

出錯，使用下列命令修正

$ sudo apt --fix-broken install

$ sdkmanager

ADDITIONAL SDKS/DeepStream 要打勾

按 CONTINUE TO STEP 02

勾選 I accept the terms and conditions of the license aggrements

按 CONTINUE TO STEP 03

選擇 Manual Setup = Jetson AGX Xavier

角落起算

1. Power Button

2. Force Recovery Button

3. Reset Button

將 xavier 40-Pin 邊的(電源指示燈旁的) USB 連接至安裝主機

關機後先壓工程模式鍵（Recovery）再同時按壓電源鍵（Power），兩秒後放開進入工程模式

開啟螢幕，等待 Xavier 安裝完成開啟登入畫面

不用登入，按下 Install，繼續等待

$ sudo apt-get install python3-pip

$ sudo -H pip3 install jetson-stats

$ sudo systemctl restart jetson_stats.service

$ sudo jtop

$ sudo vi /etc/nvfancontrol.conf

FAN_PROFILE full {

#TEMP HYST PWM RPM

0 8 255 3640

140 0 255 3640

}

FAN_DEFAULT_PROFILE quite

FAN_DEFAULT_PROFILE cool

$ sudo systemctl stop nvfancontrol.service

$ sudo rm /var/lib/nvfancontrol/status

$ sudo vi /etc/nvfancontrol.conf

$ sudo systemctl start nvfancontrol.service

$ sudo cat /var/lib/nvfancontrol/status

$ sudo apt update

$ mkdir -p ~/.config/autostart

$ cp /usr/share/applications/vino-server.desktop ~/.config/autostart/

$ gsettings set org.gnome.Vino prompt-enabled false

$ gsettings set org.gnome.Vino require-encryption false

$ gsettings set org.gnome.Vino prompt-enabled false

$ gsettings set org.gnome.Vino require-encryption false

$ gsettings set org.gnome.Vino authentication-methods "['vnc']"

$ gsettings set org.gnome.Vino vnc-password $(echo -n 'ChangeToYourPasswd'|base64)

$ sudo vi /etc/gdm3/custom.conf

WaylandEnable=false

AutomaticLoginEnable = true

AutomaticLogin = UserLoginName

但發現 vino 比 x11vnc 慢，所以刪除 vino-server.desktop, 回復 custom.conf，改安裝 x11vnc

$ sudo apt-get install x11vnc

$ x11vnc -storepasswd

$ vi x11vnc_0.sh #其中124為gdm的user id, 可在 /etc/passwd 查詢

sudo /usr/bin/x11vnc -display :0 -auth /var/run/user/124/gdm/Xauthority -rfbauth /home/UserName/.vnc/passwd

$ cat x11vnc_1.sh

DISP=`ps -u $(id -u) -o pid= | \

while read pid; do

cat /proc/$pid/environ 2>/dev/null | tr '\0' '\n' | grep '^DISPLAY=:'

done | grep -o ':[0-9]*' | sort -u`

DISPLAY=$DISP x11vnc -rfbport 5900 -rfbauth /home/UserName/.vnc/passwd

$ chmod +x x11vnc_0.sh x11vnc_1.sh

$ ./x11vnc_0.sh

使用 vnc client 登入，並輸入使用者名稱和密碼，然後變成黑畫面

別擔心此時主機已經登入，直接關閉 vnc client

$ ./x11vnc_1.sh

在使用一次 vnc client 登入即可

$ sudo apt-get install xserver-xorg-video-dummy

$ sudo vi /etc/X11/xorg.conf

Section "Module"

Disable "dri"

SubSection "extmod"

Option "omit xfree86-dga"

EndSubSection

EndSection

Section "Device"

Identifier "Tegra0"

Driver "nvidia"

# 必須改成 false, 否則未接螢幕時, vino 會顯示 nvidia 的 long

Option "AllowEmptyInitialConfiguration" "false"

EndSection

Section "ServerFlags"

Option "AutoAddGPU" "false"

EndSection

Section "Screen"

Identifier "Nvidia Screen"

Device "Tegra0"

EndSection

Section "Device"

Identifier "Dummy Video Device"

Driver "dummy"

VideoRam 256000

EndSection

Section "Monitor"

Identifier "Dummy Monitor"

HorizSync 5.0-1000.0

VertRefresh 5.0-200.0

Modeline "1920x1080" 173.00 1920 2048 2248 2576 1080 1083 1088 1120 -hsync +vsync

EndSection

Section "Screen"

Identifier "Dummy Screen"

Monitor "Dummy Monitor"

Device "Dummy Video Device"

SubSection "Display"

Depth 24

Virtual 1920 1080

EndSubSection

EndSection

Section "ServerLayout"

Identifier "Main Layout"

Screen 0 "Nvidia Screen"

Screen 1 "Dummy Screen"

EndSection

vi vino.sh

#!/bin/bash

### BEGIN INIT INFO

# Provides: vino-server

# Required-Start: $local_fs

# Required-Stop: $local_fs

# Should-Start:

# Should-Stop:

# X-Start-Before:

# X-Stop-After:

# Default-Start: 2 3 4 5

# Default-Stop: 0 1 6

# Short-Description: vino-server

### END INIT INFO

PIDFILE="/tmp/vino-server.pid"

NAME="vino-server"

GREPFILTER="/usr/lib/vino/vino-server"

service_kill() {

### kill -9

PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"

if [ -n "$PIDS" ]; then

echo "sleep 5"

sleep 5

PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"

if [ -n "$PIDS" ]; then

echo "kill -9 ${PIDS}"

kill -9 ${PIDS}

if [ -f ${PIDFILE} ]; then

echo "rm ${PIDFILE}"

rm ${PIDFILE}

}

service_status() {

PID="`cat ${PIDFILE}`"

if [ -n "$PID" ]; then

echo "${PIDFILE} : ${PID}"

else

echo "file doesnt' exist : ${PIDFILE}"

PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"

if [ -n "$PIDS" ]; then

echo "PROCESSES:"

echo "$PIDS"

}

service_start() {

PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"

if [ -n "$PIDS" ]; then

echo "process is present : ${PIDS}"

echo $PIDS > ${PIDFILE}

chmod 644 ${PIDFILE}

exit 0

else

gconftool-2 -s -t bool /desktop/gnome/remote_access/enabled true

gconftool-2 --type bool --set /desktop/gnome/remote_access/prompt_enabled 0

export DISPLAY=:0.0

/usr/lib/vino/vino-server --sm-disable &

sleep 2

PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"

if [ -n "$PIDS" ]; then

echo "$PIDS"

echo $PIDS > ${PIDFILE}

chmod 644 ${PIDFILE}

}

service_stop() {

PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"

if [ -n "$PIDS" ]; then

echo "kill ${PIDS}"

kill ${PIDS}

if [ -f ${PIDFILE} ]; then

echo "rm ${PIDFILE}"

rm ${PIDFILE}

}

case $1 in

start)

service_start;;

stop)

service_stop;

service_kill;;

kill)

service_kill;;

restart)

service_stop;

service_start;;

status)

service_status;;

echo "usage: $0 {start|stop|kill|restart|status}" ;

echo "example: $0 start" ;

esac

exit 0

Mount Disks

按左下角 Show Applications, 輸入 Disks

按下 Disks

選擇左側儲存裝置

按設定按鈕，選擇 Edit Mount Options

User Session Defaults 選擇 OFF

輸入 Mount Point

按下 Mount selected partition

2021年11月25日星期四

auto mount nfs after boot

$ sudo vi /etc/fstab

ip:/folder /media/folder auto noauto,x-systemd.automount 0 0

Install JetPack 4.6

參考 https://yingrenn.blogspot.com/2020/05/install-jetpack-44.html

$ sudo apt update

$ sudo dpkg -i sdkmanager_1.6.1-8175_amd64.deb

(Reading database ... 346387 files and directories currently installed.)

Preparing to unpack sdkmanager_1.6.1-8175_amd64.deb ...

Unpacking sdkmanager (1.6.1-8175) over (1.6.1-8175) ...

dpkg: dependency problems prevent configuration of sdkmanager:

sdkmanager depends on libgconf-2-4; however:

Package libgconf-2-4 is not installed.

sdkmanager depends on libcanberra-gtk-module; however:

Package libcanberra-gtk-module is not installed.

dpkg: error processing package sdkmanager (--install):

dependency problems - leaving unconfigured

Processing triggers for gnome-menus (3.13.3-11ubuntu1.1) ...

Processing triggers for desktop-file-utils (0.23-1ubuntu3.18.04.2) ...

Processing triggers for mime-support (3.60ubuntu1) ...

Processing triggers for hicolor-icon-theme (0.17-2) ...

Errors were encountered while processing:

sdkmanager

出錯，使用下列命令

$ sudo apt --fix-broken install

$ sdkmanager

2020年12月14日星期一

Record CSI-Camera to h264 video

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=1920, height=1080,format=NV12, framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvv4l2h264enc preset-level=4 MeasureEncoderLatency=1 ! 'video/x-h264, stream-format=(string)byte-stream, alignment=(string)au' ! filesink location=file.h264 -e

gst-launch-1.0 filesrc location=file.h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! nveglglessink -e

2020年12月9日星期三

TLT+DeepStream on ICDAR-2013

到 ICDAR-2013 的網站下載圖片和標註

ICDAR-2013 在 Robust Reading Competition 網站

要下載前要註冊

選 Challenges/Focused Scene Text

2013 的文字比較正，標註只有兩點

2015 的文字有各種角度，標註有四點

產生 kitti 資料時，直接使用訓練效果不好

轉換圖形時採用不同策略，一圖拆成多圖

截圖時不要破壞文字

TLT 有測試過 detectnet_v2 (resnet18, resnet50, darknet19), object_detection yolov3 (darknet19)

detectnet_v2 之 resnet18 precision: 14.6133 %

雖然 precision 不高，但是 tlt-infer 可以正確抓到文字

detectnet_v2 之 darknet19 precision: 18.7146 %

但 detectnet_v2 之 darknet19 失敗，tlt-infer 可以無法正確抓到文字

yolov3 之 darknet19 AP: 0.769

效果比 detectnet_v2 好

最看重 TLT 的功能是 tlt-prune, 神經網路被裁剪後

節省記憶體，加快速度

unprune Mem:2.5G PREF:8.33

0.5 Mem:2.1G PREF:28.83

0.7 Mem:2.1G PREF:30.00

DeepStream 之 nvinfer 的 config-file-path 設定檔

tlt-encoded-model=yolo_darknet19_pruned_70.etlt

model-engine-file=yolo_darknet19_pruned_70.engine

第一次啟動會很慢，因為在產生 yolo_darknet19_pruned_70.etlt_b1_gpu0_fp16.engine 檔

$ mv yolo_darknet19_pruned_70.etlt_b1_gpu0_fp16.engine yolo_darknet19_pruned_70.engine

之後會直接使用 yolo_darknet19_pruned_70.engine

以下是測試歷程記錄

2020年12月4日星期五

Build TensorRT OSS on Jetson

參考 TensorRT Open Source Software (OSS)

更新 CMake
$ wget http://www.cmake.org/files/v3.13/cmake-3.13.0.tar.gz
$ tar xpvf cmake-3.13.0.tar.gz cmake-3.13.0
$ cd cmake-3.13.0
$ ./bootstrap --system-curl
$ make -j4

$ sudo ln -s /home/nano/Data/cmake-3.13.0/cmake /usr/local/bin/cmake

$ git clone -b release/7.1 https://github.com/nvidia/TensorRT

$ cd TensorRT/

$ git submodule update --init --recursive

$ export TRT_SOURCE=`pwd`

$ cd $TRT_SOURCE

$ mkdir -p build && cd build

$ /usr/local/bin/cmake .. -DGPU_ARCHS=53 \

-DTRT_LIB_DIR=/usr/lib/aarch64-linux-gnu/ \

-DCMAKE_C_COMPILER=/usr/bin/gcc \

-DCMAKE_CUDA_COMPIILER=/usr/local/cuda-10.2/bin/nvcc \

-DTRT_BIN_DIR=`pwd`/out

$ make nvinfer_plugin -j$(nproc)$ mkdir backup
$ sudo mv /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3 backup
$ sudo cp libnvinfer_plugin.so.7.1.3 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3
$ sudo ldconfig

由此下載 tlt-converter

$ mkdir tlt-converter

$ mv tlt_7.1.zip tlt-converter

$ cd tlt-converter

$ unzip tlt_7.1.zip

$ vi ~/.bashrc 加入
export TRT_LIB_PATH=”/usr/lib/aarch64-linux-gnu”
export TRT_INC_PATH=”/usr/include/aarch64-linux-gnu”

2020年11月27日星期五

jetson nano install tensorflow

$ sudo apt-get update

$ sudo apt-get install libhdf5-serial-dev hdf5-tools libhdf5-dev zlib1g-dev zip libjpeg8-dev liblapack-dev libblas-dev gfortran

$ sudo apt-get install python3-pip

$ sudo pip3 install -U pip testresources setuptools==49.6.0

$ sudo apt-get install virtualenv

$ make envs

$ cd envs

$ sudo pip3 install -U numpy==1.16.1 future==0.18.2 mock==3.0.5 h5py==2.10.0 keras_preprocessing==1.1.1 keras_applications==1.0.8 gast==0.2.2 futures protobuf pybind11

$ python3 -m virtualenv -p python3 tensorflow-2.3.1

$ source tensorflow-2.3.1/bin/activate

$ pip3 install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v44 tensorflow==2.3.1+nv20.11

$ python3 -m virtualenv -p python3 tensorflow-1.15.4

$ source tensorflow-1.15.4/bin/activate

$ pip3 install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v44 tensorflow==1.15.4+nv20.11

2020年11月26日星期四

xpra in jsetson nano

參考 Jetson Nano平台通过Xpra运行图形程序

以下部分步驟，直接跳過避不開的錯誤

$ sudo apt-get install libx11-dev libxtst-dev libxcomposite-dev libxdamage-dev \

libxkbfile-dev python-all-dev

$ sudo apt-get install libgtk-3-dev python3-dev python3-cairo-dev python-gi-dev cython3

$ sudo apt-get install xauth x11-xkb-utils

$ sudo apt-get install libx264-dev libvpx-dev yasm

$ sudo apt-get install libavformat-dev libavcodec-dev libswscale-dev

$ sudo apt-get install libturbojpeg-dev

$ sudo apt-get install libwebp-dev

$ sudo apt-get install uglifyjs brotli libjs-jquery libjs-jquery-ui gnome-backgrounds

$ sudo apt-get install python3-opengl python3-numpy python3-pil

$ sudo apt-get install python3-rencode python3-lz4 python3-dbus python3-cryptography \

python3-netifaces python3-yaml

$ sudo apt-get install python3-setproctitle python3-xdg python3-pyinotify python3-opencv

$ sudo apt-get install libpam-dev quilt xserver-xorg-dev xutils-dev \

xserver-xorg-video-dummy xvfb keyboard-configuration

$ sudo apt-get install python-kerberos python-gssapi

$ sudo apt-get install python-avahi

$ sudo apt-get install gstreamer1.0-pulseaudio gstreamer1.0-alsa \

gstreamer1.0-plugins-base gstreamer1.0-plugins-good \

gstreamer1.0-plugins-ugly python-gst-1.0

$ sudo apt-get install cups-filters cups-common cups-pdf python3-cups

$ sudo apt-get install openssh-client sshpass python3-paramiko

$ sudo apt-get install dh-systemd

$ wget https://www.xpra.org/src/xpra-4.0.4.tar.xz

$ tar -xf xpra-4.0.4.tar.xz

$ cd xpra-4.0.4

$ sudo ./setup.py install

$ vi setup.py

將第一行改成 python3

$ sudo ./setup.py install

$ sudo mkdir /usr/local/share/xpra/www/js/lib

$ sudo ./setup.py install

2020年11月25日星期三

LXDE 桌面 lightdm, VNC Server, 中文輸入

$ sudo dpkg-reconfigure lightdm

$ sudo apt update

為避免 lightdm 黑畫面，無法登入，需要鎖住 systemd 不要升級

$ sudo apt-mark hold systemd

若有一天問題解決，解開封鎖

$ sudo apt-mark unhold systemd

$ sudo apt upgrade

$ sudo apt install x11vnc

$ sudo x11vnc -storepasswd

$ sudo vi /etc/systemd/system/x11vnc.service

# Description: Custom Service Unit file

# File: /etc/systemd/system/x11vnc.service

[Unit]

Description="x11vnc"

Requires=display-manager.service

After=display-manager.service

[Service]

ExecStart=/usr/bin/x11vnc -loop -nopw -xkb -repeat -noxrecord -noxfixes -noxdamage -forever -rfbport 5900 -display :0 -auth guess -rfbauth /home/nano/.vnc/passwd

ExecStop=/usr/bin/killall x11vnc

Restart=on-failure

RestartSec=2

[Install]

WantedBy=multi-user.target

$ sudo systemctl enable x11vnc.service

$ sudo systemctl start x11vnc.service

$ xrandr -q

$ sudo vi /etc/X11/xorg.conf

Section "Monitor"

Identifier "DSI-0"

Option "Ignore"

EndSection

Section "Screen"

Identifier "Default Screen"

Monitor "Configured Monitor"

Device "Default Device"

SubSection "Display"

Depth 24

Virtual 1280 800

EndSubSection

EndSection

$ sudo apt-get install fcitx fcitx-chewing

Start up/Preferences/Language Support

Keyboard input method system: fcitx

Reboot

Start up/Preferences/Fcitx Configuration

按 + 並取消打勾 Only Show Current Language

加入 Chewing

2020年10月26日星期一

Rtsp service on jetson nano

下載 test-launch.c v1.18
安裝必須套件
$ sudo apt-get install libgstrtspserver-1.0 libgstreamer1.0-dev

編譯

$ gcc test-launch.c -o test-launch $(pkg-config --cflags --libs gstreamer-1.0 gstreamer-rtsp-server-1.0)

若出現

undefined reference to `gst_rtsp_media_factory_set_enable_rtcp'

編譯失敗，刪除相關程式碼即可

測試啟動 service

$ ./test-launch "videotestsrc ! omxh265enc ! rtph265pay name=pay0 pt=96"

開啟畫面 (Jetson 平台)

$ gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! application/x-rtp, media=video ! queue ! decodebin ! nvvidconv ! nveglglessink

開啟畫面 (一般平台)

$ gst-launch-1.0 rtspsrc location=rtsp://192.168.0.105:8554/test ! application/x-rtp, media=video ! queue ! decodebin ! nvvideoconvert ! nveglglessink

使用 CSI 攝影機
$ ./test-launch "nvarguscamerasrc ! \
video/x-raw(memory:NVMM), format=NV12, width=640, height=480, framerate=30/1 ! \
nvvidconv ! video/x-raw ! omxh265enc ! rtph265pay name=pay0 pt=96 config-interval=1"

2020年9月8日星期二

GStreamer 安裝

依據說明文件安裝

sudo apt-get install libgstreamer1.0-0

sudo apt-get install gstreamer1.0-plugins-base

sudo apt-get install gstreamer1.0-plugins-good

sudo apt-get install gstreamer1.0-plugins-bad

sudo apt-get install gstreamer1.0-plugins-ugly

sudo apt-get install gstreamer1.0-libav

sudo apt-get install gstreamer1.0-doc

sudo apt-get install gstreamer1.0-tools

sudo apt-get install gstreamer1.0-x

sudo apt-get install gstreamer1.0-alsa

sudo apt-get install gstreamer1.0-gl

sudo apt-get install gstreamer1.0-gtk3

sudo apt-get install gstreamer1.0-qt5

sudo apt-get install gstreamer1.0-pulseaudio

還需另外安裝

sudo apt-get install libgstreamer1.0-dev

sudo apt-get install libgstreamer-plugins-base1.0-dev

sudo apt-get install libgtk2.0-dev

sudo apt-get install clutter-gst-3.0

sudo apt-get install libclutter-gst-3.0-0

sudo apt-get install libclutter-gst-3.0-dev

下載教學範例

git clone https://gitlab.freedesktop.org/gstreamer/gst-docs

Makefile 中

PKGS := gstreamer-1.0 gtk+-2.0 clutter-gst-3.0

CFLAGS += `pkg-config --cflags $(PKGS)`

LIBS += `pkg-config --libs $(PKGS)`

$(MAKECMDGOALS).o: $(MAKECMDGOALS).c Makefile

$(CC) -c -o $@ $(CFLAGS) $<

$(MAKECMDGOALS): $(MAKECMDGOALS).o Makefile

$(CC) -o $@ $@.o $(LIBS)

2020年9月2日星期三

為在 jetson nano 的 conky 加速

conky 若是利用 exec, execp 等命令，會大量耗費資源

之前使用 execi 執行 tegrastats 取得一些系統資訊

直接改用 C, 編寫出 jetson_fan,

jetson_temp_ao, jetson_temp_cpu, jetson_temp_gpu,

jetson_power_cpu, jetson_power_gpu, jetson_power_all

等變數

$ vi .conkyrc

RAM:${jetson_ram_usage}${alignr}${jetson_ram}

RAM lfb:${alignr}${jetson_ram_lfb}

SWAP:${jetson_swap_usage}${alignr}${jetson_swap}

SWAP Cached:${alignr}${jetson_swap_cached}

EMC Bandwidth:${jetson_emc} ${jetson_emc_bar}

CPU0:${jetson_cpu_frq 0} ${jetson_cpu_bar 0}

CPU1:${jetson_cpu_frq 1} ${jetson_cpu_bar 1}

CPU2:${jetson_cpu_frq 2} ${jetson_cpu_bar 2}

CPU3:${jetson_cpu_frq 3} ${jetson_cpu_bar 3}

GPU ${jetson_gpu} ${jetson_gpu_bar}

${font :blod:size=12}Temperature ${hr}

${font :normal:size=10}Thermal:${alignr} ${jetson_temp_thermal}

PLL:${alignr} ${jetson_temp_pll}

CPU:${alignr} ${jetson_temp_cpu}

Wifi:${alignr} ${jetson_temp_wifi}

PMIC:${alignr} ${jetson_temp_pmic}

GPU:${alignr} ${jetson_temp_gpu}

AO:${alignr} ${jetson_temp_ao}

Fan Speed:${jetson_fan} ${jetson_fan_bar}

以下是 conky source 的 patch

2020年9月1日星期二

conky in jetson nano

參考 Conky Installation

安裝相依套件

$ sudo apt-get install cmake

$ sudo apt-get install libimlib2-dev

$ sudo apt-get install libncurses5-dev

$ sudo apt-get install libx11-dev

$ sudo apt-get install libxdamage-dev

$ sudo apt-get install libxft-dev

$ sudo apt-get install libxinerama-dev

$ sudo apt-get install libxml2-dev

$ sudo apt-get install libxext-dev

$ sudo apt-get install libcurl4-openssl-dev

$ sudo apt-get install liblua5.3-dev

確認 cmake 版本要 >=3.8

$ cmake --version

2020年8月27日星期四

學習 How to configure your NVIDIA Jetson Nano for Computer Vision and Deep Learning

參考 How to configure your NVIDIA Jetson Nano for Computer Vision and Deep Learning

更新 system level packages

$ sudo apt-get update

$ sudo apt-get upgrade

安裝 system-level 相關套件

$ sudo apt-get install git

$ sudo apt-get install cmake

$ sudo apt-get install libatlas-base-dev

$ sudo apt-get install gfortran

$ sudo apt-get install libhdf5-serial-dev

$ sudo apt-get install hdf5-tools

$ sudo apt-get install python3-dev

$ sudo apt-get install locate

$ sudo apt-get install libfreetype6-dev

$ sudo apt-get install python3-setuptools

$ sudo apt-get install protobuf-compiler

$ sudo apt-get install libprotobuf-dev

$ sudo apt-get install openssl

$ sudo apt-get install libssl-dev

$ sudo apt-get install libcurl4-openssl-dev

$ sudo apt-get install cython3

$ sudo apt-get install libxml2-dev

$ sudo apt-get install libxslt1-dev

訂閱：文章 (Atom)

網頁

2025年6月18日 星期三

2025年6月12日 星期四

2024年10月16日 星期三

2024年8月28日 星期三

2023年6月6日 星期二

2022年11月25日 星期五

2022年10月19日 星期三

2021年11月25日 星期四

2020年12月14日 星期一

2020年12月9日 星期三

2020年12月4日 星期五

2020年11月27日 星期五

2020年11月26日 星期四

2020年11月25日 星期三

2020年10月26日 星期一

2020年9月8日 星期二

2020年9月2日 星期三

2020年9月1日 星期二

2020年8月27日 星期四

2025年6月18日星期三

2025年6月12日星期四

2024年10月16日星期三

2024年8月28日星期三

2023年6月6日星期二

2022年11月25日星期五

2022年10月19日星期三

2021年11月25日星期四

2020年12月14日星期一

2020年12月9日星期三

2020年12月4日星期五

2020年11月27日星期五

2020年11月26日星期四

2020年11月25日星期三

2020年10月26日星期一

2020年9月8日星期二

2020年9月2日星期三

2020年9月1日星期二

2020年8月27日星期四