網頁

2022年10月25日 星期二

tensorflow predict memory leak

記憶體越吃越多,直到系統當機
$ top
可看到 VIRT RES 越來越大
$ jtop
看到 Mem 也隨時間越來越大

查詢目前程式占用的記憶體
import psutil
psutil.Process().memory_info().rss / (1024*1024*1024),
psutil.Process().memory_info().vms / (1024*1024*1024),

查詢目前程式碼使用記憶體狀況
from memory_profiler import profile
@profile(precision=4,stream=open('memory_profiler.log','w+'))
def function()
@profile # 直接在 stdout 輸出
def function()
但是看不出所以然

網路上常說因為 numpy 到 tensor 轉換的原因
state = tf.convert_to_tensor(state)
model.predict(state)
state = tf.convert_to_tensor(state)
model.fit(states)
但是沒有用

垃圾收集
import gc
gc.collect()
但是也沒有用

最後一招,有用
import tensorflow as tf
tf.keras.backend.clear_session()

2022年10月20日 星期四

gym tensorflow 衝突

env.render()
出現錯誤
    from pyglet.gl import *
  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/gl/__init__.py", line 243, in <module>
    import pyglet.window
  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/window/__init__.py", line 1897, in <module>
    gl._create_shadow_window()
  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/gl/__init__.py", line 220, in _create_shadow_window
    _shadow_window = Window(width=1, height=1, visible=False)
  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/window/xlib/__init__.py", line 173, in __init__
    super(XlibWindow, self).__init__(*args, **kwargs)
  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/window/__init__.py", line 595, in __init__
    config = screen.get_best_config(template_config)
  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/canvas/base.py", line 192, in get_best_config
    configs = self.get_matching_configs(template)
  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/canvas/xlib.py", line 220, in get_matching_configs
    configs = template.match(canvas)
  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/gl/xlib.py", line 58, in match
    have_13 = info.have_version(1, 3)
  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/gl/glx_info.py", line 86, in have_version
    client_version = self.get_client_version().split()[0]
IndexError: list index out of range


解決方案為 env.render() 後才能 import tensorflow
import gym
env = gym.make("CartPole-v0")
env.render()
import tensorflow as tf

python 出現 cannot allocate memory in static TLS block

其實會出現這個問題是 gym tensorflow 衝突 原因
解決這個問題,就部會出現下列問題

OSError: /usr/lib/xxxx/libxxxx.so.0: cannot allocate memory in static TLS block

解決方法為
$ export LD_PRELOAD=/usr/lib/xxxx/libxxxx.so.0:$LD_PRELOAD

tensorflow 在 Xavier 出現 cannot allocate memory in static TLS block 錯誤

其實會出現這個問題是 gym tensorflow 衝突 原因
解決這個問題,就部會出現下列問題

Traceback (most recent call last):
  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/tensorflow/python/pywrap_tensorflow.py", line 62, in <module>
    from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: /home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/tensorflow/python/../../tensorflow_cpu_aws.libs/libgomp-d22c30c5.so.1.0.0: cannot allocate memory in static TLS block

$ vi .bashrc
export LD_PRELOAD=/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/tensorflow/python/../../tensorflow_cpu_aws.libs/libgomp-d22c30c5.so.1.0.0

2022年10月19日 星期三

安裝 Xavier JetPack 5.0.2

Download NVIDIA SDK Manager 1.8.4
https://developer.nvidia.com/nvidia-sdk-manager
選擇 .deb Ubuntu
$ sudo dpkg -i sdkmanager_1.8.4-10431_amd64.deb
[sudo] password for mark: 
Selecting previously unselected package sdkmanager.
(Reading database ... 340238 files and directories currently installed.)
Preparing to unpack sdkmanager_1.8.4-10431_amd64.deb ...
Unpacking sdkmanager (1.8.4-10431) ...
dpkg: dependency problems prevent configuration of sdkmanager:
 sdkmanager depends on libgconf-2-4; however:
  Package libgconf-2-4 is not installed.
 sdkmanager depends on libcanberra-gtk-module; however:
  Package libcanberra-gtk-module is not installed.

dpkg: error processing package sdkmanager (--install):
 dependency problems - leaving unconfigured
Processing triggers for gnome-menus (3.13.3-11ubuntu1.1) ...
Processing triggers for desktop-file-utils (0.23-1ubuntu3.18.04.2) ...
Processing triggers for mime-support (3.60ubuntu1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Errors were encountered while processing:
 sdkmanager
出錯,使用下列命令修正
$ sudo apt --fix-broken install
$ sdkmanager
ADDITIONAL SDKS/DeepStream 要打勾
按 CONTINUE TO STEP 02
勾選 I accept the terms and conditions of the license aggrements
按 CONTINUE TO STEP 03
選擇 Manual Setup = Jetson AGX Xavier

角落起算
1. Power Button
2. Force Recovery Button
3. Reset Button

將 xavier 40-Pin 邊的(電源指示燈旁的) USB 連接至安裝主機
關機後先壓工程模式鍵(Recovery)再同時按壓電源鍵(Power),兩秒後放開進入工程模式

開啟螢幕,等待 Xavier 安裝完成開啟登入畫面
不用登入,按下 Install,繼續等待


$ sudo apt-get install python3-pip
$ sudo -H pip3 install jetson-stats
$ sudo systemctl restart jetson_stats.service
$ sudo jtop
$ sudo vi /etc/nvfancontrol.conf
        FAN_PROFILE full {
                #TEMP   HYST    PWM     RPM
                0       8       255     3640
                140     0       255     3640
        }
FAN_DEFAULT_PROFILE quite
FAN_DEFAULT_PROFILE cool

$ sudo systemctl stop nvfancontrol.service
$ sudo rm /var/lib/nvfancontrol/status
$ sudo vi /etc/nvfancontrol.conf
$ sudo systemctl start nvfancontrol.service
$ sudo cat /var/lib/nvfancontrol/status

$ sudo apt update
$ mkdir -p ~/.config/autostart
$ cp /usr/share/applications/vino-server.desktop ~/.config/autostart/
$ gsettings set org.gnome.Vino prompt-enabled false
$ gsettings set org.gnome.Vino require-encryption false
$ gsettings set org.gnome.Vino prompt-enabled false
$ gsettings set org.gnome.Vino require-encryption false
$ gsettings set org.gnome.Vino authentication-methods "['vnc']"
$ gsettings set org.gnome.Vino vnc-password $(echo -n 'ChangeToYourPasswd'|base64)
$ sudo vi /etc/gdm3/custom.conf
WaylandEnable=false
AutomaticLoginEnable = true
AutomaticLogin = UserLoginName

但發現 vino 比 x11vnc 慢,所以刪除 vino-server.desktop, 回復 custom.conf,改安裝 x11vnc
$ sudo apt-get install x11vnc
$ x11vnc -storepasswd
$ vi x11vnc_0.sh #其中124為gdm的user id, 可在 /etc/passwd 查詢
sudo /usr/bin/x11vnc -display :0 -auth /var/run/user/124/gdm/Xauthority -rfbauth /home/UserName/.vnc/passwd
$ cat x11vnc_1.sh
DISP=`ps -u $(id -u) -o pid= | \
    while read pid; do
        cat /proc/$pid/environ 2>/dev/null | tr '\0' '\n' | grep '^DISPLAY=:'
    done | grep -o ':[0-9]*' | sort -u`
DISPLAY=$DISP x11vnc -rfbport 5900 -rfbauth /home/UserName/.vnc/passwd
$ chmod +x x11vnc_0.sh x11vnc_1.sh
$ ./x11vnc_0.sh
使用 vnc client 登入,並輸入使用者名稱和密碼,然後變成黑畫面
別擔心此時主機已經登入,直接關閉 vnc client
$ ./x11vnc_1.sh
在使用一次 vnc client 登入即可

$ sudo apt-get install xserver-xorg-video-dummy
$ sudo vi /etc/X11/xorg.conf
Section "Module"
    Disable     "dri"
    SubSection  "extmod"
    Option  "omit xfree86-dga"
    EndSubSection
EndSection

Section "Device"
    Identifier  "Tegra0"
    Driver      "nvidia"
    # 必須改成 false, 否則未接螢幕時, vino 會顯示 nvidia 的 long
    Option      "AllowEmptyInitialConfiguration" "false"
EndSection

Section "ServerFlags"
    Option "AutoAddGPU" "false"
EndSection

Section "Screen"
    Identifier  "Nvidia Screen"
    Device      "Tegra0"
EndSection

Section "Device"
    Identifier  "Dummy Video Device"
    Driver      "dummy"
    VideoRam    256000
EndSection

Section "Monitor"
    Identifier  "Dummy Monitor"
    HorizSync   5.0-1000.0
    VertRefresh 5.0-200.0
    Modeline    "1920x1080" 173.00 1920 2048 2248 2576 1080 1083 1088 1120 -hsync +vsync
EndSection

Section "Screen"
    Identifier  "Dummy Screen"
    Monitor     "Dummy Monitor"
    Device      "Dummy Video Device"
    SubSection  "Display"
        Depth   24
        Virtual 1920 1080
    EndSubSection
EndSection

Section "ServerLayout"
    Identifier "Main Layout"
    Screen 0 "Nvidia Screen"
    Screen 1 "Dummy Screen"
EndSection

vi vino.sh
#!/bin/bash

### BEGIN INIT INFO
# Provides:          vino-server
# Required-Start:    $local_fs
# Required-Stop:     $local_fs
# Should-Start:
# Should-Stop:
# X-Start-Before:
# X-Stop-After:
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: vino-server
### END INIT INFO

PIDFILE="/tmp/vino-server.pid"
NAME="vino-server"
GREPFILTER="/usr/lib/vino/vino-server"

service_kill() {

       ### kill -9
       PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"
       if [ -n "$PIDS" ]; then
           echo "sleep 5"
           sleep 5
           PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"
           if [ -n "$PIDS" ]; then
               echo "kill -9 ${PIDS}"
               kill -9 ${PIDS}
           fi
       fi
       if [ -f ${PIDFILE} ]; then
           echo "rm ${PIDFILE}"
           rm ${PIDFILE}
       fi
}

service_status() {
       PID="`cat ${PIDFILE}`"
       if [ -n "$PID" ]; then
           echo "${PIDFILE} : ${PID}"
       else
           echo "file doesnt' exist : ${PIDFILE}"
       fi
       PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"
       if [ -n "$PIDS" ]; then
           echo "PROCESSES:"
           echo "$PIDS"
       fi
}

service_start() {
       PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"
       if [ -n "$PIDS" ]; then
            echo "process is present : ${PIDS}"
            echo $PIDS > ${PIDFILE}
            chmod 644 ${PIDFILE}
            exit 0
       else

            gconftool-2 -s -t bool /desktop/gnome/remote_access/enabled true
            gconftool-2 --type bool --set /desktop/gnome/remote_access/prompt_enabled 0
            export DISPLAY=:0.0
            /usr/lib/vino/vino-server --sm-disable &

            sleep 2

            PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"
            if [ -n "$PIDS" ]; then
                 echo "$PIDS"
                 echo $PIDS > ${PIDFILE}
                 chmod 644 ${PIDFILE}
            fi
       fi
}
service_stop() {
       PIDS="`ps aux |grep -v grep |grep -P "${GREPFILTER}" |awk '{print $2}' | tr '\n' ' ' `"
       if [ -n "$PIDS" ]; then
           echo "kill ${PIDS}"
           kill ${PIDS}
       fi
       if [ -f ${PIDFILE} ]; then
           echo "rm ${PIDFILE}"
           rm ${PIDFILE}
       fi
}

case $1 in
   start)
     service_start;;
   stop)
     service_stop;
     service_kill;;
   kill)
     service_kill;;
   restart)
     service_stop;
     service_start;;
   status)
     service_status;;
   *)
     echo "usage: $0 {start|stop|kill|restart|status}" ;
     echo "example: $0 start" ;
esac
exit 0

Mount Disks
按左下角 Show Applications, 輸入 Disks
按下 Disks
選擇左側儲存裝置
按設定按鈕,選擇 Edit Mount Options
User Session Defaults 選擇 OFF
輸入 Mount Point
按下 Mount selected partition