debug in Ubuntu

參考 讓程序崩潰時產生core dump
$ g++ -g test.cpp 

$ gcc -g test.cpp 

# 編譯程式時加入 -g 

$ ulimit -a core file size        (blocks, -c) 0

# 表示部會產生 core dump

$ ulimit -c unlimited

# 將 core file size 改為 unlimited

$ sudo apt install systemd-coredump

$ coredumpctl

# 列出所有的 core dump

$ coredumpctl gdb

# 打開最近的 core dump

$ coredumpctl gdb 123

# 打開 pid=123 的 core dump

(gdb) bt

# back trace 顯示錯誤的地方

(gdb) q

# 離開 gdb

Jetpack L4T 35.1.0 風扇

jtop 無法在這個版本控制和查看 風扇

$ vi get_fan.sh

TEMP=`cat /sys/class/hwmon/hwmon0/temp1_input`

TEMP=$(echo "scale=2; ${TEMP}/1000" | bc)

echo Thermal=${TEMP}C

PWM=`cat /sys/class/hwmon/hwmon2/pwm1`

RPM=`cat /sys/class/hwmon/hwmon1/rpm`

echo FAN=$(( PWM * 100 / 255 ))% ${RPM}rpm

Ubuntu 開關(當)機相關命令

顯示 Linux 開機資訊指令

dmesg -T

顯示系統關機記錄和運行級別改變的日誌

last -x

/var/log/syslog

/var/log/syslog.1

The system log typically contains the greatest deal of information by default about your Ubuntu system.

tensorflow predict memory leak

記憶體越吃越多，直到系統當機

$ top

可看到 VIRT RES 越來越大

$ jtop

看到 Mem 也隨時間越來越大

查詢目前程式占用的記憶體

import psutil

psutil.Process().memory_info().rss / (1024*1024*1024),

psutil.Process().memory_info().vms / (1024*1024*1024),

查詢目前程式碼使用記憶體狀況

from memory_profiler import profile

@profile(precision=4,stream=open('memory_profiler.log','w+'))

def function()

@profile # 直接在 stdout 輸出

def function()

但是看不出所以然

網路上常說因為 numpy 到 tensor 轉換的原因

state = tf.convert_to_tensor(state)

model.predict(state)

state = tf.convert_to_tensor(state)

model.fit(states)

但是沒有用

垃圾收集

import gc

gc.collect()

但是也沒有用

最後一招，有用

import tensorflow as tf

tf.keras.backend.clear_session()

gym tensorflow 衝突

env.render()

出現錯誤

    from pyglet.gl import *

  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/gl/__init__.py", line 243, in <module>

    import pyglet.window

  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/window/__init__.py", line 1897, in <module>

    gl._create_shadow_window()

  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/gl/__init__.py", line 220, in _create_shadow_window

    _shadow_window = Window(width=1, height=1, visible=False)

  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/window/xlib/__init__.py", line 173, in __init__

    super(XlibWindow, self).__init__(*args, **kwargs)

  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/window/__init__.py", line 595, in __init__

    config = screen.get_best_config(template_config)

  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/canvas/base.py", line 192, in get_best_config

    configs = self.get_matching_configs(template)

  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/canvas/xlib.py", line 220, in get_matching_configs

    configs = template.match(canvas)

  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/gl/xlib.py", line 58, in match

    have_13 = info.have_version(1, 3)

  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/pyglet/gl/glx_info.py", line 86, in have_version

    client_version = self.get_client_version().split()[0]

IndexError: list index out of range

解決方案為 env.render() 後才能 import tensorflow

import gym

env = gym.make("CartPole-v0")

env.render()

import tensorflow as tf

python 出現 cannot allocate memory in static TLS block

其實會出現這個問題是 gym tensorflow 衝突 原因

解決這個問題，就部會出現下列問題

OSError: /usr/lib/xxxx/libxxxx.so.0: cannot allocate memory in static TLS block

解決方法為

$ export LD_PRELOAD=/usr/lib/xxxx/libxxxx.so.0:$LD_PRELOAD

tensorflow 在 Xavier 出現 cannot allocate memory in static TLS block 錯誤

其實會出現這個問題是 gym tensorflow 衝突 原因

解決這個問題，就部會出現下列問題

Traceback (most recent call last):

  File "/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/tensorflow/python/pywrap_tensorflow.py", line 62, in <module>

    from tensorflow.python._pywrap_tensorflow_internal import *

ImportError: /home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/tensorflow/python/../../tensorflow_cpu_aws.libs/libgomp-d22c30c5.so.1.0.0: cannot allocate memory in static TLS block

$ vi .bashrc

export LD_PRELOAD=/home/UserName/envs/tf2.10.0/lib/python3.8/site-packages/tensorflow/python/../../tensorflow_cpu_aws.libs/libgomp-d22c30c5.so.1.0.0