Co když zkombinujete stereo-kameru s malým počítačem s podporou pro výpočet neuronek? Přidáte-li možnost nasvícení scény a vše to zabalíte do malé krabičky tak dostanete OAK-D od firmy Luxonis. Ale to rozhodně není vše … Blog update: 18/3/2024 — Debugging DepthAI pipeline

OAK-D Pro (převzato z webu Luxonis)

• https://shop.luxonis.com/products/oak-d-pro
• DepthAI Hardware Documentation
• OAK-D Pro doc
• DepthAI’s Documentation
• First steps with DepthAI
• GitHub Luxonis

Obsah

• 220322 - Měsíc čekání
• 220325 - Druhý krok
• 220328 - Standalone mode
• 220329 - IMU a SLAM
• 220907 - FollowMe a USB kabel (Jirka)
• 221008 - SLAM na 6 řádek
• 240122 - OSGAR driver
• 240210 - Two Towers
• 240226 - Staronoví manažeři a přehrávání videa
• 240304 - Detekce kuželu pomocí YoloV7
• 240316 - Software syncing
• 240318 - Debugging DepthAI pipeline

Blog

22. březen 2022 — Měsíc čekání

A když už jsem měl krabičku doma, tak jsem se zase nemohl rozhoupat jí vůbec otevřít. Začal jsem totiž pořádně „listovat” dokumentací a narazil jsem na: Due to the large power consumption of dot projector/illumination LED, a Y-adapter is required when using these functions! F-slovo. Znamená to, že teď budu muset objednávat nějaky Y-adaptér, abych si mohl kameru vůbec pustit?! To mi asi vzalo trošku vítr z plachet. Pak jsem alespoň otevřel krabičku a našel jsem tam toto:

Obsah krabičky

• Y-adaptér je součástí balení
• trošku to vypadá, že tam něco chybí, třeba kabel?

DepthAI selfie

Myšák

25. březen 2022 — Druhý krok

Ddr used / total - 0.13 / 339.99 MiB
Cmx used / total - 2.06 / 2.50 MiB
LeonCss heap used / total - 7.54 / 78.28 MiB
LeonMss heap used / total - 2.95 / 41.54 MiB
Chip temperature - average: 24.12, css: 25.67, mss: 23.69, upa: 23.44, dss: 23.69
Cpu usage - Leon CSS: 10.64%, Leon MSS: 1.98 %

28. březen 2022 — Standalone mode

29. březen 2022 — IMU a SLAM

• IMU Accelerometer & Gyroscope — GitHub imu_gyroscope_accelerometer.py
• IMU Rotation Vector — GitHub imu_rotation_vector.py

Accelerometer timestamp: 5613.242 ms
Accelerometer [m/s^2]: x: 5.200206 y: 0.019154 z: -8.820239
Gyroscope timestamp: 5613.925 ms
Gyroscope [rad/s]: x: 0.014914 y: 0.003196 z: -0.008522
Accelerometer timestamp: 5615.379 ms
Accelerometer [m/s^2]: x: 5.190629 y: 0.000000 z: -8.820239
Gyroscope timestamp: 5616.424 ms
Gyroscope [rad/s]: x: 0.009587 y: 0.001065 z: -0.005326

Rotation vector timestamp: 2707.787 ms
Quaternion: i: 0.000305 j: -0.964600 k: -0.001465 real: 0.263672
Accuracy (rad): 3.141602
Rotation vector timestamp: 2709.946 ms
Quaternion: i: 0.000305 j: -0.964600 k: -0.001465 real: 0.263672
Accuracy (rad): 3.141602

• SLAM with OAK … tak to asi nebude jen „copy and paste”: We plan on creating our own visual-inertial SLAM project that would use on-device feature tracking (using FeatureTracker node) and intertial data (using IMU node), if the device has an IMU sensor onboard.

7. září 2022 — FollowMe a USB kabel (Jirka)

8. října 2022 — SLAM na 6 řádek

import depthai
import spectacularAI
import time

pipeline = depthai.Pipeline()

vio_pipeline = spectacularAI.depthai.Pipeline(pipeline)
# optional config args: vio_pipeline = spectacularAI.depthai.Pipeline(pipeline, config, useStereo=False)

with depthai.Device(pipeline) as device, \
    vio_pipeline.startSession(device) as vio_session:

    while True:
        out = vio_session.waitForOutput()
        print(out.asJson())

mkvirtualenv slam
workon slam
pip install depthai
pip install spectacularAI

(slam) md@md-ThinkPad-P50:~/git/slam$ python vio_slam.py
Traceback (most recent call last):
  File "vio_slam.py", line 10, in 
    with depthai.Device(pipeline) as device, \
RuntimeError: No available devices

(slam) md@md-ThinkPad-P50:~/git/slam$ python vio_slam.py
Spectacular AI SDK: WARN: unrecoginzed OAK board name 'OAK-D-PRO'
you may need to manually set IMU-to-camera extrinsics (configuration.imuToCameraLeft)
{
  "acceleration":{"x":3.1929334554640985e-06,"y":-1.8542926966551022e-06,"z":-4.7221780476648234e-07},
  "angularVelocity":{"x":0.0015828333562240005,"y":0.012700891122221947,"z":0.00035301301977597177},
  "orientation":{"w":0.20955791688410802,"x":0.6385671367191919,"y":-0.7166392482573564,"z":-0.18640114051633144},
  "position":{"x":0.006880179754191055,"y":-0.02585324999779014,"z":-0.0037064156909441524},
  "status":"TRACKING","time":343134.285796941,
  "velocity":{"x":-0.1518426175218836,"y":0.050841540677380274,"z":0.0363924090447811}
}
…

{'x': -9.319226940036552, 'y': 1.5651737606558058, 'z': -5.689111896116633}
{'x': -9.724962378082667, 'y': 1.683750553335214, 'z': -5.987711949231651}
{'x': -10.173431395097694, 'y': 1.8181065640771044, 'z': -6.323567343157043}
{'x': -10.596265664774739, 'y': 1.947684970617886, 'z': -6.645771182786724}
{'x': -11.028292374467366, 'y': 2.0827255306275148, 'z': -6.980379856466121}
{'x': -11.50425102807769, 'y': 2.2345457195224334, 'z': -7.355377866839724}
{'x': -11.956292577500529, 'y': 2.3815888137119883, 'z': -7.717576736330567}

22. ledna 2024 — OSGAR driver

(osgar) md@md-ThinkPad-P50:~/git/osgar$ cat config/test-oak-camera.json
{
  "version": 2,
  "robot": {
    "modules": {
      "app": {
          "driver": "osgar.drivers.oak_camera:OakCamera",
          "init": {
            "fps": 10,
            "is_color": true,
            "is_depth": true,
            "laser_projector_current": 1200,
            "is_imu_enabled": true,
            "number_imu_records": 10,
            "disable_magnetometer_fusion": false,
            "cam_ip": "169.254.1.222",

            "mono_resolution": "THE_400_P",
            "color_resolution": "THE_1080_P",
            "color_manual_focus": 130,

            "stereo_median_filter": "KERNEL_3x3",
            "stereo_mode": "HIGH_ACCURACY",
            "stereo_extended_disparity": false,
            "stereo_subpixel": false,
            "stereo_left_right_check": true
          }
      }
    },
    "links": []
  }
}

(osgar) md@md-ThinkPad-P50:~/git/osgar$ python -m osgar.record –duration 10 config/test-oak-camera.json
2024-01-17 23:10:13,192 __main__   INFO     /home/md/git/osgar/test-oak-camera-240117_221013.log
2024-01-17 23:10:13,230 __main__   INFO     SIGINT handler installed
2024-01-17 23:10:13,752 osgar.drivers.oak_camera WARNING  IP 169.254.1.222 was not found!
2024-01-17 23:10:13,752 osgar.drivers.oak_camera INFO     Found devices: 1.6
2024-01-17 23:10:13,752 osgar.drivers.oak_camera INFO     Used the first available device.

(osgar) md@md-ThinkPad-P50:~/git/osgar$ python -m osgar.logger test-oak-camera-240117_221013.log
 k                 name    bytes | count | freq Hz
----------
 0                  sys      803 |  5 |   0.5Hz
 1            app.depth 36361514 | 71 |   6.7Hz
 2            app.color 25732424 | 71 |   6.7Hz
 3 app.orientation_list    38570 | 70 |   6.6Hz

Total time 0:00:10.574191

RGB obrázek přes OSGAR driver

hloubková data

[2024-01-26 16:19:36.660] [depthai] [warning] Insufficient permissions to communicate with X_LINK_UNBOOTED
                                              device having name "1.3". Make sure udev rules are set

RuntimeError: No available devices

# Allow all users to read and write to Myriad X devices
    echo "Installing udev rules…"
    echo 'SUBSYSTEM"usb", ATTRS{idVendor}"03e7", MODE="0666"' |
                 sudo tee /etc/udev/rules.d/80-movidius.rules > /dev/null
    sudo udevadm control –reload-rules && sudo udevadm trigger

10. února 2024 — Two Towers

RGB obrázek

hloubkový obrázek

RGB obrázek

hloubkový obrázek

26. února 2024 — Staronoví manažeři a přehrávání videa

from depthai_sdk import OakCamera
with OakCamera(replay='https://www.youtube.com/watch?v=Y1jTEyb3wiI') as oak:
    oak.replay.set_loop(True)  # <— Enable looping of the video, so it will never end
    color = oak.create_camera('color')
    nn = oak.create_nn('vehicle-detection-0202', color)
    oak.visualize(nn, fps=True)
    oak.start(blocking=True)

Realtime anotování YouTube videa

• https://docs.luxonis.com/projects/sdk/en/old_sdk_managers/getting-started/ (BlobManager, NNetManager, atd.)

• https://docs.luxonis.com/projects/sdk/en/latest/oak-camera/ (OakCamera vládne všem)

4. března 2024 — Detekce kuželu pomocí YoloV7

git clone https://github.com/luxonis/depthai-experiments.git
cd depthai-experiments/gen2-yolo/device-decoding
python main.py -config ~/git/Coneslayer/coneslayer.json -recording ~/git/osgar/follow-path.mp4

img = cv2.imread(str(args.input_image), cv2.IMREAD_COLOR)  # BGR, HWC
img = cv2.resize(img, (args.width, args.height))
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)  # RGB, HWC
img = img.transpose(2, 0, 1)  # RGB, CHW
img = numpy.expand_dims(img, 0)  # NCHW, i.e. including a "batch" dimension.
img = (img / 255.).astype(numpy.float32)  # [0..1] range

Detekce kuželu přes ONNX

(depthai) md@md-ThinkPad-P50:~/git/depthai-experiments/gen2-yolo/device-decoding$ python main.py -h
usage: main.py [-h] [-conf CONFIG] [-recording RECORDING] [-rgbr RGBRESOLUTION]
               [-rgbf RGBFPS] [-monor MONORESOLUTION] [-monof MONOFPS] [-fps FPS]
               [-defaultRes {None,min,max}] [-isp ISPSCALE] [-sharpness SHARPNESS]
               [-lumaDenoise LUMADENOISE] [-chromaDenoise CHROMADENOISE]
               [-manualFocus MANUALFOCUS] [-afMode AFMODE] [-awbMode AWBMODE]
               [-sceneMode SCENEMODE] [-abMode ANTIBANDINGMODE]
               [-effectMode EFFECTMODE] [-dct DISPARITYCONFIDENCETHRESHOLD]
               [-lrct LRCTHRESHOLD] [-sig SIGMA] [-med {0,3,5,7}]
               [-lrc STEREOLRCHECK] [-ext EXTENDEDDISPARITY] [-sub SUBPIXEL]
               [-maxd MAXDEPTH] [-mind MINDEPTH] [-sbbsf SBBSCALEFACTOR]
               [-openvinoVersion {2020_3,2020_4,2021_1,2021_2,2021_3,2021_4,2022_1,UNIVERSAL}]
               [-xls XLINKCHUNKSIZE] [-tun CAMERATUNING] [-dev DEVICEID]
               [-usbs {usb2,usb3}] [-irDotBrightness IRDOTBRIGHTNESS]
               [–irFloodBrightness IRFLOODBRIGHTNESS]
…

16. března 2024 — Software syncing

python -m osgar.logger oak-cone-detection-240315_203533.log -stream oak.detections  oak.color \
                                                            -format "{timestamp} {len(data)}"

0:00:03.442914 219488
0:00:03.487640 4
0:00:03.489948 214840   <====
0:00:03.517288 309346
0:00:03.540213 301433
0:00:03.566320 4
0:00:03.566519 290162
0:00:03.589637 4
0:00:03.599398 299662
0:00:03.615203 304556
0:00:03.638670 4
0:00:03.664591 4
0:00:03.664854 283100
0:00:03.681872 222737
0:00:03.713446 4
0:00:03.724757 4
0:00:03.743658 733428
0:00:03.747007 375482
0:00:03.776240 4
0:00:03.787632 4
0:00:03.790691 386878
0:00:03.812691 408704

seq_num = packets[-1].getSequenceNum()  # for sync of various outputs
dt = packets[-1].getTimestamp()  # datetime.timedelta
timestamp_us = ((dt.days * 24 * 3600 + dt.seconds) * 1000000 + dt.microseconds)

5        oak.depth_seq     1864 | 166 |   8.3Hz
6        oak.color_seq      341 |  31 |   1.5Hz
7   oak.detections_seq      341 |  31 |   1.5Hz

OSGAR time, OSGAR stream, [seq_num, timestamp_us]
0:00:03.472923 6 [0, 163893263641]
0:00:03.516000 7 [0, 163893263641]
0:00:03.518708 6 [2, 163893313635]
0:00:03.547843 6 [3, 163893338631]  <==
0:00:03.567436 5 [0, 163893348213]
0:00:03.569827 6 [4, 163893363629]
0:00:03.596117 7 [2, 163893313635]
0:00:03.596373 6 [5, 163893388627]
0:00:03.619423 7 [3, 163893338631]  <==
0:00:03.619587 6 [6, 163893413624]
0:00:03.646627 6 [7, 163893438618]
0:00:03.650795 5 [1, 163893448204]
0:00:03.666352 7 [4, 163893363629]

18. března 2024 — Debugging DepthAI pipeline

výřez pro preview - převzato z Luxonis webu

Detekce kuželů přes OSGAR HIL (hardware-in-the-loop)