Todo#
Overall plan (feel free to edit this, I just wrote this down in 1 min)#
- Get raspi and camera
- Get communication between voxl2 and raspi working
- Get two camera streams from CSI cameras to displaying in raspberry pi
- try out algorithms for moving object or drone detection (can test by just walking around with the setup and moving other objects in the room simultaniously)
- mount the raspberry pi and cameras in the drone
- Adapt the algorithm to work on omnidirectional on the drone
- Test detection algorithm outdoors by looking at detections while having the drone hover and walking around, throwing objects, flying drones
- get prioritization algorithm and tracking algorithm working
- Test it indoors
- Test it outdoors
hardware#
360 camera with raspberry pi 5 (dont use 4 because we need 2 camera slots)
- Problem: exactly how many cameras, how to put them on
- answer: 220 fov cam: https://www.arducam.com/product/arducam-ultra-wide-angle-fisheye-5mp-ov5647-camera-for-raspberry-pi/
Setups to model in Onshape:#
FOV analysis above and below right and left front back Once FOV looks good, figure out mounting mechanism and compute weight:
- < 500g (add up the weights of everything raspi, cameras, mounts, screws, power mechanism)
software#
- Literature search:
- Problem statement: there exist 2 180 images, what detection algorithm do we run to detect flying objects
- There are SOTA algorihtms according to Zakhor: try google scholar search on “moving object detection” + sky
- This paper from Zakhor could be related: https://www-video.eecs.berkeley.edu/papers/lyang/Drone_Object_Detection_Using_RGB_IR_Fusion.pdf
- look at related work when searching for papers and get top mentioned papers
- idea from Zakhor: use a simulator like unreal engine to create synthetic data and train a neural network. (I think this is overkill and much simpler methods would be more reliable and fast but idk)
- Problem statement: there exist 2 180 images, what detection algorithm do we run to detect flying objects
- How do we decide between multiple tracked objects?
- How to re-identify lost objects and occluded objects
- there are algorithms for this according to Zakhor. Need literature search on this
Definition of a threat (discussed with Zakhor): Not a ground object and moving#
on the debate: USB cameras -> voxl2 vs CSI cameras -> raspberry pi with classical vision -> voxl2#
Bandwidth limitations of USB means we would most likely encounter problems if hooking up two cameras via USB. Comparison of using raspi vs usb camera
- Pros
- fast CSI connection from cameras mean higher frame rates and resolutions possible (probably leads to better detection and tracking)
- Raspberry pi is very common platform so many cameras that do >180 fov
- Cons
- Raspberry pi has low compute compared to Voxl so we have limited ability to run big algorithms (although there are things like google coral that can boost compute)
- Need to find a way to power the raspberry pi
- extra complexity of raspberry pi to voxl2 communication (although this should be very common problem and easily solved)
Notes#
Links to hardware (usb cameras too if thats useful)
https://www.arducam.com/product/arducam-imx291-usb-camera-with-case-b026101/
Mathematically proved >110 degree fov cameras can do full coverage with 6 cameras
anything less than 180 needs 4 cameras
180 degrees needs 2 cameras (this one is the best)