czech english

Robot Challenge 2014

let's beat the human race in AirRace!

Robot Challenge is well established competition with many categories and many participating robots. One of them is flying 8-figures in net protected area — AirRace. There used to be also a semi-autonomous category for the first two years, where the best competitor Flying Dutchman reached 26 rounds. It is time to beat his score with autonomous drone now! Blog update: 11/4 — Conclusion

Note, that this is not just „English translation” of Czech article. It seems that fund raising campaign failed (well, there is still a week to go, but I doubt it will jump to required limit), so I decided to „log” my progress at least in English. Let me know, if you find this interesting .
We are playing with two robots" Heidi and Isabele. Both are AR Drone 2.0 from Parrot.
The source code is available at github: https://github.com/robotika/heidi
Here are some pictures from older Czech blog:

Links:



Blog

February 10th, 2014 - cv2

Yesterday I was fighting with OpenCV for Python, which seems to be just perfect tool to help us reach the goal. My problem was that I was following only the Python setup and it kind of worked (loaded and displayed image, for example). But as soon as I need to to parse video with H264 codec I was doomed :-(. There are many comments on the web that integration with ffmpeg is complicated, but there were already some DLLs prepared?!
Well, next time start with basic OpenCV installation and then the Python part. The only critical detail is that you have to set system path to binaries, ffmpeg DLLs in particular … yeah, quite obvious, I know.
Here is the first image dummy processing, and here is video replay code recorded by Isabele.

February 10th, 2014 - PaVE and cv2.VideoCapture

cv2.VideoCapture is nice function for access to video stream. Thanks to integrated FFMPEG you can read videos with H264 codec, and this is something used by Parrot AR Drone 2.0. The bad news is that it is not clean H264 stream, but there are also PaVE (Parrot Video Encapsulation) headers.
Does it matter? Well, it does. Yesterday I have seen couple time crashing Linux implementation, and on Windows you get several warnings like:
[h264 @ 03adc9a0] slice type too large (32) at 0 45
[h264 @ 03adc9a0] decode_slice_header error
The reason is that H264 control bytes 00 00 00 01 are quite common in PaVE header and then the decoeer is confused. Note, that most projects ignore this:
It is OK, if you just need „some” images, but once the decoder is confused, you loose up to 1 second of video stream until I-frame is successfully found.
So how to work around it? I had two ideas, both crazy and ugly. One is to create „TCP Proxy”, which would read images from AR Drone 2.0 and offer them to OpenCV via TCP socket. Another possibility is temporary file, where ret value is sometime False, but you can replay the video in parallel.
cap = cv2.VideoCapture( filename )
while(cap.isOpened()):
    ret, frame = cap.read()
There are probably more complex solutions like directly control FFMPEG API, but it is pity to lose this otherwise nice and clean code . Any ideas?

February 18th, 2014 - I-Frames only

Well, cv2 is still winning. I tried both — temporary file as well as TCP Proxy. It took a bit too long to start simple server (several seconds), which is not acceptable in „real-time video processing”. Temporary file looked more promising, but on Linux it was completely unusable. Hacking around cv2.VideoCapture is maybe not the best idea, and proper solution with buffer frame by frame decoding would be the right way, but …
Today's fall back was usage of I-Frames only. It does not work if you create video from I-Frames only, but you can create new file, write there single I-Frame, and then read it with cv2.VideoCapture. I know crazy. But it works for now and hopefully in meantime we will find some more effective solution. For details see source diff.
Heidi has now new flying code airrace_drone.py. It is glued together not to interfere with drone Isabele. Simple processFrame is then in airrace.py. Adaptive thresholding, detection of contours and for reasonable size contours compute bounding rectangle (with rotation).

February 19th, 2014 - Rotated rectangles

Writing a robo-blog in English is not as much fun as in Czech. At first my English is much worse and at second I do not have a clue if there is anybody reading it. That's something Fandorama was good for. I've got clear feedback how many people are interested so much that they would even support it. And almost nobody was interested in this blog in Czech, so … you get the picture. If you find a bit of information useful for you let me know. It helps to brighten up mine otherwise depressed spirit .
What I want to do now is revision of logged data from yesterday. The cv2 logs are normal text files with results of image analysis + number of updates between them. The last program was supposed to turn drone based on third parameter of first rectangle. Just a quick hack that I see that there is some feedback going on and that I can replay already recorded data with exact match.
NAVI-ON
-26.5650501251
-15.945394516
-45.0
-88.9583740234
NAVI-OFF
This is the output in console — angles used for rotation. It is also my first TODO, to add there video timestamps.
182
[((817.0, 597.0), (10.733125686645508, 12.521980285644531), -26.56505012512207),
 ((223.17218017578125,
 440.579833984375), (319.85919189453125, 60.09449768066406), -88.75463104248047)]
198
[((298.0, 508.0), (58.51559829711914, 319.50067138671875), -15.945394515991211)]
198
[((982.2500610351562, 492.25), (29.698482513427734, 17.677669525146484), -45.0),
 ((951.9046020507812,
507.705078125), (60.70906448364258, 84.2083511352539), -69.02650451660156)]
185
[((741.7918090820312, 247.95079040527344), (43.32011032104492, 72.9879379272461),
 -88.9583740234375),
 ((1035.0001220703125, 14.999971389770508), (16.970561981201172,
12.727922439575195), -45.0)]
199
My second TODO would be to round/truncate floating numbers to integers. It would be easier to read and remember, my goal is to turn left/right, i.e. something if mass center is bigger or smaller than width/2 …
182
[((817, 597), (10, 12), -26), ((223, 440), (319, 60), -88)]
198
[((298, 508), (58, 319), -15)]
198
[((982, 492), (29, 17), -45), ((951, 507), (60, 84), -69)]
185
[((741, 247), (43, 72), -88), ((1035, 14), (16, 12), -45)]
199
Better? I think so . Now I would almost guess that it is ((x,y,(width,height),rotation), but API is so simple that it is worth it to test it:
import cv2
import numpy as np
frame = np.zeros( (1200,720), np.uint8)
rect = ((223, 440), (319, 60), -88)
box = cv2.cv.BoxPoints(rect)
box = np.int0(box)
cv2.drawContours( frame,[box],0,(0,0,255),2)
cv2.imshow('image', frame)
cv2.imwrite( "tmp.jpg", frame )
cv2.waitKey(0)
cv2.destroyAllWindows()
Well, you probably know how to make it simper, but this is whole working program, which will display image and save it as reference. The width/height of image is swapped and I do not see there anything :-(, so it is not that great. In general I would recommend OpenCV-Python Tutorials. Note, that they are for non-existing (or better to say for not so easily available) version 3.0, and some cv2.cv.* functions are renamed (see notes).
So how to fix the program? Back to Drawing Functions example. Great!
# Create a black image
img = np.zeros((512,512,3), np.uint8)
so my fix is
frame = np.zeros( (720,1200,3), np.uint8)
and now I see red rectangle. With modification rect = ((223+300, 440), (319, 60), -88) and rect = ((223+600, 440), (319, 60), 0) I see that first coordinate is really (x,y), then (width,height) and finally angle in degrees, where Y-coordinate is pointing down … maybe I was too fast going to conclusion. One more test rect = ((223+600, 440), (319, 60), 45), so (x,y) is the center of rectangle! And now is time to read the manual . Hmm … The function used is cv2.minAreaRect(). It returns a Box2D structure which contains following detals - ( top-left corner(x,y), (width, height), angle of rotation ). Test first! Here is my accumulated image:
So it was not that boring at the end . No wonder drone did what she did. First it navigated to random square, then to perpendicular angle (but be aware … width is not necessary bigger than height, see ((298, 508), (58, 319), -15) … homework for you, was for this particular box correct to navigate to angle 0?

February 21st, 2014 - The Plan

Aleš wrote me e-mail, that it is not clear (from my description) what is the exact plan. And he is right!
The plan is to use standard notebook (one is running Win7 OS for Heidi, another Linux for Isabele) where all image processing will be handled. Navigation data plus video stream from drone is received via WiFi connection. WiFi is also used for sending AT commands to drone. The code is written in Python. For image processing is used OpenCV. The code is publicly available at https://github.com/robotika/heidi.
Aleš is also right regarding task complexity. According to his estimates it will be necessary to flight with average speed at least 1.5m/s. With video sampling rate 1Hz (current status), that will be quite hard. It is a challenge … like what the name of competition says „Robot Challenge”! See you in the area!
p.s. if this blog was sufficient impulse for Aleš to install and start to play with „OpenCV for Python” then I think that at least one alternative goal was already reached

February 26th, 2014 - Version 0 (almost)

Version 0 should be the simplest program solving given task. In the context of AirRace this means complete at least one 8-loop. And we are not there yet. Camera integration always slows down the development …
In meantime I also checked the other competitors. There are (or yesterday were) 12 of them! So it is clear that while in 2012 one completed round would be enough to win the competition, in 2013 it was enough for 3rd place (see Heidi) and in 2014? We will see.
Here is the source code diff for test-14-2-25. There were some surprises, as usual .

Z-coordinate

Probably the biggest surprise was test recorded in meta_140225_183844.log. Drone aggressively started and reached 2 meters height instead of usual 1 meter. That would be still OK, and soon it would reduce the height to desired 1.5 meter, but according to navdata the height was 1.43m! Moreover the restriction for minimum sizes of detected rectangles was set to 200 pixels so no navigation data was used/recognized.

Rectangles

One more comment about rectangles detection. Originally there was no limit on the size of rectangle, and that was wrong. Here are some numbers so you get an idea: at 1m the 30cm long strip is detected as 300 pixels rectangle. This means that at 2m it should be approximately 150 pixels (current filter value). Thanks to Jakub bad rectangles are now visible in the debug the output:
Now I am browsing through recorded videos and looking for „false positive” or how is it called — rectangles which should not be selected and were selected, and … here is a bad surprise:
We are still processing recorded video which should be complete. So it is possible that PaVE packet contained only part of the video frame. Another TODO.

Corrections in Y-coordinate

It turned out that it is necessary to make also sideways corrections. Sure enough there was a mistake in sign: image x-coordinate is drone Y coordinate and it should be positive for x=0. Fixed.

Hovering

The most important step was insertion of hovering. Just to get the drone at least a little bit under control. The video frames are processed at 1Hz so by setting the proportion between 0.0-1.0 splits the time for steering and hovering. 0.5s was too slow 0.8s too fast and 0.7 was reasonable at this stage. The plan is to trigger hovering only when the error grows above some predefined limit.

Results 14-2-25

The drone is extremely slow at the moment. In one minute it was not able to complete a semi-circle. It will be necessary to recognize which part of 8-loop is it and steer instead of go forward as default command.
Another detail to think about is going backward instead of forward. The down pointing camera is shifted little bit back. It may simplify navigation in some cases. Like test logged as meta_140225_185421.log, when Isabele flew across our table (we do not have protective nets), because only in one frame it had seen perpendicular strip … and then it was too late .

February 28th, 2014 - Bug in log process

Today I was investigating timestamps of video vs. navigation data. What scared me a lot was five seconds gap between initial frames:
1548.280426
0
1549.273254
-16
1550.32669 <— !!!!!!!!
-18
1555.131744 <— !!!!!!!!
-9
1557.108245 <— !!!!!!!!
-14
… here you see time in seconds and detected angle in degrees, just to find/match corresponding frame. So what is the problem? Here is the log file for image processing:
…
[((1009, 643), (74, 48), -18), ((1013, 624), (16, 12), -45), ((464, 482), (46, 266), -7), …]
178
[]
750
[((172, 209), (15, 8), -9)]
188
[]
184
[((327, 551), (43, 239), -14), ((244, 235), (43, 216), -14)]
190
[((1045, 676), (31, 38), -45), ((268, 483), (43, 236), -16)]
199
The results are interlaced with the number of updates of drone internal loop (running at 200Hz). The problematic number is 750 … so for such a long time there was no result?! This would mean that drone navigation was based on video frame several seconds old!
Well, good news is that in reality that was not so bad and the mistake is in logging. It is necessary to have all results different in order to store changes. Here for several seconds were no rectangles detected (i.e. empty list []) and these results were packed into one. If there would be timestamps and frame number the results would be always different and then it should work fine.
It is fixed now, and ready for new test (see github diff).

March 3rd, 2014 - Z-scaling

A quick observation based on today's Jakub tests: it is about strip (x,y) coordinates scaling based on the drone height (Z-scaling). The distance is estimated individually for each strip. I could not see any reason why three strips in line:
are not linear in the log viewer:
?! I already told you the reason … the z coordinate is estimated separately for each strip (based on its length) and the middle is much shorter (due to a light reflection), so it is expected to be further away …

March 4th, 2014 - A Brief History of Time

I could not resist to use Stephen Hawking book title, sorry . AR Drone 2.0 has at least two time sequences: NavData and Video. They are encoded differently (see for example Parrot forum), but good news is that they are probably related to same beginning of time. In this case not Bing Bang, but only Drone Boot.
The video stream is delayed. Not much for recording but maybe a lot for fast drone navigation. The number vary around 0.3s, but you can see also values like 0.44s and probably more. Here you can see the difference if you tight video frame to current pose or if you keep poseHistory:
before time correction
before time correction
after time correction
after time correction

March 5th, 2014 - test-14-3-4

Yesterday tests were quite discouraging :(. We had visitors and the best we could do was finish 1/2 of 8-loop. After the second turn Isabelle decided to „turn right” on the crossing and that was it. So we still do not have working version 0!!!
Now it is time to go through the log files and look for anomalies. Starting from the last test I can see 90 degrees jump in heading. This is probably normal as internal compass is taken into account and it takes while before enough data are collected.
Isabelle is not very high (approx. 1m) at the start position. It is pity that it won't recognize partially visible strip. Moreover the recognized strip is, due to light reflection, shorter so distance estimate is not very accurate.
There is an obvious problem with Y-coordinate correction. It is „one shot” only, with dead zone 5cm. For 10cm error it creates beautiful oscillations . On the other hand correction in angle with dead-zone +/- 15 degrees (!) does not correct for first three image snapshots at all … OK, now it is clear. Who programmed that?!
Another source of problems is failure to detect transition from circle to line. As result circle corrections are used instead of line navigation.
There are only 3 more tries/test evenings to go, so we will have to pick from the following list:
  • switch to smaller images, stream preview video (i.e not recorded), faster frame rate
  • C/Python binding to FFMPEG and parsing intermediate images (i.e. more than 1Hz video update rate)
  • MCL (Monte-Carlo Localisation) based on strips position
  • improve quality of strips image recognition
  • big image of the floor from matched and joined mosaic images
  • rectangle merging (detected two small which are split due to light reflection)
  • position driven corrections

March 6th, 2014 - Cemetery of Bugs

Hopefully I can call with this title today's commit . Sometimes it is hard to believe how many bugs can be in few lines of code. I already mentioned couple yesterday, so few more today:
drone.moveXYZA( sx, -sy/4., 0, 0 )
This was supposed to be replacement for hover() function in Y coordinate. The idea was that you send the drone with the opposite speed then which is flying now. So what is the problem? sy is not the drone speed! It is the command after image analysis for tilt right or left. It should be drone.vy instead.
Another bug, probably not so critical, is a block of code with condition, if any strip was detected. If there was no strip then history of poses was constantly increasing (in proper way it is cut with the timestamp of the last decoded video frame). Solved via completely removing the if statement — it is no longer needed. I no longer worry that I do not see any rectangle.
Then there was a surprise with ARDrone2.vy speed itself. I never need it, except for drone pose computation, and so it was with signs as you can get them from Parrot. Fixed in ardrone2.py.
So what next? Hopefully today will Jakub test new code with reference circles. Instead of navigation based on old images, Isabelle should now use circular path estimated from the image. I should be much better, but will be?

March 10th, 2014 - Shadow

Jakub sent me results of Friday tests with navigation along reference line with comment: we've got problems with shadows … and he was absolutely right:
This is not the first time when
ret, binary = cv2.threshold( gray, 0, 255, cv2.THRESH_OTSU )
fails. On the other hand it is not surprise — there are three peaks in histogram instead of two. The worst thing on this particular picture is that it not only does not detect any of three black strips, but it evaluates garbage as perpendicular strip!
For quick test I used only Irfan histogram to show what I mean:
Image histogram
Image histogram
Note, that I ignored red/green/blue pixels and picked blue histogram, because there you can see the best three peaks.

March 12th, 2014 - Finally Version 0!

I can sleep again . Not much (it is 5am now), but still it is much better . Yesterday Heidi finally broke the spell and after several weeks completed the figure eight loop!
Are you asking why Heidi and not here sister Isabelle? Well, there was a surprise. Isabelle does not have working compass! Yesterday I wanted to add extra direction filtering, basically to make sure that I am following the proper line. So I integrated magneto tag into ARDrone2 variables. I was using last 2 minutes test from Jakub (OT that was maybe another moment for celebration when Isabelle was able to autonomously fly and navigate for 2 minutes) and to my surprise there were constant raw data readings from compass:
compass:
-55	-57	-191
193.7109375	187.55859375	671.484375
-120.508392334	108.018455505	671.484375
314.219329834	79.5401382446	0.0	-121.496063232
0.0	-137.727676392 0x01	2
402.916381836	-241.082290649	0.0546875
This corresponds to following named variables:
values = dict(zip(['mx', 'my', 'mz',
  'magneto_raw_x', 'magneto_raw_y', 'magneto_raw_z',
  'magneto_rectified_x', 'magneto_rectified_y', 'magneto_rectified_z',
  'magneto_offset_x', 'magneto_offset_y', 'magneto_offset_z',
  'heading_unwrapped', 'heading_gyro_unwrapped', 'heading_fusion_unwrapped',
  'magneto_calibration_ok', 'magneto_state', 'magneto_radius',
  'error_mean', 'error_var'], magneto))
This was kind of magic — it is almost hard to believe how data fusion of gyro + compass perfectly hide this problem. In particular if you watch variables heading_unwrapped, heading_gyro_unwrapped, heading_fusion_unwrapped then first (compass heading) was constant (as well as mx,my,mz and others). Second gyro heading was slowly changing and finally in fusion heading was originally winner compass, but later on it was loosing against gyro . It is really hard to believe that it worked so well and except longer flights you would not notice it.
I could not resist and tried immediately Heidi at home in the kitchen. I cut the motors as soon as they started to turn (it was 6am, so neighbours would probably complain). There were two test: first drone pointing south and then drone pointing north. Values were different, so it is issue of Isabelle.
Now, what is worse than not working sensor? Any idea? Well, there is even worse case — sometimes not working sensor! After several tests flights with Heidi yesterday we tried final working algorithm also with Isabelle. And it worked fine! Do you see the persistent direction of 8 figure?
Just to get an idea, visible only form raw data, here is older 2 minutes flight mentioned sooner:
Note, that I had to change the scale so you will see the crazy trajectory …
Fun, isn't it? Now I would guess that communication with compass module sometimes fails and the only possibility is to power off the drone and restart the whole system. Maybe. Another TODO to go through older log files and check for this behavior.
So that was the bad news … now the good ones … and the title of today's post. We finally managed to complete the 8-loop and repeat it several times. There were three changes (see older diff):
  • decreased speed from 0.5m/s to 0.3m/s
  • wait on start if no strip was detected
  • fixed bug — missing abs()
The speed is obvious and does not need any explanation. The second point is related to elimination of most bad starts — the takeoff sequence is completed at about 1m above the ground and reasonable view from bottom camera is at about 1.5m. The algorithm slowly increases the height, but it should not leave the start area.
And the third one is the funny one . That was the reason why Isabelle or Heidi took wrong turn on the crossing. It was quite repeatable … fixed.
p.s. I wanted to post also link to YouTube video, but my video converter no longer works … it looks like there were some changes in FFmpeg … but maybe it is only to change avcodec_get_context_defaults2 to avcodec_get_context_defaults3 and replace AVMEDIA_TYPE_VIDEO by NULL as in this example. It looks OK and here is the video … note the strange behaviour near the end. Still plenty of work to do.

March 13th, 2014 - Frame contour

Did you watch the video of the first Isabelle's successful flight to the end? Can you guess what happened? If you have no idea then frame number 3630 from video_rec_140311_192140.bin can help you:
And here is the report of changes near the end:
FRAME 3540 I I
FRAME 3570 I I
TRANS I -> X
FRAME 3600 X I
TRANS I -> L
FRAME 3630 L L
NO dist change! -0.989344682643
TRANS L -> X
FRAME 3660 X I
FRAME 3690 I I
FRAME 3720 ? I
FRAME 3750 ? I
TRANS I -> L
FRAME 3780 L L
FRAME 3810 L L
FRAME 3840 L L
FRAME 3870 L L
NAVI-OFF 120.001953
Isabelle wrongly interpreted this image as turn left (!) … well, yeah … one pair looks like two following strips on left turn circle. And then there is a transition from circle to crossing, which should not ever happen where drone picked wrong line for navigation and started to fly loops in the opposite direction.
p.s. note, that Heidi (older sister with several injuries from the last year) in reality finished the 8-loop sooner, but in her last attempt (meta_140311_191029.log) unexpectedly landed in the middle of the test. What happened there? Low battery (on the start 39% and after landing 0%) … OK, so now we know that if battery is low done will automatically land .

March 14th, 2014 - cv2.contourArea(cnt, oriented=True)

VlastimilD wrote: the function cv2.contourArea has parameter "oriented", which is by default False and if set to True, it returns negative area which means the contour has the opposite orientation and that should be white on black i this case. So if you don't want those, setting oriented to True should filter them out in the following "if". … and he is absolutely right! Thanks for hint .
For details see OpenCV documentation.
p.s. I am sorry for misspeled name in the commit.

March 16th, 2014 - MP4_360P_H264_360P_CODEC

We have decided to switch to video recording in MP4_360P_H264_360P_CODEC mode. The previous mode was MP4_360P_H264_720P_CODEC, i.e. with higher resolution (1280x720) at 30fps. Our main motivation was to get rid of split images — cases, where you get mixed old and new image (for details see mixed video frame).
Another motivation was to reduce WiFi traffic and the size of recorded reference videos. So far so good. And yes, there was at least one surprise . The 720P video was not only with higher resolution but also on 30fps, and 360P video is running at 15fps! For us it is actually good news, because expected size will not be 4 times smaller but 8 times (in reality it won't be so big difference, because large and more frequent pictures are easier to pack).

March 17th, 2014 - Scale Blindness

Did you experience the feeling when the mistake is so big, out of range, that you won't see it? Here is the story: low resolution video does not work. The navigation algorithm is much much worse. Why? The error is too big to be seen … and it is already presented on all previous images .
Just for comparison here is one not working 360P image:
Example of 360P image
Example of 360P image
Do you see now, why cv2.THRESH_OTSU used to work fine before, but it is not working any more? No?
boring, right? The bottom facing camera has QVGA (320*240) resolution, according to Parrot specification . That resolution is neither compatible with 1280x720 nor with 640x360. But what is maybe not expected is that for high resolution you scale it 3 times (720=240*3) and fill the margin with black pixels, while for 640x360 you scale it twice (640=320*2) and cut the overlapping pixels! Histogram for high resolution video contained (1280-960)*720 totally black pixels …

March 18th, 2014 - Version 1 and MSER

It is still quite depressing — we would like to get 30 rounds and at the moment we are happy if we complete 3! Well, it is called progress, but up-to-now Version 0 with 8 times 8-loop was still better.
What is the main difference between Version 0 and Version 1? First of all it is the video input with smaller resolution. After yesterday discovery we know that the image is covering only 3/4th of reality visible on high resolution video. This means that you see two complete strips less frequently than before. There is a new class StripsLocalisation, which should compensate for this by taking into account also previous image and the movement of drone between snapshots.
There is not much time left — only one week so almost zero chance that we will integrate Version 2, which would handle absolute localization and compensate for multiple failures … oh well.
There was a good news at the end too: it is possible to use MSER (Minimally Stable External Regions) implementation in OpenCV in real-time image processing. Isabelle was able to complete several rounds with this algorithm. The video delay was comparable to normal binary threshold set for 5% of all pixels count.

March 19th, 2014 - MSER and negative area

The trick with negative area does not work for MSER output (!). And by the way it is also the function cv2.contourArea(cnt, oriented=True) which fails on Linux:
OpenCV Error: Unsupported format or combination of formats (The matrix can not
be converted to point sequence because of inappropriate element type) in
cvPointSeqFromMat, file
/build/buildd/opencv-2.4.2+dfsg/modules/imgproc/src/utils.cpp, line 59
0
Traceback (most recent call last):
  File "airrace.py", line 180, in 
    testPaVEVideo( filename )
  File "airrace.py", line 155, in testPaVEVideo
    print frameNumber( header )/15,  filterRectangles(processFrame( frame, debug=True ), minWidth=150/2)
  File "airrace.py", line 31, in processFrame
    area = cv2.contourArea(cnt, oriented=True)
cv2.error: /build/buildd/opencv-2.4.2+dfsg/modules/imgproc/src/utils.cpp:59:
error: (-210) The matrix can not be converted to point sequence because of
inappropriate element type in function cvPointSeqFromMat
Why it does not work is probably due to missing external contour, but it sometimes works, so I am not sure. If you want to play, here is the original picture, uncomment g_mser creation (airrace.py:27) and you can see yourself:
./airrace.py mser-orig.jpg
No, you won't see it! That's even more scary!! Forgot about oriented parameter for MSER area!!! The stored JPG file is a little bit different, but you get completely different results than from original video (see the sign for 1412):
m:\hg\robotika.cz\htdocs\competitions\robotchallenge\2014>m:\git\heidi\airrace.py mser-orig.jpg
1412.5
-3237.5
-7165.5
-897.5
-1488.0
-1261.0
492.5
-6224.0
vs.
m:\git\heidi>airrace.py logs\video_rec_140318_195005.bin 62
62 -6079.5
-3189.0
-1648.5
-32.5
-1202.0
-4846.5
-369.5
[((406, 49), (124, 23), -61), ((346, 209), (132, 23), -77)]
… I do not feel comfortable now :-(. For MSER just forget about area. It is filtered out at the beginning anyway, right?
g_mser = cv2.MSER( _delta = 10, _min_area=100, _max_area=300*50*2 )
I should probably learn more about the parameters…

March 23rd, 2014 - mosaic.py (5days remaining)

Couple days ago I wanted to play a little, so instead of reviewing more and more tests results, I tried to write mosaic.py — script for collection multiple images into one. The first plan was to feed the script with a list of images only, and the rest will be automatic. Very naive with my energy and mental level that evening. So I drop it that night.
Later I started to play with MCL (Monte-Carlo Localization) and one crucial "detail" is how big is the drone position estimation error? My only reference is list of the collected images, so mosaic.py implementation would have place even in this tight schedule . This time it was fully manual: press arrow keys for shifting new image and q/a keys for image rotation. It worked. But it was a tedious work even for three samples. And it did not quite solve the original task — how big is the error?
So I slightly changed the interface and instead of list of images I am passing there CSV file containing filename, x, y, degAngle on every line. It also prints this on standard output when you do the correction, so you can use it next time.
Well, so far so good. The struggle came with exporting CSV from airrace_drone.py. Y-coordinate on images is upside-down, rotation is not in the center but in top left corner, camera is rotated -90 degrees to robot/drone pose … and probably much more, like image size dependent on height or tilt compensation.
The result is still not worth presentation — it is mainly for me, to write the notes down, so I will realize where is the mistake?
The images are added via cv2.add(img1,img2), so overlapping parts are brighter. The shift in the turn is probably due to scaling but at the moment I have no idea why the drone basically stops on the straight segment (white area at the bottom). Maybe I should manually correct the mosaic, so you get an idea how nice/useful it could be. BTW the absolute error in direction is very small (less then degree), so compass is great (if it works at all! — we should get replacement for Isabelle next week, ha ha ha).
So here is mosaic with shift correction only
mosaic with shift correction
mosaic with shift correction
Every image is single video I-frame, i.e. 1Hz and approximately 15seconds of flight. The black dash-line segments are 30cm long with 10cm spacing.
And here is the mosaic corrected with angle (13 degrees correction for the last image, i.e. 1 degree per frame):
Exercise for you — how fast was Isabelle flying?

March 24th, 2014 - MSER and battery (4days remaining)

Here are some results of today testing:
  1. MSER is superior to normal thresholding
  2. there is again problem like before with normal contours and negative area:
    Inverse MSER region
    Inverse MSER region
  3. we finally completed 10 loops (see video), but Heidi battery died after 6 minutes

March 25th, 2014 - Bermuda Triangle (3days remaining)

My Bermuda Triangle is the center area, where the lines are crossing, and drones are loosing their navigation skills. The wrongly detected strip has following properties:
  • it is white
  • it is on the image border
  • it is triangle
At first I want to attack the 3rd point, triangle. The idea was: OK, if it is rectangle than minimum fitting rectangle will have almost the same area while triangle will have only half of it. Does it sound reasonable?
I verified 6 minutes video that I still get identical results as in the real flight yesterday (sometimes these "stupid" tests are the ones most important) and it was OK. So I added area computation and print:
rect = cv2.minAreaRect(cnt)
area = cv2.contourArea(cnt)
print "AREA %.2f\t%d\t%d" % (area/float(rect[1][0]*rect[1][1]), area, rect[1][0]*rect[1][1])
Well, I expected percentage of covered area, but …
AREA 0.05       121     2482
AREA 1.60       4936    3084
AREA 1.26       5033    3983
AREA 0.35       253     731
AREA 1.93       1830    950
AREA 0.57       1366    2415
AREA 2.24       6819    3044
… nice, isn't it . So what the hell is that?!
It looks like it is just an array of points, unsorted. Now it makes sense to apply convex hull as in one of MSER examples, before the contour is drawn. So I am not looking for area, but all I need is len(cnt)
AREA 0.90       2242    2482
AREA 0.91       2806    3084
AREA 0.88       3514    3983
AREA 0.86       626     731
AREA 0.83       787     950
AREA 0.90       2178    2415
AREA 0.93       2845    3044
That looks much better .
Do you want to see numbers for yesterday image (video_rec_140324_195834.bin, frame 17)?
AREA 0.58       1438    2475
AREA 0.57       1843    3219
AREA 0.26       4041    15301
AREA 0.32       5284    16650
AREA 1.01       624     620
AREA 0.99       818     825
AREA 0.86       1586    1850
AREA 0.85       2104    2481
AREA 0.87       1320    1519
AREA 0.85       1673    1964
AREA 0.82       2269    2772
AREA 0.91       1602    1767
AREA 0.84       2361    2794
AREA 0.91       3016    3299
First two are probably detected triangles for different threshold levels. After that is a big crossing area and then reasonable strips. So 70% would be maybe enough. And here is the result:
And now it is time for repeated video processing with identity assert — will it fail? Yes, even the first very picture :-(. I had to add condition to skip first 10 frames (seconds) … during take-off there are many random objects. Then it went fine to frame 218 … hmm, but I do not see anything interesting there (*). Time to re-log whole video and compare text results.
OK, there are 17 different results, frames 5, 920, 1055, 1220, 1835, 2180, 2480, 2810, 3470, 3800, 4460, 5405, 5420, 6035, 6185, 6350, 6530 (divide it by 15 to get I-frame order). Boring? Who told you that this is going to be fun? Off topic — I can pass I-frame index to see/analyze only particular frame, but what should I do with these "times 15" numbers? Bad mathematician and lazy programmer end up with the following code:
testPaVEVideo( filename, onlyFrameNumber=int(eval(sys.argv[2])) )
The result looks OK, actually even better. It filters not only the bad triangles but also slightly inflated rectangles … so far so good. We will see during tonight tests. Here is the code diff.
(*) 218 was not frame number. It was number of updates between frame 905 and 920, so the first failing frame was in reality 920/15

March 26th, 2014 - Isabelle II (2days remaining)

Believe it or not, we have new drone now. It came yesterday as "hot swap" for Isabelle with broken compass. And she already flew several rounds … thanks Apple Store
The second most important progress is, that MSER now works also on Jakub's computer with Linux and OpenCV 2.4.2. The workaround is here (I am sure there is some faster/proper way to do it directly in NumPy, but there is not much time left for study/experiments).
The first test went as always: Heidi took-off, detected strip, started to turn, missed transition to straight segment and continued in the circle … damn. The second test went fine. This scenario is already repeated like 5(?) times … and the magic is … height. We added extra time to reach the 1.5m height and since that (knock, knock, knock) it works fine. Here is the next changeset.
There is left/right tilt compensation correction implemented now. Why? We had added strips filtering that they not only have to be spaced by 40cm and similar angle, but that Y-coordinate offset is smaller then 25cm. But then we have seen drone following line and on one image it is tilted to the right and on the second image (a second later) it is tilted to the left. The difference was only 6 degrees, but for 1.5meter height this means 15cm and that already matters.
Another "break through" (ha ha ha) came to mine mind on the bus on the way to CZU. We should not follow the line exactly but with an offset! That way we increase the chance to see transition line to circle. Here is the code.
OK, if you finished reading up to here … here is video from testing (not edited). You can see there whole team except me + one visitor.

Internal notes …

Now is time to review yesterday log files. There are 7 of them + 500MB from Jakub. First two logs are without offset, then for speed 0.4m/s, 0.6m/s with increasing step 0.05 per loop, 0.75m/s and 0.7m/s.
I am not able to replay meta_140325_192727.log … my notes are not very good … I see that was the moment when we had speed 0.4m/s, step 0.05 BUT there was constant angle correction independent on desiredSpeed. OK, now replay works without assert.
The last log ends nicely:
BATTERY LOW! 8
!DANGER! - video delay 1.097775
… and in 5 seconds it hit the column.
Now, refactoring … I promised to delete all unused code, sigh. It must be done. Now. MCL — gone. classifyPath — gone. PATH_UNKNOWN and PATH_CROSSING — gone. Diff looks now a bit scary. Hopefully three reference tests were enough for this refactoring. There is one more round of test tonight, so this was the last chance.

March 27th, 2014 - Sign mutation (1day remaining)

Are you also sometimes lazy to think about slightly more complicated mathematical formula that you just try it instead? And if it does not work, you randomly change signs? I would call it genetics programming (to complete the evolution you can add to this process also copy and paste). And sometimes it is not good idea…
I suppose that I had to use it for navigation along the right turn curve. It is like left turn, just the signs are opposite, right? Some mutants are quite healthy, and this one was for a relatively long time! Have a look at the video again and closely watch the right turns (1:01-1:16, 1:41-1:57). It is easier to hear it than to see it. What is the problem? We expected some magnetic anomalies, but the truth is that the signs were not quite optimal. I am not sure if I should be ashamed or happy that we found the bug . Simply for right/left angle correction was used the wrong side of the circle (imagine circle and its sides, … I mean there is a place where your error is close to zero and there is also the opposite place where you are close to math.pi, -math.pi singularity. It actually works (as you can see it on the video), except that you steer with maximum left, right, left, right …
OK, that was good one. There are two more issues, which I would call nightmares. I know about them, but I am not quite sure what to do about them:
  1. losing WiFi connection
  2. video delay
The first one looks like this:
Traceback (most recent call last):
  File "M:\git\heidi\airrace_drone.py", line 237, in 
    launcher.launch( sys.argv, AirRaceDrone, competeAirRace )
  File "M:\git\heidi\launcher.py", line 82, in launch
    task( robotFactory( replayLog=replayLog, metaLog=metaLog, console=console ))
  File "M:\git\heidi\airrace_drone.py", line 215, in competeAirRace
    drone.moveXYZA( sx, sy, sz, sa )
  File "M:\git\heidi\ardrone2.py", line 633, in moveXYZA
    self.movePCMD( -vy, -vx, vz, -va )
  File "M:\git\heidi\ardrone2.py", line 539, in movePCMD
    self.update("AT*PCMD=%i,1,"+ "%d,%d,%d,%d"%(f2i(leftRight),f2i(frontBack),f2
i(upDown),f2i(turn)) + "\r")
  File "M:\git\heidi\airrace_drone.py", line 79, in update
    ARDrone2.update( self, cmd )
  File "M:\git\heidi\ardrone2.py", line 258, in update
    Pdata = self.io.update( cmd % self.lastSeq )
rocess Process-1:
Traceback (most recent call last):
  File "C:\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap
  File "M:\git\heidi\ardrone2.py", line 113, in update
Process Process-2:
Traceback (most recent call last):
  File "C:\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.command.sendto(cmd, (HOST, COMMAND_PORT))
socket.error: [Errno 10065] A socket operation was attempted to an unreachable
host
    self.run()
  File "C:\Python27\lib\multiprocessing\process.py", line 114, in run
    self.run()
  File "C:\Python27\lib\multiprocessing\process.py", line 114, in run
     self._target(*self._args, * *self._kwargs)
   self._target(*self._args, * *self._kwargs)
  File "M:\git\heidi\video.py", line 30, in logVideoStream
     data = s.recv(10240)
e File "M:\git\heidi\video.py", line 30, in logVideoStream
rror: [Errno 10054] An existing connection was forcibly closed by the remote host
    data = s.recv(10240)
error: [Errno 10054] An existing connection was forcibly closed by the remote host
I wanted to somehow clean the dump for you, and I did not realize that it is actually two crashing processes with interlaced error output (*). One process is handling navdata and sending flight commands while the other is video processing. Both crashed on socket operation, where An existing connection was forcibly closed by the remote host.
This happens after several minutes flight from my Win7 notebook. The drone stops and is hovering at last position. Network is temporarily unavailable (in the worst case the notebook can switch to another WiFi access point, if you are not cautious enough). So what? Take a break and restart the whole program in 5 seconds? What about details like take-off, trim gyros or that first segment is not PATH_TURN_LEFT any more?
The second nightmare looks like this
NAVI-ON
!DANGER! - video delay 158.220822
FRAME 20 [-1.6 1.2] TRANS L -> I
I True 5
!DANGER! - video delay 158.392327
FRAME 22 [-1.6 1.2] I True 4
!DANGER! - video delay 158.393941
FRAME 24 [-1.6 1.2] I True 3
!DANGER! - video delay 158.490557
FRAME 26 [-1.6 1.2] SKIPPED2
I True 2
!DANGER! - video delay 158.642409
And here is the story how it happened: I asked Jakub to read configuration files form Heidi and Isabelle II, because for some unknown reasons Heidi is still faster. Maybe more experienced? My old hacked program crashed after the read, and it almost did not matter … except the recorded video was not transfered, so the drone tried to do that in the next run.
You can see similar reports from my tests last year. I.e. it is expected, but we should be careful about it. It is quite hard to navigate from more than 2 minutes old video .
(*) there are actually three failing processes: navdata, video and recorded video

March 28th, 2014 - WiFi Sentinel (0days remaining)

I cannot sleep again (the clock says 2:47am now) :-(. Yes, the nightmare(s) again. Yesterday I took „drone day off”, but Jakub did some more tests with WiFi failure. We had several iterations with wifisentinel.py, which is a script for securing WiFi connection. You write program arguments as usual except you put this wrapper at first place, for example:
python wifisentinel.py python airrace_drone.py test
The sentinel first verifies if you are connected via one of three allowed IP addresses. If yes then it starts the drone code otherwise it waits 3 seconds. When the drone code crashes, for example due to Unreachable host problem, it again waits until connection is reestablished and runs the code again and again, and again …
Probably the funniest bug was this one, when instead of drone flying code it was calling self again. Then there were some issues with shell=True … you can always put the shell name in front, so now it is shell=False when using the subprocess.call(). Finally the code for Python ipconfig is extremely ugly and fits only my and Jakub's operating system — I have no idea why he choose Czech Linux , where you get strange console outputs like:
…
inet adr:10.110.13.102  Všesměr:10.110.15.255 Maska:255.255.248.0
…
AKTIVOVÁNO VŠESMĚROVÉ_VYSÍLÁNÍ BĚŽÍ MULTICAST  MTU:1500 Metrika:1
…
If you know how to do this better, both on Linux and Windows without need of installing extra package, let me know. I will change it (not today and not tomorrow).
The test was following:
  1. start drone program
  2. turn WiFi off during flight
  3. turn WiFi on again (after couple seconds)
The old code would crash immediately on the step 2 and drone would be hovering in place. Surprisingly you get the same bad result also with previous revision of the new code . Yeah, testing is important. In that case the navdata communication channel crashed, but both video processes remaining active! At least I learned that there is daemon value for multiprocess too. And then it went fine. Jakub repeated several WiFi failures and Isabelle always recovered.
So what is the problem? There is still video delay. Say, the drone recovers within 15 seconds. During that period it is recording area below it. It is slightly moving, and it is still recording. Then it starts to fly and the log file looks like:
NAVI-ON
USING VIRTUAL LEFT TURN CIRCLE!
!DANGER! - video delay 20.909436
FRAME 71 [-1.7 0.2] TRANS L -> I
I True 4
!DANGER! - video delay 20.406242
FRAME 72 [-1.7 0.2] I True 3
!DANGER! - video delay 19.874204
FRAME 73 [-1.7 0.2] I True 2
!DANGER! - video delay 19.346308
FRAME 74 [-1.7 0.2] I True 1
!DANGER! - video delay 18.818665
FRAME 75 [-1.7 0.2] I True 0
!DANGER! - video delay 18.260648
i.e. the delay is getting smaller and smaller as the drone is downloading the old video from period without connection. But it also tries to fly based on these video frames!!! Believe it or not, it could be OK for couple seconds until you have to switch from line to circle navigation. It is using only the old video as reference but it has no knowledge how far it is on the 8-loop.
So now the dilemma: change otherwise tested code or not? Was is just luck that it worked fine yesterday or the drone should rather stop and wait another couple seconds? It seems that it is capable to download 2s of video within 1s, so to decrease the video delay by second on every image … no, stupid me. There are different times. The time difference cannot be more than 1 second. If the processing would take 0.001s the images are recorded at fixed times, every second, so it would decrease it by 0.999s, but it would took 0.001s. So maybe it is not that bad.
Let's review this example FRAME 71: 20.9s delay and FRAME 103: 1.6s delay. So 32 frames and 19.3 seconds, 0.6s per frame, so approximately 0.4s it needs for processing on Jakub's older notebook. It also means that it took 0.4*32=12.8s in real time, at slow speed (0.4m/s) this means more than 5m! OK, I will change the code NOW!!
I am looking at the logs in detail and it is scary, see this picture:
The default configuration is to look forward, and one of the first steps is to switch to down pointing camera. But it will navigate based on detected strips for a while … I am not going to change this part anyway. So you can tell even from images that the drone was restarted.
Code for The Last Prague Test is ready and I can go sleep again … it is 4:46am, so I can at least try .
p.s. yesterday came very important info from Karim (Robot Challenge organization). The schedule is updated and Air Race --- Each robot has to pass the homologation in order to be allowed to participate in the competition. Please don't wait till the last minute during the homologation time. In the competition, each robot has multiple attempts - the best one counts. Please don't wait till the last minute with your runs. I am waiting for confirmation but in this case we could test STABLE (current) version and RISKY (to be prepared during drive to Vienna) version . And we have two drones to crash! Now I have reason why to take spare indoor protective hull . Držte nám palce

March 29th, 2014 - Falling Down (11 rounds)

It is over now. Do you know the movie Falling Down? Somehow it is the first association I have now . But let's start from the beginning …
… I just realized why the robot died, as always. Originally a good idea killed that …
Yesterday evening we have studied the last Friday tests, including speed profiles. Here is the graph for max desired speed set to 0.8m/s, dropping to 0.5m/s when no strip was detected and slowing down to 0.4m/s for transitions between turns and straight lines:
The robot at the end failed due to large shadows wrongly detected as strip. So we added also maxWidth limit. That was all we changed … we were too afraid and tired to do any other change.
This morning we did some more changes. First of all we set the radius to 1.25m, because it was smaller in Vienna than in Prague. We did first test and there were problems with video delay, crossing, and Isabelle height. So we decided to increase desiredHeight to 1.7m, limited max allowed video delay to 2s and implemented crossing detection. The crossing on Robot Challenge 2014 looked like this:
And that was it. That was the final code.
How it went I will leave for tomorrow morning … the hint, why robot crashed after completed 11 loops is already mentioned … but maybe not as Jakub just showed me the latest images.
p.s. Air Race 2014 - Isabelle CULS video is finally uploaded … again it is without censure and editors cuts, so sorry if you will hear some rude words …

March 31st, 2014 - 3rd place


April 1st, 2014 - Altitude, Pressure and Temperature

Yesterday I tried to investigate „the falling down” problem, which caused problems with navigation and Isabelle crash after 11th loop was completed. Here are some data for NAVDATA_ALTITUDE_TAG:
It is a little bit hard to see but the sonar (green) and vision (red) graphs are almost identical. These are raw values in millimeters, if I remember correctly. The deadly reference has yellow color. Finally you can see also blue graph of maximal rectangle size (0 for no rectangle detected) and you can see that it was slowly growing but did not reach limit 200. So Jakub was right and we did not broke it with maxWidth filter set to 200.
So why is the yellow reference so wrong? Well, first of all to use altitude (drone.coord[2]) was not good idea. It is integrated over time also from PWM and pressure sensor. It has to compensate for terrain changes and detected obstacles (I know that the floor was flat, but Isabelle maybe did not know it). There was also small breeze in the hall, which could cause the first bigger drop (??).
The temperature was slowly increasing from 291 to 424 — no idea about resolution and units, so I would guess that it is 1/10th of degree. It was maybe getting hot due to heavier battery??
Anyway — temperature and pressure data are parsed now so you can use it in ARDrone2 class if you want.
I just replay some older test (meta_140326_201259.log) and the temperature there was 515, dropping to 510 and then rising to 525 … so I am not sure if it means 52 degrees of Celsius?! Could it be that if the drone is cold the estimation is not very reliable and we usually tested several times so the drone warmed up?? I am looking now at test meta_140325_180914.log and there is the reference estimate lower than readings from sonar and vision … so I would conclude, for Air Race do not use altitude, but only sonar and/or vision distance from the ground estimate.
p.s. if there will be next time we will have to look forward — the video footage would be much more interesting than to look down

April 11th, 2014 - Conclusion

Originally, I wanted to write more info about logs, wifi failures, competitors and plans for Air Race 2015. There is no time left and priorities sifted again … so I would like to thank the CZU team (mainly Jakub who did many many tests for several weeks — 9GB of log files).
Did we beat the last year best semi-autonomous score? No. We did not. But the winning team Sirin, with navigation strategy only above the cross, could have done that. They had times around 20s per loop, so 30 loops per 10 minutes. But they had also some problems … as everybody.
And what about Air Race 2015? Two Russian teams mentioned that they plan to build/use new drone, in particular Alex was mentioning PX4 platform. There is also a plan to modify the rules — we will see. I am definitely voting for multiple 10 minutes attempts . That way we will have chance to test stable and risky version and speed up if other competitors will also speed up.
p.s. if you would like to compete with your drone in a maze, here is another contest organized by Russian company … but there is age limit set to 18 … never mind … but it is the lower limit