czech english

SubT Virtual Cave Circuit

Robotika Team

Cave is the last of three training Circuits before the finals in 2021 (if everything goes well). The environment is interesting of variable 3D terrain and combination of huge and narrow openings. Update: 17/11/2020 — Final Results!


Content


24th March 2020 — Robotika scores the first point in Simple Cave 1

Since we returned from successful mission in USA (see 3rd place in Urban Circuit, 2nd among DARPA non-sponsored teams) thanks to The Virus is is much easier to continue preparation for Cave Circuit in Virtual world rather than in System Track.
There is new release 2020-03-23 containing 3 Simple Cave worlds for testing. Yeah, I could not resist to see it, and surely it was too early (#360). But the simulation were re-triggered and we could see what the left and right robot has seen during 5 minutes of simulated exploration:
If you are wondering what the robots are doing … well it is version ver59 used in Urban Circuit where the robots starts on optimized trajectory to „enter the gate”, i.e. smarter way of turn and go straight for distance defined by unique robot origin position. This is a bit which we will surely fix ASAP.
But enjoy the terrain and huge and narrow spaces. Yes, this time we want also to fly, communicate among robots and map 3D properly … and win! … so there is not much time to waste.
BTW we scored 1 point and Robotika is leading all Simple Cave X boards — well, two of them with zeros. But see what robots have seen:
The nearest deadline is 31st March 2020 (7 days) to submit new robot models. Here is preview of Freyja and Kloubák K2 is work in progress:

31st March 2020 — Robotika in the 'Video Friday' feature on the IEEE Spectrum website

Today I received mail from Kara: I thought you might like to see an interesting article featuring Robotika in the ‘Video Friday’ feature on the IEEE Spectrum website.
There are plenty of robotic videos, including one from SubT Urban System Cerberus — you can see their flying ball from Flyability autonomously navigating downstairs in Alpha Course.
Yeah, you can find there also video of our first attempt in Virtual Cave Circuit mentioned above.

14th April 2020 — Flying in the Cave and return home

The last week PavelJ finished the first experiment with virtual drone X3. It uses the same map as MOBoS back in Olympia and visual odometry instead of wheel odometry.
Maybe even more interesting it is under the hood, i.e. lidarview video. Note, that the drone has only RGBD camera, so it has to turn left in order to confirm that there is no other exit.
p.s. the schedule and release of qualification world is still unknown (due to the Virus), but the original plan was early April (see video).
p.s.2 another video: SubT Robotika - First longer flight … i.e. 5 minutes of exploration including detection of backpack.

21st April 2020 — New Cave Artifacts: Rope and Helmet

Today should be released Cave Qualification World which will contain also two new (still unofficial) artifacts:
The release 2020-04-21 is already listed but docker images are not build yet …
BTW I did not realize that SubT Virtual moved not only to GitHub, but also changed from Mercurial to Git! At least some bits are easier now
OT Problems with Cloudsim #339 — Malcom Stagg/Sodium 24 probably found the root cause related to Tunnel and Urban Virtual simulator problems:
CloudSim uses WeaveNet to provide networking and network isolation to the containers. This works as expected for unicast packets, but WeaveNet NPC has some counter-intuitive behavior that it does not block multicast packets from being sent to all hosts (even other competitors), as detailed in this issue: weaveworks/weave#3272. Ignition Transport uses multicast UDP packets for topic advertisements among other things, so all topic advertisements from any running robot were unexpectedly being sent to all hosts, where most of the topics would get filtered out on the receiving end. I was able to confirm this was the case by running two simulations simultaneously and logging all the topics before they were filtered out, and I found the isolation between those two simulations was indeed broken.
I also wrote some test code: https://github.com/sodium24/cloudsim_net_test which confirmed that when enough topic advertisements are received (on the order of several thousand), even though most are eventually filtered out, the infrastructure containers stop working correctly, and topic data, especially tf data, begins to become lost. This can be seen in the ign_saturation test cases. I was able to reproduce the issue of my robots either not moving or spinning in circles outside the entrance on my local computer. Anyone feel free to use/update this test code btw. There are also some test cases there which might be helpful to ensure things are working alright despite some common network impairments.
Based on this strong evidence, I believe the Urban Circuit issues were caused by the excessive rate of multicast Ignition Transport packets unexpectedly being delivered to all hosts, resulting from the WeaveNet issue combined with the large number of simulations and robots running at the same time.

22nd April 2020 — Robotika is the first qualified team!

… in less than 24 hours!
It is 23 hours after official DARPA SubT Challenge Cave Circuit Announcements (private access) … i.e. officially releasing qualification world for Cave Circuit, Cave Artifacts Specification and new Qualification Guide. At first I overlooked but other team members highlighted the point 4:
All Virtual teams will need to qualify using the Cave Circuit qualification scenario, i.e., no qualification waivers will be issued for Cave Circuit.
So this time you should see all teams on Cave Qualification Leaderboard before they can compete in Virtual Cave Circuit. The good news is that all you need is score one point. This means that there could be more competitors now and it reminds me old time Eurobot 2001 - Space Odyssey.
Another important note for both System and Virtual is Schedule Shift of the Cave Circuit: First, due to the circumstances and restrictions surrounding COVID-19, Cave Circuit will be no earlier than October 2020. If I count right it means half a year.

9th May 2020 — mdnet0.64.64.13.4.relu.pth - [Msg] Total score: 3

Murphy is really a Joker!
My friends did quite good job on on the new Robotika docker image, in particular it has base image with Torch and TensorFlow for CudaDNN experiments now, and when I ran that on Cave Qualification World on CloudSim I could see (there is a new feature that you can see some simulation outputs online):
…
[Msg] SimTime[646 392000000] OnNewArtifact Msg=type: 13
pose {
  position {
    x: 63.76
    y: -8.27
    z: 2.03
  }
}

[Msg]   [Total]: 1
[Msg] Total score: 3
[Dbg] [VisibilityRfModel.cc:80] Range: 16.0115, Exp: 2.5, TX: 20, RX: -50.1108
[Msg] SimTime[646 392000000] OnNewArtifact Msg=pose {
  position {
    x: 92.75
    y: -0.56
    z: 3.05
  }
}

[Msg]   [Total]: 0
[Msg] Total score: 3
…
I was excited because then we could easily lead again the Leaderboard, but … Murphy works even on weekends so the simulation is somehow stuck now, sigh.
So if you check it now you will see this picture with leading Kankanwadi.
Note, that all other teams already participated in Tunnel and Urban circuits. Missing are only two leaders from urban circuit: MTRI and Coordinated Robotics, and also CYNET-ai is not there yet, but they probably work now on NASA Space Robotics Challenge Phase 2.
p.s. small update …
p.s.2


30th July 2020 — Cave Circuit Rules and Schedule Released

Today DARPA announced the schedule for Virtual Cave Circuit on the community forum:

Competition Rules for the Cave Circuit are now available on the Resources page, along with an update to the Qualification Guide.
Some major updates:
  • The SubT Credit cost of all robots have been reduced by half, which will allow for larger robot teams including the recently released robot models.
  • Teams are encouraged to explore increased robot team diversity, as robot teams may now only have a maximum of five models of a given type.
  • Teams can now submit multiple Cave Circuit submissions within the competition window – only the last submission will be scored.
We are also announcing the schedule for the Cave Circuit Virtual Competition:
  • 02 AUG — Robot Model Submission Deadline
  • 15 SEP — Qualification Deadline
  • 08 OCT — Competition Submission Opens
  • 15 OCT — Competition Submission Deadline
  • 17 NOV — Showcase and Awards Ceremony
We look forward to seeing your results on the leaderboard as you practice!

This means that there are only 77 days left till final submission on 2020-10-15. And it could be interesting to have say 10 robots … cool!
p.s. I noted that the last blog update here was more than two months ago! There was not much of visible progress on the surface, but … in meantime there were 4(?) CloudSim releases with new platforms including our robots Freja and Kloubák K2. BTW the Kloubák K3 platform is on title page of Forbes magazine … and yes, there is also info about new virtual platforms …

4th August 2020 — Virtual Live!

Today I have one good and several bad news …
The good news is that Virtual Circuit probably is not going to be black box anymore! I was starting simulation test for communication between robots and the teambase … and I found PLAY button, which you can press while simulation is running. And you can see this:
Note, that this is not pure video but full online 3D world browser — basically your Gazebo window. I am so used to the command line SSH interface that this is like „new world” for me.
Cool! I like it … no more secrets!
And now the bad news — we are no longer leading the Cave Qualification table:
You can tell, that Kevin (Coordinated Robotics) switched from System to Virtual. And that is also the 2nd bad news — System teams (including DARPA sponsored teams) are encouraged to participate in Virtual, and soon we may see also the other Czech team (CTU-CRAS), which is currently submitting model of their threaded robot (see PR #520) … so the fight will be hard, but it could be interesting to watch.

2nd September 2020 — Freyja in Practice Worlds

Today I had a lot of fun watching multiple simulations of robot Freyja in the Virtual World on CloudSim. I some cases it would be nice to terminate the simulation (see #361) when the robot is upside-down, but as long as there are other machines available it is OK.
The situation will be much worse when all qualified teams (at the moment 13) will try to push and test last moment changes.
Robotika is still on the first page, but it is no longer leading the board in the Cave Qualification World:
Yesterday was the CloudSim code freeze for qualification with the last release 2020-09-01. Luckily our PR fixing Freyja lights was accepted but the docker images are probably still not uploaded:
Just few minutes ago Angela wrote on community forum:
The SubT Virtual Testbed repository is under a code freeze until the qualification deadline (15 SEP). Please make sure to update your Catkin install and Docker images so your local development environment matches the resources on Cloudsim.
Maybe one more detail to mention — why test in Practice Worlds? I asked Viktor and he confirmed that:
Question: Is there a plan (similar to Urban Circuit) to use one of the Practice Worlds also as a Competition World (with differently distributed artifacts)?
Answer: Yes, one of the three Cave Circuit Practice Worlds will be included as a Cave Circuit Competition World with artifacts relocated. You are encouraged to use the Practice Worlds on Cloudsim to prepare your team!

16th September 2020 — 17 Qualified Teams (29 days)

This is the screenshot taken today from Cave Qualification Leaderboard. Today is the deadline for qualification where the team has to get at least one point so qualify. The list is not final/official because the teams have to fill also some registration forms etc. I guess that by the end of week DARPA will announce the final official list.
Note, that I suppose that OR_Hello_World is a test from organisers.

26th September 2020 — Score 3-100-141 (19 days)

Well, 10 days is gone and 19 days is remaining to the Virtual Cave Circuit END. There is no official list of qualified teams so they may leave it after the submission deadline. So let's assume 17 competitors …
Today we finally made a breakthrough — we scored 1 point in one of the Cave Practice Worlds. It was very demoralizing so now we could celebrate, almost. … but do not ask what was the main problem! I am ashamed to even think about it. We are working with various types of robots and one of them had still old artifact detector in the configuration. No wonder it did not work very well. It even did not know about new cave artifacts. Fixed and scored in P1-Practice Cave (1 point) and S2-Cave Simple (4 points).
What is new? The robots now communicate, teambase collects positions of all robots and some of them even fly!

3rd October 2020 — Pending with no errors 3-101-642 (12 days)

A week ago I wanted to post an update about breadcrumbs (ver75) and some interesting exploration details, but there was always something more urgent … until now: Pending With no errors is what we see on CloudSim. The number of parallel simulations per team was reduced from 10 to 3, but it was obviously not enough. We are already waiting for 11 hours to start „morning” simulation. So the time when we are no longer use CloudSim is already here, sigh.
Note, that there is now also the official list of 17 qualified teams:
The following teams have qualified for the Subterranean Challenge Cave
Circuit Virtual Competition:
    AAUNO
        KANKANWADI PTY LTD, Australia
    Andersons
        MRSL Real-time Systems Lab
    BARCS: Bayesian Adaptive Robot Control System (DARPA-funded)
        Michigan Technological University/Michigan Tech Research Institute
    COLLEMBOLA: Communication Optimized, Low Latency Exploration, Map-Building and
                Object Localization Autonomy (DARPA-funded)
        Scientific Systems Company, Inc.
    Coordinated Robotics
        Coordinated Robotics
    CTU-CRAS-NORLAB: Czech Technical University - Center for Robotics and
                     Autonomous Systems - Northern Robotics Laboratory
        Czech Technical University, Czech Republic
        Université Laval, Canada
    CYNET.ai
        Jersey Media Network Corp.
    Dynamo
        Hilario Tomé, Spain
    Flying Fitches
        Sophisticated Engineering UG, Germany
    Irgum
        András Polgár, Hungary
    Lizard & Dragonfly Robotics
        Distek Integration, Inc.
        Zone24x7 (Pvt) Ltd.
    Map Quickly
        Steven Gray
    MARBLE: Multi-agent Autonomy with Radar-Based Localization for Exploration
        University of Colorado, Boulder
    NUS SEDS: National University of Singapore Students for Exploration and Development
              of Space National University of Singapore, Singapore
    Robotika
        Robotika International, Czech Republic and United States
        Robotika.cz, Czech Republic
        Czech University of Life Science, Czech Republic
        Centre for Field Robotics, Czech Republic
        Cogito Team, Switzerland
    Smart Tools
        Smart Tools Ltd.
    SODIUM-24 Robotics
        Malcolm Stagg
Cave Circuit self-funded Virtual teams may compete for the three top prizes, provided they finish in the top five overall ranking: $250,000; $150,000; or $100,000, respectively.

8th October 2020 — Submission window opened 4-213-654 (7 days)

There is daily progress, but no blog updates — I should at least log the max score changes and release notes! Because to be honest I do not remember when we finally got 4th point in qualification or 2nd point in Cave Practice 1. Was it today? Or yesterday??
So what is new? I see the website changed — now you do not need to scroll for half an hour to find your robot. There are new groups … yes, and the drone just crashed, so the ver82rc1 is not good one. I am glad that at least we have Teambase cutting the test time. Here I used T300 i.e. 5 minutes. A single drone is almost real time…
… but that was not the case yesterday, when we tested the full house:
I was curious (and expected problem) that all robots available for 1000 credits will not fit on the starting line. And they do not — see two lines of robots. So it makes sense to have first starting robots, or ground vehicles, in front and the rest in the 2nd line.
What else is new? Now all our robots use dual-classifier, which should help to detect more artifacts robustly. It looks interesting and we are collecting data for maybe yet another training cycle.
Now the drone crash interrupt … what I like on OSGAR is that in order to debug the problem I download the logfile, wrote python -m osgar.replay --module app logfile-name and in couple of seconds I see replayed control loop and crash with stack trace. So let's fix it and then I can continue writing these non-senses …
OK, fixed — or better to say, we will see soon. To be continued … (it is again midnight)

16th October 2020 — Robotika Solution Submitted

The game is over, or maybe saying the rocket has been launched would fit better … all we can do now is to wait. And it is probably the only moment when we can summarize what happened last couple of days/weeks/months?!
I will start from The END because that is something I remember quite clearly. The deadline was 15th of October 2020, but AoE (Anywhere of Earth). One would expect, based on our former many years experience, that we (Robotika) would submit the final solution early at the beginning of the window, i.e. a week ago. But in reality the very last version Ver92 was submitted 30 minutes before the end. It even did matter if end of the day is at noon?! Luckily it was not … so 2pm of CET (Prague).
Why? I can imagine Jirka saying why not? … there were couple important upgrades since Urban Circuit, in particular you could submit your solution several times and only the last one is taken into account for evaluation. We took advantage of this and submitted our solution two(?) days ago. We actually submitted it twice, just in case, to be sure that the first submission was canceled and the second was accepted (if this would be broken there would be still some time to fix it on the DARPA/OSRF side). It worked.
The second major change, not announced anywhere, was that CloudSim mostly worked! Even in the very last week. That was surprise (good one) for me, and today, we were still able to verify/test our latest image. This means that the submission was no longer a blind guess as the last time.
A long time ago I asked for manual stop of the simulation — if you find out that you no longer need it, or that there is a serious bug, let's spend the computer time on newer version and drop the current run. This became even more obvious with new visualization feature, when you can actually see the simulation online! If you see your robots up-side-down obviously unable to do anything useful then why not stop it? Also the number of parallel simulations per team was reduced to 3 so you may use the team resources for something else … and believe it or not two days before the end there was new stop button! And it worked at the end (the day after), great.
The fine tuning of parameters showed some unexpected behavior (thanks Cogito for validation and triggering alarm!) so this morning we reverted (or better to say balanced) some parameters.
The analysis of large runs went on and it turned out that the wheeled Freyja robots detected artifacts but the near by flying drones did not report it to the base station?! Very strange. Again the excuse is simple: the drones have much higher chance to handle complexity of 3D caves so the focus was on them. And there was a missing link in Freyja configuration … and as it turned out a bit later today, there was one more missing line feeding the reporter module with simulation time. The robots are supposed to talk a lot and maybe somebody will hear them. But the trigger is the time … and this bit was missing for Freyja robot. Fixed.
That was today. I have no idea what was yesterday — I would have to go through GitHub history to check. Yeah, maybe crazy Zbyněk's idea that on the way back the drones should not navigate to desired Z-coordinate immediately but use linear interpolation = instead of setting desired_altitude the main controller sends desired_z_velocity. Is this change I would like to integrate so late? Definitely not, except it behaved much more smoothly, drones moved faster and there were also less collisions … your choice. BTW that was one of many „adrenaline moments” when the drone dive into the large holes in the ground and after exploring it returns sometimes straight up! Some videos would be nice … maybe CloudSim will support that too, one day. Then you would know what I am talking about.
Another quest — if there are two robots following the left wall, will they both end on the same place? There are many inputs like sensor noise, acceptable width of holes from different angles, etc, but I still was surprised that on T-junction one robot can correctly turn left, while the second robot turns right … and the opening is large enough on both sides. How is this possible? This can be quiz for you — if you see it you may send us note.
What else? Communication among robots, as independent agents. Easy right? Work for half an hour, 1hour max. Such a tasks are the worst ones, if you underestimate the complexity then you are depressed (I mean much more than usually) that it is not done after a day or even two and you discover new situations which do not have any solution, and the algorithm must simply guess, sigh. Funny. The task was easy — spread knowledge about detected artifacts with their positions and responses from all robots. Distributed system. Race conditions. Communication failures. ROS … actually this part (CommsBroker) is not ROS yet, but it will probably change to the finals.
So that was two days ago? I am not really sure.
Artifacts — what a nightmare number II. I am glad Cogito and Jakub took over. It worked fine and then we dropped the first breadcrumbs to improve communication with the base station. Large orange objects … something unseen by the detectors/classifiers. Cool, another training cycle for DNNs … now is it better/worse … and the time is ticking.
The rest is amnesia unless I wrote it here in the blog. OK, too much text, no pictures. But at least some update. I may pick some screenshots from hundreds of e-mails we exchanged …
p.s. I completely forgot localization and invalid positions after collisions, diverged position of robot waiting several minutes on start … a long story …

27th October 2020 — Bugs, modern art and config set of future winners

Today was not great. Zbyněk discovered first serious bug for virtual robot Freyja which could lower our chance to reach 4th place (probably the best we can think of according to the results in cave practice worlds). The bug is related to mix of different cameras for artifact detection … I overlooked that wide lenses camera is used for visual odometry and artifact detection while depth is taken from narrow RGBD camera. I think that it is obvious from these two pictures:
So if you find an artifact on color image and directly measure distance on depth image … it will be wrong up to several meters. Sigh.
Maybe related funny note, that in Cave Practice World 2 Freyja (exploration time set to 1500s) detected helmet, twice, on the way back to staging area even within 5 meters radius … but no point! The issue was that helmet_5 did not exist in some configuration files (see #683) … so currently leading Coordinated Robotics with 3 points had to find some other/harder artifact!
This leads me to the secrets of two winning teams: MTRI and Coordinated Robotics. Both agreed to share the information about their submitted robot configurations:
  • Coordinated Robotics — 4x Marble HD2, 5x X3 and a team base
  • MRTI — 5 SSCI X4 config 1 and 2 X1's with config 7 (breadcrumbs)
I tried to contact/find also new Dynamo team (see DARPA teams), and it is Hilario Tomé from Spain … hey, DARPA updated the website and now there is a picture and description:
We come from a humanoid robotics background, an exciting and complex field, where real-world applications still seem too far into the feature. The DARPA SubT Challenge seemed a very interesting challenge with comparable complexity but with direct applicability to a real-world problem.
Our solution relies on a team of quadcopters. Both humanoids and quadcopters share a basic characteristic, which is that they are underactuated; they have more degrees of freedom than the number of control inputs, and thus they share similar control techniques.
Shifting from the very expensive humanoids robots, the goal is to use the lowest cost sensors available, and to leverage the complexity of the problem to the software. This can only be achieved using sensor fusion techniques, optimization, and machine learning.
… OK, so now I learned something … and Tom is surely from PAL Robotics.
Note, that the configurations will not be really secret — the hint I got from CTU-CRAS-NORLAB team (yes, now our competitor also in Virtual world), that from each simulation there will be video clip generated by tool in PR #633.
BTW CTU-CRAS-NORLAB is using Explorer X1 and drones.
OK, let's finish it with modern art which actually pushed me to write this post. Jirka/Cogito wrote: The agreement of three odometry sources, Kalman, and SLAM is not great. But maybe we could make living from generating modern art? … so what do you think?
Modern Art
Modern Art
p.s. just for my reference, current Robotika score is 4-413-654 and the top score is G-934-88A

1st November 2020 — Missing ROS sensor data (again!) and Urban Worlds

Nate wrote: You are free to utilize Cloudsim for practice. … and so we did. Last week we discussed if Robotika will continue to finals and under which conditions. Also we talked how finals may look like, how DARPA will combine all 3 types of world: Tunnel, Urban and Cave? The conclusion and our current guess is that we will submit only one docker solution (in Virtual) and it will be tested against all worlds. So why to wait for results and not to go ahead to check where are we now?
After return from Olympia, where the results were announced, I really could not look at the Urban competition worlds. Do you remember this line?
Robotika results at Urban
Robotika results at Urban
I think that it was aggregated score over 3 simulation runs in each world. I really hated that result: you can hear it in my voice. Sigh. To the point! Well it is BACK AGAIN! In the Urban World 3 the drone A was not receiving bottom laser data! (#667, 670) Yes, this is fatal and WE CANNOT DO ANYTHING ABOUT IT … you see it makes me angry again. Well, it is a game, lottery, … we will see. (Actually I was very surprised that after return from USA Zbyněk and Jirka were willing to continue in virtual … and I am glad that they fulfill their promises …)
Back to Urban Worlds. It is actually funny to watch CloudSim with two drones (we did not use drones at that time), i.e. we run A600L and B10W600R drones (600 seconds = 10 minutes, exploring Left/Right side and returning home, battery life time is 20 minutes) on all 8 worlds and see what they can do. The current score is [1, 0, 1, 5, 2, 2, 1, 3], and we have seen already some bugs:
  • drones configuration does not have link for Gas Detector, so even it is correctly detected it is not reported
  • the staging area for Cave is much larger and because reports are filtered out if they are in staging area we did not report backpack in Urban 1
  • there is no detector for Vent
  • unexpected teambase location (#690)
So wish us luck, we will need it!
p.s. the announcement will be on 17th November 2020, i.e. in 16 days

14th November 2020 — Live in 3 days!


Follow along with the DARPA Subterranean (SubT) Challenge Cave Circuit Virtual Competition on November 17. Watch as 16 teams from around the world use simulated robots to map, navigate, and search complex, realistic cave environments that feature natural terrain and dynamic rock falls.
p.s. mail from DARPA (16th of November):
SubT Challenge Teams,
I hope you are as excited as we are to see the Cave Circuit Virtual Competition Showcase and Awards Ceremony tomorrow. You can tune in to SubTv https://www.subtchallenge.com/SubTv.html to follow along as the 16 teams navigate the natural terrain and dynamic rock falls. The Showcase will start at 2pm on SubTv and will run approximately 2 hours following the teams' progress throughout 8 worlds, culminating with the Awards Ceremony where you can find out who won.
Sincerely,
The SubT Challenge Team

17th November 2020 — Final Results!


I highly recommend to watch the showcase --- it was 2 hours dramatic show till the very end!

Partners


Supporters