In parallel with System Track Urban Circuit starts also Virtual Track.
This time we will keep two separate threads/blogs although they are connected
— one team Robotika. While System Track is still "big unknown" for
Virtual Track is already qualification world prepared, so let's have a look …
Update: 5/3/2020 —
Virtual Urban Worlds
Qualification for the Virtual Urban Circuit will require submitting a docker
solution against the Urban Circuit Qualification scenario through the Tech Repo
(https://subtchallenge.world/compete). In order to qualify, a team will
need to submit a docker solution that is able to successfully report at least
five of the artifacts within one hour of simulation time. Teams may submit
against the Urban Qualification scenario as many times as they choose until the
deadline for the Virtual Urban Circuit qualification, January 3, 2020
Tunnel Circuit Virtual Teams,
We hope your fall is going well. This is a notice that your team has received
a Team Qualification waiver for the Urban Circuit of the Virtual Competition
based on your successful participation at the Tunnel Circuit event.
However, please stay tuned for the upcoming release of an updated
Qualification Guide. We do plan to release additional guidance related to team
registration and additional requirements.
We look forward to your continued participation in the SubT Challenge!
Any details? Well, there are some … for example we are not sure when it is
guaranteed that the ROS bag recorder is ready
Also the version on AWS is more than 2 weeks old
No, the version numbering is not available (yet?)
so the only hint is output from simulation:
There is one close deadline: Qualification ends on Jan 3, 2020, i.e. in 9
days. If you check the scoreboardUrban Qualification then you can tell that new teams will have a hard time!
Robotika is „leading” with zero points at the moment and you can see
two new teams TunnelVision and Map Quickly. The team OSRF SubT
RedTeam I believe is just testing team of the organizers and it will later
disappear. If you want to qualify you have to score 5 points, and in particular
you have to be able to navigate on the railway to next stations where are
artifacts hidden. Definitely not easy task. The old teams from Virtual
Tunnel Circuit have qualification waiver so that is the reason they do not
worry much … (yeah, including robotika, but … if you are not able to
qualify how you want to win?!).
The new logging system seems to be working fine, so I was able to download logs
from two robots A10F300L and B50F300R,
200670743174.dkr.ecr.us-east-1.amazonaws.com/subt/robotika:ver36. BTW I
like the new info feature
The results were not great. I am not able to extract log for robot A at the
moment, but that could be an issue on my side (work in progress). And what
about B robot? See some pictures from RGBD camera with 640x360 resolution (BTW
that was another „lost case”, because this sensor should simulate
but Resistance is useless,
In short it is not good for new teams: No team reached limit 5 points! There
was a request to extend the deadline or lower the threshold on the
forum (restricted access) and the DARPA answer was:
The Virtual Urban Circuit qualification deadline and threshold will remain as
outlined in the SubT Qualification Guide. We understand the challenges of
autonomous navigation and artifact detection in these environments are
difficult and require significant development time. You are encouraged to
continue developing and submitting your solution to the Urban Qualification
world and note there are more opportunities to compete in the Cave Circuit and
the Final Event.
I agree that it is very hard to get some points in Urban Qualification
world — see SubT Virtual Urban Qualification -
1st point/video. The point was collected by big wheeled platform X1 with QVGA
camera (320x240) and 5 meters range lidar, the weakest sensor configuration.
The motivation for X1 was large wheels so that it can overcome the bolts
holding the rails. Note bad orientation on the rail, probably caused by the
robot tilt (I hope). Also note, that the robot had to go all the way to the
next station to be able to turn — it is not possible to do it safely on the
railway. Yes at the final stage, when the robot is supposed to navigate on its
own path, it failed too. The robot did not climb back to the platform.
Nevertheless robot reported the artifact and the wall to base station was not
too big so it scored 1 point!
The next attempts were with ROBOTIKA_X2_SENSOR_CONFIG_1 configuration
(robot X2 with RGBD camera 640x360). Here the robot has to enter the rails very
precisely to be able to explore it.
Well, we (robotika) are qualified thanks to waiver from Tunnel Circuit
but there is a lot of work needed to successfully score in 25 remaining days to final
8th January 2020 — Virtual Urban Circuit Qualification Results
OK, now is the qualification disasterofficial
Congratulations to the 8 teams who qualified for Virtual Urban Circuit! These
teams provided narratives and had received score waivers for successfully
submitting qualifying solutions to Tunnel Circuit. The teams competing in
Virtual Urban Circuit are:
We appreciate the new members of the SubT Challenge community who made
significant progress with their solutions to the Virtual Urban Qualification
world. We hope you will continue development to qualify for Virtual Cave
Circuit and the Final Event.
Also the list of teams is now
up-to-date (it seems to me that there are no new teams in both tracks).
25th January 2020 — Vent, Ver48, RTAP-Map and Urban submission window
OK, it is time for a short report and check where are we now? I would
start with review of new robotika releases:
Ver41 — navigate first part via Trace for more reliable gate entering
Ver42 — limit size of ROS bag to 2GB (simulation limit for user logfiles)
so that we do not lose the beginning of the recording, depth2scan (detect large changes
Ver44 — integrate Gas detection and reporting as artifact, use for app.pose2d
global (unified) coordinate system
Ver45 — integrate vertical_scan from depth camera
Ver46 — parameter tuning for depth2danger to avoid wooden pallet for example
Ver47 — integrate depth2dist much more dense RGBD camera filtering
Ver48 — implement and integrate down-drops detection
Here you can see collected map all all five robots testing Ver47 in Urban
Practice 3. You can notice several „details” like that some robots returned
and navigated around start area (it was robot A for couple of times), sometimes
the position is shifted (i.e. wrong) due to lost odometry readings on heap of
stones or a wooden pallet, some robots ended in infinite (or almost infinite)
cycles, and some lost position (probably stuck on some obstacle, or they ended
Note, that this overview picture is quite useful for analysis. Imagine 5
robots exploring the world for 1 hour. In total around 10GB of logfiles to be
reviewed for changes and errors. Believe me — it is quite time consuming task!
If you would like to try it use osgar.tools.log2map which now accepts also
multiple logfiles (the motivation to have unified global coordinate system) and
can also display positions of detected artifacts (white dots).
Below you can see some hard to handle obstacles with planar lidar only and
Vent which we have seen for the first time:
So what is the plan with RTAP-Map
(Real-Time Appearance-Based Mapping)? At the moment we are developing two
solutions in parallel, where one is based on
OSGAR only while the other should be
hybrid combining also ROS packages. The plan for the last few remaining days (5
to the final submission) is to use RTAB-Map to independently check robot
location and use it for recovery in the case of failure … and if I did not
stress it enough, the submission windows is opened now! … see
29th January 2020 — The Day Before Last Day
There is one day left to the end of submission window and it is quite crazy
again. Exhausted, sleepy, tired …
There are few bright moments like when ver51p1b scored 5 points, but there
are many dark times like that CloudSim does not work at all and robot is
not receiving data (b is actually repeated run, where the first totally
failed — robot hit the wall(no lidar data) or did not start (no data) or
was not able leave start area (no IMU data). Really crazy. See issue
… but it is different now and all 8 competitors are uploading their latest
solutions for test.
31st January 2020 — Robotika final submission
It is 5 hours to AoE deadline and Robotika final solution is submitted now.
Thanks God! It was as bad as last time and much worse … but we are getting
used to it, so DO NOT PANIC!
As final report I would say that we did not learn. Since Virtual Tunnel
Circuit we know, that closer to the submission deadline are, the harder it is
to test on AWS anything. And the same happened again. On the other hand the
development peak was definitely this last week, so what is the solution to this
riddle?! It started with
stops sending some topics (again) — shortly some robots did not get any
LIDAR scan data for example, or IMU. They did not start, or hit the wall
shortly after the start because they had no idea, there is a wall in front of
them. At that point it no longer had sense to trust AWS CloudSim.
But that was not the end. As the machines were more and more overloaded
Sophisticated Engineering reported
with Cloudsim. The simulations did not start, or failed (Terminated with
errors) or took hours (days?) and then failed.
Can this be worse? Yes, it can. The Unknown Error caused that competitors
were no longer able to submit anything. Not just testing solution but also
finals. This was the state since midnight till at least 2am, when I went to bed
to re-try now, in the morning. The deadline is AoE (Anywhere on Earth) so till
12 hours the day after in Central European Time. Nate wrote me: The server
was down for a brief period of time. Sorry about that. Everything is back to
normal. Please try again whenever you are ready..
There are five robotika robots with version Ver56. Robots A and B take the
risk and try to explore the whole world for 40 minutes. It is risky, because
there is only one hour for the competition, so they have to manage to return in
20 minutes if everything goes fine and they do not fall from elevator vertical
shaft, for example. Then robot C tries normal exploration and should be
able to return in time (again, if everything goes well). Robots D and E are
house keeping robots and after some shorter time they return home and check
if everything is OK. Yes and hopefully they report some artifacts too …
Ver56 what is it? First of all it is (again) version which was never
running on AWS. Just could not. Not even Ver55, which contains refactoring
with ROS interface! Yes, no kidding, 24 hours to final submission we changed
core C++ code used to communicate from OSGAR to ROS master. The motivation was
loosing sensor data when the system is overloaded, but would you do this in
normal production? Definitely NO!
What else? There are final tune-ups from Jirka not to fall from stairs while
still navigate narrow corridors and climb steep ramps. The wall distance
was set to 80cm at the end.
Finally there were major changes in artifact detection. Full RGBD (image and
depth) data were used and besides estimation of artifacts via colors they were
re-filtered by clip in distance plane (expected median distance +/- 20cm) and
majority of pixels should remain. The last strike was filtering by object
hight, mainly to filter out other red robots and not to report them as
Sigh, fingers crossed! Now there will be 3 weeks pause with no indication what
is going on, so do not ask.
p.s. Thanks Jirka, PavelJ and Zbyněk for extraordinary support!
27th February 2020 — Virtual SubT Urban final results
I do not have enough space left on my hard drive but with the new circuit there
is also a new feature robot_paths.svg! There is no long need to extract
paths of all robots — we get it for free + position of correctly reported and
not reported artifacts.
But that is approximately all for good news. When I looked at the first 4 runs
it looked nice:
… but then there were 20 failing (totally) runs:
It is true that we were probably on the edge of resources … yeah, we should
and we are happy with the 3rd place we got!