Reviewer 3 of ICRA 2011 submission 870

Comments to the author
======================

The paper describes a visual localization/place recognition
system. The system is designed with outdoor urban
applications in mind, and incorporates a number of features
to attempt to increase overall system performance:
detection and filtering of dynamic objects, selection of
stable features, weighting of visual words, geometric
verification and a probabilistic model for robot location.

My overall feeling is that while this seems like a
well-performing system, I am not fully convinced that it
advances the state of the art. I also have a number of
questions about how certain aspects of the probabilistic
model have been designed. 
I think this would have been a good contribution two or
three years ago, but visual place recognition is now quite
a well-studied problem, and I think the authors need to
make a stronger case that what they propose improves on
current techniques. Many systems have been demonstrated
which tackle full appearance-based SLAM (e.g. author's
references [13] and others), and the simpler problem of
appearance-based localization is even more thoroughly
studied e.g. [8],[9],[11],[12].  The authors make the
argument that Indian roads provide a more challenging task
than has been previously presented in the literature. I do
not really find this convincing. While the data shown in
the paper is certainly challenging, it seems no harder than
that from e.g [A]. To make a strong case for their work,
the authors should show quantitatively that their proposed
modifications outperform current approaches. I do not think
that this case has been made.

The most novel contribution seems to me the filtering of
dynamic objects. Although this is a straight-forward idea,
and the technique presented is fairly heuristic, I am not
aware of an existing system which incorporates it.
Unfortunately, in the evaluation section of the paper, the
effect of this component is not analyzed separately. The
"direct retrieval" baseline (to which it is compared)
removes both feature weighting and dynamic object
filtering, so it is not clear how much each component is
responsible for improving performance. Isolating the effect
of each component and showing a significant contribution
from filtering dynamic obstacles would strengthen the
paper. 

The weighting scheme proposed in the paper is also novel.
However, the baseline to which it is compared is unweighted
features. As almost anything would be expected to
outperform unweighted features, the standard tf-idf
weighting scheme or some other previously published method
would be a more natural point of comparison. Outperforming
that benchmark would make a stronger case that the authors'
proposed scheme is useful. 
Ideally it would be good to include a comparison to an
existing pubic dataset also (several of which exist for
this task).

One advance is claimed that is not novel:
"Our method addresses this problem by using closely sampled
images instead of well separated key images but only adding
features that are stable to the database".
This technique was used by Sivic and Zisserman in 2003
(authors' reference [8]). 

Detailed comments
-----------------------------

Probabilistic model:
A number of probabilistic models for this task have been
proposed before (e.g. [13], [12], tf-idf, etc ). While the
authors model does differ from these approaches, it is
never really explained why the authors developed a new
model or what its advantages were with respect to previous
approaches. (I am not suggesting it does not have any,
merely that it needs to be more clearly motivated in the
paper).

First equation on page 5: 
This implicitly assumes video rate data, which is
potentially a weakness but also a differentiator from
[13],[12].
How was N(x_0) set and how sensitive are the results to
this parameter?
More discussion of the model would be helpful. e.g. If the
set N(x_0) is small, the quality of this model may be worse
than the explicit detector model adopted in [13], though if
the stability of features is strongly location-dependent it
may be better. etc.

Column 2, page 5:
"The weights assigned to the features depend on this prior
distribution and change with P_X(x)"
I do not understand why this should be the case, or why
equation 1, giving P_X(x | Z_j = 0) in terms of this prior,
is written as it is. Intuitively, there seems no reason for
feature distinctiveness should change with position prior.
(Of course, some features will certainly be rarer in some
areas of the map than others, but this will be accounted
for by the multiplication of the prior with the likelihood 
P_X(x | Z_j ). No reweighing of features seems necessary.)
Why could the equation for P_X(x | Z_j = 0) not be written
identically to that for P_X(x | Z_j = 1)?

Also, I did not understand the purpose of the discussion of
mutual information and conditional entropy. It was not
clear how they participated in the scoring. Why can
location not be computed directly from P_X(x | Z_j = 0) and
P_X(x | Z_j = 1)? Surely these, combined with a prior,
specify the entire probability distribution?

Equation 2: Perhaps I am misunderstanding here. 
Firstly, I do not understand why this equation is required
at all. Surely normalizing the scores over all matched
database images will give a pdf over location? Why this
secondary model?
Secondly, it seems that the probability of being at
location x is being given in terms of a normal distribution
with mean c, where c is the number of matched features. Is
this quantity normalized at some point (as otherwise it
seems it will be >1)? Also, it seems that all features are
being weighted equally here, unlike in previous steps.

In section II.D, orientation is discussed. Is this SIFT
feature orientation? Or orientation in some other sense? If
what is  being proposed is broadly to make use of the
feature position within the image, then a reference to the
Pyramid Match Kernel is likely appropriate:
The pyramid match kernel: Discriminative classification
with sets of image features
Grauman, K. and Darrell, T.
2005


Small issues:
--------------------------------------------------

Please number all equations.

Bottom of page 3, typo: "to ensure that unstable"

In Figure 9, it would be helpful to indicate what the
ground truth plot would look like.

Also in Fig 9, typo: "the red graph in the centre". The
graph is blue.


Literature review:
-----------------
While the references are generally fine, I would suggest
perhaps including some more recent works on the topic. e.g

Milford, Michael and Wyeth, Gordon (2008) Single camera
vision-only SLAM on a suburban road network. In:
Proceedings of IEEE International Conference on Robotics
and Automation, 19-23 May 2008, Pasadena Conference Center,
Pasadena, CA.

Mark Cummins and Paul Newman
Highly Scalable Appearance-Only SLAM - FAB-MAP 2.0
RSS 2009, Seattle

Avoiding confusing features in place recognition
Jan Knopp, Josef Sivic, and Tomas Pajdla
In ECCV 2010


References
----------------

[A] 
Mark Cummins and Paul Newman
Highly Scalable Appearance-Only SLAM - FAB-MAP 2.0
RSS 2009, Seattle