Some problem about calculating MAP

[akturtle]

Hi, it seems not precise how you calculate the MAP. I check your function "mean_average_precision". When the relevant_num ==0, this query will be ignored when calculating the mean. That eliminates some extreme bad case (no similar images is retrieved) and the MAP will be a little higher.

[caozhangjie]

I'm sorry for this bug. I have fixed it. But it doesn't have obvious impact on our result since it's unlikely for our method to find 0 relevant code in the database similar to a query.

[akturtle]

Hi, I have one more thing to discuss with you. The relevant_num for calculating average precision in your code is how many similar images in the query result. But from my understanding, the relevan_num is the number that how many similar images should be in the query result (retrieval all similar images). Take the imageNet experiment as an example, the relevant_num should be fixed to 1000

[caozhangjie]

In our code, the relevant_num is how many similar images within the closest 1000 images to the query image. MAP 1000 means that we retrieve the closest 1000 images and calculate the MAP.

[akturtle]

Think about this: If only the top 2 retrieval images are similar, the relevant_number= 2, then the average precision equals to 100%. But there are more than 1000 similar images in your database. The AP should be calculated as (1/1+2/2)/1000 = 0.2%. check this discussion

[kunhe]

AP is a ranking metric. If the top 2 retrievals in the ranked list are relevant (and only the top 2), AP is 100%. You're talking about Recall, which in this case is indeed 0.2%.

[akturtle]

@kunhe. Actually, I'm not pretty sure about this. Have you checked the above discussion? I think when we calculate the MAP, the recall should guarantee to equal to 100%.

[kunhe]

@akturtle If you want 100% recall, you should go down the ranked list until all relevant items are found, say at position K. Sure, you can compute MAP on the sublist from 1 to K, knowing that recall is 100%. But here MAP is computed by fixing K=1000, and 100% recall may or may not happen at that point.