DIRECTions: Design and Specification of an IR Evaluation Infrastructure

Information Retrieval (IR) experimental evaluation is an essential part of the research on and development of information access methods and tools. Shared data sets and evaluation scenarios allow for comparing methods and systems, understanding their behaviour, and tracking performances and progress over the time. On the other hand, experimental evaluation is an expensive activity in terms of human effort, time, and costs required to carry it out.

Software and hardware infrastructures that support experimental evaluation operation as well as management, enrichment, and exploitation of the produced scientific data provide a key contribution in reducing such effort and costs and carrying out systematic and throughout analysis and comparison of systems and methods, overall acting as enablers of scientific and technical advancement in the field. This paper describes the specification for an Information Retrieval (IR) evaluation infrastructure by conceptually modeling the entities involved in Information Retrieval (IR) experimental evaluation and their relationships and by defining the architecture of the proposed evaluation infrastructure and the APIs for accessing it.

Unable to display preview. Download preview PDF.

1. Harman, D.K.: Information Retrieval Evaluation. Morgan & Claypool Publishers, USA (2011)

   Google Scholar 

2. Rowe, B.R., Wood, D.W., Link, A.L., Simoni, D.A.: Economic Impact Assessment of NIST’s Text REtrieval Conference (TREC) Program. RTI Project Number 0211875, RTI International, USA (2010), http://trec.nist.gov/pubs/2010.economic.impact.pdf

3. Allan, J., et al.: Frontiers, Challenges, and Opportunities for Information Retrieval – Report from SWIRL 2012. In: The Second Strategic Workshop on Information Retrieval in Lorne, SIGIR Forum, vol. 46 (in print, February 2012)

   Google Scholar 

4. Sanderson, M.: Test Collection Based Evaluation of Information Retrieval Systems. Foundations and Trends in Information Retrieval (FnTIR) 4, 247–375 (2010)

   Article  MATH  Google Scholar 

5. Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Improvements That Don’t Add Up: Ad-Hoc Retrieval Results Since 1998. In: Proc. 18th International Conference on Information and Knowledge Management (CIKM 2009), pp. 601–610. ACM Press, New York (2009)

   Google Scholar 

6. Zhang, J.: Visualization for Information Retrieval. Springer, Heidelberg (2008)

   Book  MATH  Google Scholar 

7. Newman, D., Baldwin, T., Cavedon, L., Huang, E., Karimi, S., Martínez, D., Scholer, F., Zobel, J.: Visualizing Search Results and Document Collections Using Topic Maps. Journal of Web Semantics 8, 169–175 (2010)

   Article  Google Scholar 

8. Banks, D., Over, P., Zhang, N.F.: Blind Men and Elephants: Six Approaches to TREC data. Information Retrieval 1, 7–34 (1999)

   Article  Google Scholar 

9. Sormunen, E., Hokkanen, S., Kangaslampi, P., Pyy, P., Sepponen, B.: Query performance analyser: a web-based tool for ir research and instruction. In: Järvelin, K., Beaulieu, M., Baeza-Yates, R., Hyon Myaeng, S. (eds.) Proceedings of SIGIR 2002, p. 450. ACM, New York (2002)

   Chapter  Google Scholar 

10. Keim, D.A., Mansmann, F., Schneidewind, J., Ziegler, H.: Challenges in Visual Data Analysis. In: Banissi, E. (ed.) Proc. of the 10th International Conference on Information Visualization (IV 2006), pp. 9–16. IEEE Computer Society, Los Alamitos (2006)

    Chapter  Google Scholar 

11. Di Buccio, E., Dussin, M., Ferro, N., Masiero, I., Santucci, G., Tino, G.: To Re-rank or to Re-query: Can Visual Analytics Solve This Dilemma? In: Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., de Rijke, M. (eds.) CLEF 2011. LNCS, vol. 6941, pp. 119–130. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

12. Ferro, N., Sabetta, A., Santucci, G., Tino, G.: Visual Comparison of Ranked Result Cumulated Gains. In: Miksch, S., Santucci, G. (eds.) Proc. 2nd International Workshop on Visual Analytics (EuroVA 2011), pp. 21–24. Eurographics Association, Goslar (2011)

    Google Scholar 

13. European Union: Riding the wave. How Europe can gain from the rising tide of scientific data. Printed by Osmotica.it, Final report of the High level Expert Group on Scientific Data (2010)

    Google Scholar 

14. Agosti, M., Di Nunzio, G.M., Ferro, N.: Scientific Data of an Evaluation Campaign: Do We Properly Deal with Them? In: Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M. (eds.) CLEF 2006. LNCS, vol. 4730, pp. 11–20. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

15. Di Nunzio, G.M., Ferro, N.: DIRECT: A System for Evaluating Information Access Components of Digital Libraries. In: Rauber, A., Christodoulakis, S., Tjoa, A.M. (eds.) ECDL 2005. LNCS, vol. 3652, pp. 483–484. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

16. Dussin, M., Ferro, N.: Managing the Knowledge Creation Process of Large-Scale Evaluation Campaigns. In: Agosti, M., Borbinha, J., Kapidakis, S., Papatheodorou, C., Tsakonas, G. (eds.) ECDL 2009. LNCS, vol. 5714, pp. 63–74. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

17. Agosti, M., Ferro, N., Thanos, C.: DESIRE 2011: First international workshop on data infrastructures for supporting information retrieval evaluation. In: Proc. of the 20th ACM International Conference on Information and Knowledge Management, pp. 2631–2632. ACM, New York (2011)

    Google Scholar 

18. Voorhees, E.M., Harman, D.K.: TREC: Experiment and Evaluation in Information Retrieval. The MIT Press, MA (2005)

    Google Scholar 

19. Elmasri, R., Navathe, S.B.: Fundamentals of Database Systems, 4th edn. Addison Wesley, Reading (2003)

    Google Scholar 

Download references

Authors and Affiliations

1. Department of Information Engineering, University of Padua, Italy

   Maristella Agosti, Emanuele Di Buccio, Nicola Ferro, Ivano Masiero, Simone Peruzzo & Gianmaria Silvello

Editors and Affiliations

1. Department of Computer, Control and Management, Engenerring Antonio Ruberti, Sapienza University of Rome, Via Ariosto 25, 00185, Rome, Italy

   Tiziana Catarci

2. Center for the Evaluation of Language and Communication Technologies (CELCT), Via alla Casata 56/c, 38123, Povo, TN, Italy

   Pamela Forner

3. Department of Computer Science, Database Group, University of Twente, PO Box 217, 7500 AE, Enschede, The Netherlands

   Djoerd Hiemstra

4. UNED Natural Language Processing and Information Retrieval Research Group, E.T.S.I. Informática de la UNED, c/ Juan del Rosal 16, 28040, Madrid, Spain

   Anselmo Peñas

5. Department of Computer, Control and Management, Engeneering Antonio Ruberti, Sapienza University of Rome, Via Ariosto 25, 00185, Rome, Italy

   Giuseppe Santucci

Cite this paper

Agosti, M., Di Buccio, E., Ferro, N., Masiero, I., Peruzzo, S., Silvello, G. (2012). DIRECTions: Design and Specification of an IR Evaluation Infrastructure. In: Catarci, T., Forner, P., Hiemstra, D., Peñas, A., Santucci, G. (eds) Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. CLEF 2012. Lecture Notes in Computer Science, vol 7488. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33247-0_11

Download citation

• .RIS
• .ENW
• .BIB

• DOI: https://doi.org/10.1007/978-3-642-33247-0_11

• Publisher Name: Springer, Berlin, Heidelberg

• Print ISBN: 978-3-642-33246-3

• Online ISBN: 978-3-642-33247-0

• eBook Packages: Computer ScienceComputer Science (R0)