[robotics-worldwide] CfP and PROGRAM: ICRA 2012 Workshop on the Conditions for Replicable Experiments and Performance Comparison in Robotics Research at ICRA 2012 (St. Paul, Minnesota, USA, May 18, 2012)

Fabio Bonsignorio fabio.bonsignorio at uc3m.es
Tue May 8 04:29:02 PDT 2012


(apologies for multiple postings)

Dear All,

This is an  invitation to participate actively to the Workshop on
the Conditions for Replicable Experiments and Performance Comparison
in Robotics Research at ICRA 2012 (St. Paul, Minnesota, USA, May 18,
2012).
This is a quite hot and intriguing topic with several facets.  We will
leave a lot of space for interventions from the audience and
discusssion.

We will raise some questions and we expect some more from you.
Do we need new scientific insights to achieve long term robotics research aims?
Which are the roles of benchmarks, challenges and competitions for the
advancement of research and practical exploitation of research?
How to connect long term fundamental research, like that of the
robotics FET flagship proposal and short term exploitation?
Is 'experimental methodology' a practical issue, first of all?
(despite its obvious 'theoretical' interest) ?
...

We will discuss sometimes neglected issues such as measuring energy efficiency.
We will check the results obtained so far in benchmarking,
competitions and experimental methods.

* We believe it is now possible to draw some conclusions on a number
of subfields to the benefit of the community. *

PROGRAM:

Friday,  May, 18th

8:45  Opening

9:00  On the FET  Flagship proposal Robot Companions for Citizens
        P. Dario
        Scuola Superiore S. Anna, Italy

9:30  Competitions for Robotics Research and Applications
        S.Balabirsky,
        National Institute of Standards, USA

10:00 Coffee break

10:30 Benchmarking Grasping
         D. Kragic
         Royal Technology Institute KTH, Sweden

11:00 Analyzing Performance of Visual Servoing Approaches
         V. Kyrki
         Lappeenranta University of Technology, Finland

11:30 Characterization of Textile Grasping Experiments
         G. Alenya, A. Ramisa, F. Moreno-Noguer, C. Torras
         CSIC- Universidad Politecnica de Catalunia CSIC-UPC, Spain

12:00 Lunch

13:30 An Analysis of Experimental Trends in Autonomous Robotics Papers
         F. Amigoni, V. Schiaffonati, Politecnico di Milano, Italy
         M.Verdicchio Universita' di Bergamo, Italy

14:00 Towards Stochastic Characterization of Robot Traction in Granular Terrain
          J. Williams, J.Trinkle
          Rensselaer Polytechnic Institute, USA

14:30  Energy in robots: measuring performance and efficiency
          C. Stefanini
          Scuola Superiore S. Anna, Italy

15:00 Coffee break

15;30  Practical and Theoretical  Motivations for Replication of
Experiments in Robotics:
         The complementary roles of benchmarks, challenges and competitions
         F.Bonsignorio
         Universidad Carlos III of Madrid, Spain and Heron Robots

16:00 Discussion

17:00 Conclusions and future initiatives

Each slot will include a minimum of 10 minutes for discussion,
interaction in person and by social media is more than welcome.

Please be aware  that this and other workshops will be videotaped and
archived. These videos will be available to all the workshop
registrants.  Any potential proceeds will go back to the  RAS
Technical Committees.

We are working on other intriguing novelties, too

All the Best

Fabio, Angel and John

Workshop details

Title: Conditions for Replicable Experiments and Performance
Comparison in Robotics Research

Date: May 18th, (full-day workshop)

Abstract:

In all field of science experiments play an important role, in order
to confirm/refute a theory and to discover new theories. It is a
widespread opinion that experimental methodologies in robotics have
not yet achieved a level of maturity comparable with that in
traditional science.
On the other hand being able of objectively measure and comparing
performances is a critical aspect of modern engineering.
In this workshop, we will discuss fundamental issues about the role of
experiments in
robotics, such as how can results be replicable and refutable on the
one hand, and quantitatively comparable according to
community-endorsed metrics to enable a faster cumulative progress, or
even appreciate disruptive changes, on the other end.
We will particularly focus on how might be possible, by providing the
proper kind and amount of data to enable the replication of
experiments as a prerequisite to quantitative comparison of
capabilities.
A key point to allow replication and comparison of results is having
adequate data support: all the data necessary to repeat a given
experiment, how to achieve it with today's digital media will be
addressed.
We will concentrate on three main subfields: visual servoing and
grasping, slam and navigation.
In these fields it is already possible to outline several compelling
criteria. These issues, when viewed in the context of some general
principles about experiments in science and engineering, will allow us
to do some insightful considerations on the role of experiments in
robotics and its scientific and epistemological foundations.
This workshop is a joint initiative of the IEEE Technical Committee on
Performance Evaluation and Benchmarking of Robotic and Automation
Systems (PEBRAS) and the
EURON Special Interest Group on Good Experimental Methodology for
Robotics (SIG GEM).

Proposers:

Fabio Bonsignorio
Board Member European Robotics Network EURON
Co-Chair Euron SIG on Good Experimental Methodologies and Benchmarking
Professor, Banco de Santander Chair of Excellence in Robotics
Universidad Carlos III de Madrid
Avda. Universidad, 30
28911 Leganés, Spain
Tel: +34 916246014
fabio.bonsignorio at uc3m.es

CEO
Heron Robots srl
V.R.Ceccardi 1/18, 16121 Genova,
Italy,
Tel: +39-339-8406011,
fabio.bonsignorio at heronrobots.com

Angel P. del Pobil, Ph.D,
Board Member European Robotics
Network EURON
Co-Chair IEEE-TC PEBRAS
Co-Chair Euron SIG on Good Experimental Methodologies and Benchmarking
Professor, Department of Engineering and Computer Science, Universitat Jaume I
Director, UJI Robotic Intelligence Laboratory
12071 Castellon, Spain,
Tel:+34-964-728293,
pobil at icc.uji.es

John Hallam,
Board Member European Robotics Network EURON
Co-Chair Euron SIG on Good Experimental Methodologies and Benchmarking
Mærsk-McKinney Møller Institute,
University of Southern
Denmark,
Odense, Denmark,
Tel: +45-65503546,
john at mmmi.sdu.dk

With the support of : Raj Madhavan, Past Chair IEEE-TC PEBRAS,
raj.madhavan at nist.gov

List of topics:
* Design of Experiments in Robotics;
* Execution of Experiments in Robotics;
* Reporting experiments in Robotics;
* Epistemological issues;
* Examples of Good Practice;
* Evaluation of Experimental Robotics Work;
* Proposals for Promotion of Good Experimental Work;
* Metrics for sensory motor coordination and visual servoing
effectiveness and efficiency; * Metrics for SLAM effectiveness and
efficiency;
* Metrics for grasping effectiveness and efficiency:
* Benchmarking autonomy and robustness to changes in the environment/task;
* Scalable autonomy measurements;
* Requirements for robots in terms of performance, the approaches to
meeting these requirements, the trade-offs in terms of performance;
* Experimental scenarios to evaluate performance, demonstrate
generality, and measure robustness
* Performance modeling of the relationship between a task and the
environment where it is performed
* Relationship between benchmarking and replication of experiments with robots

Motivation and objectives:

In most cases the measurement robotic system performance, their
evaluation, comparison, characterization etc. involve practical
experimentation, which has to be carried out responsibly and reported
well.
Recently, the interest in experimental methodologies increased
dramatically within
the robotics community, both from researchers, aiming at more grounded
and fast research advancement, and from public funding agencies,
according to the idea that good experimental activities could reduce
the gap between research and industrial applications.
Some projects have been funded by the European Commission and a series
of workshops (see below) have been held in the latest years. While the
definition of proper benchmarking procedures has become a key
requisite of European project funding application.
The aim of this workshop is to
discuss fundamental issues about the role of experiments in robotics,
such as how can results be replicable and refutable on the one hand,
and quantitatively comparable according to community endorsed metrics.
A major output of the EURON SIG GEM is a set of guidelines for good
experimental and reporting practice:
http://www.heronrobots.com/EuronGEMSig/Downloads/GemSigGuidelinesBeta.pdf
An open and deep discussion of these issues, with regards to some
general principles about experiments in science and engineering, will
lead us to derive some insightful considerations on practical issues
and best practices, as well of theoretical natures on the role of
experiments in robotics.
We also aim to lay some foundations toward a replicable robotics
research publishing thread based on the publication of fully
replicable experiments.

* We believe it is now possible to draw some conclusions on a number
of subfields to the benefit of the community. *
The best contributions will be invited to submit to a refereed edited
book summarizing the activities carried on in the latest years in this
perspective.

Primary/secondary audience:

Primary: Robotics researchers from any subfield of the discipline from
both academia and industry.
Secondary: industry members interested in the exploitation of research
results, others interested in methodologies in scientific and
engineering disciplines.
Relation to previous ICRA or IROS workshops

This workshop is a joint initiative of the IEEE-TC on Performance
Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS)
and the EURON Special Interest Group on Good Experimental Methodology
for Robotics (SIG GEM). The proposers are co-chairs of IEEE-TC PEBRAS
and/or EURON SIG GEM and have jointly co-organized over 10 successful
related events in the last 5 years, such as 5-in-a-row workshops on
Benchmarking and Performance Metrics at IROS (2006-2010), three
workshops at RSS (2008-2010) and the Performance Metrics for
Intelligent Systems Workshop (PERMIS) series (2006-2010) and two
organized at ICRA 2010 and 2011 on replication of experiment in
robotics research.

Support
This workshop is supported by IEEE RAS TC-Pebras and the Euron GEM Sig


-- 
CONFIDENTIAL AND PRE-DECISIONAL INFORMATION


Fabio P. Bonsignorio
IEEE Senior Member

Ceo
Heron Robots s.r.l.
Via R.Ceccardi 1/18
I-16121 Genova
Italy
www.heronrobots.com

Professor
Banco de Santander Chair of Excellence in Robotics
Universidad Carlos III de Madrid
Edificio Agustín de Betancourt, Despacho 1.3B01
Avda. Universidad, 30
28911 Leganés
Spain


More information about the robotics-worldwide mailing list