[robotics-worldwide] First Call: ICRA 2012 Workshop on the Conditions for Replicable Experiments and Performance Comparison in Robotics Research at ICRA 2012 (St. Paul, Minnesota, USA, May 18, 2012)

Fabio Bonsignorio fabio.bonsignorio at uc3m.es
Thu Jan 19 04:49:28 PST 2012


(apologies for multiple postings)

Dear All,

This is an early invitation to participate actively to the Workshop on
the Conditions for Replicable Experiments and Performance Comparison
in Robotics Research at ICRA 2012 (St. Paul, Minnesota, USA, May 18,
2012).
This is a quite hot and intriguing topic.

* We believe it is now possible to draw some conclusions on a number
of subfields to the benefit of the community. *

The best contributions will be invited to submit to a refereed edited
book summarizing the activities carried on in the latest years in this
perspective.

Please be aware  that this and other workshops will be videotaped and
archived. These videos will be available to all the workshop
registrants.  Any potential proceeds will go back to the  RAS
Technical Committees.
It is important that all presenters sign a Video Release Form.

People wishing to contribute must send an IEEE formatted extended
abstract to one of the organizers (and will be asked in case of
acceptance to sign the form above).
We are working on other intriguing novelties, too

All the Best

Fabio, Angel and John

Workshop details

Title: Conditions for Replicable Experiments and Performance
Comparison in Robotics Research

Date: May 18th, (full-day workshop)

Abstract:

In all field of science experiments play an important role, in order
to confirm/refute a theory and to discover new theories. It is a
widespread opinion that experimental methodologies in robotics have
not yet achieved a level of maturity comparable with that in
traditional science.
On the other hand being able of objectively measure and comparing
performances is a critical aspect of modern engineering.
In this workshop, we will discuss fundamental issues about the role of
experiments in
robotics, such as how can results be replicable and refutable on the
one hand, and quantitatively comparable according to
community-endorsed metrics to enable a faster cumulative progress, or
even appreciate disruptive changes, on the other end.
We will particularly focus on how might be possible, by providing the
proper kind and amount of data to enable the replication of
experiments as a prerequisite to quantitative comparison of
capabilities.
A key point to allow replication and comparison of results is having
adequate data support: all the data necessary to repeat a given
experiment, how to achieve it with today's digital media will be
addressed.
We will concentrate on three main subfields: visual servoing and
grasping, slam and navigation.
In these fields it is already possible to outline several compelling
criteria. These issues, when viewed in the context of some general
principles about experiments in science and engineering, will allow us
to do some insightful considerations on the role of experiments in
robotics and its scientific and epistemological foundations.
This workshop is a joint initiative of the IEEE Technical Committee on
Performance Evaluation and Benchmarking of Robotic and Automation
Systems (PEBRAS) and the
EURON Special Interest Group on Good Experimental Methodology for
Robotics (SIG GEM).

Proposers:

Fabio Bonsignorio
Board Member European Robotics Network EURON
Co-Chair Euron SIG on Good Experimental Methodologies and Benchmarking
Professor, Banco de Santander Chair of Excellence in Robotics
Universidad Carlos III de Madrid
Avda. Universidad, 30
28911 Leganés, Spain
Tel: +34 916246014
fabio.bonsignorio at uc3m.es

CEO
Heron Robots srl
V.R.Ceccardi 1/18, 16121 Genova,
Italy,
Tel: +39-339-8406011,
fabio.bonsignorio at heronrobots.com

Angel P. del Pobil, Ph.D,
Board Member European Robotics
Network EURON
Co-Chair IEEE-TC PEBRAS
Co-Chair Euron SIG on Good Experimental Methodologies and Benchmarking
Professor, Department of Engineering and Computer Science, Universitat Jaume I
Director, UJI Robotic Intelligence Laboratory
12071 Castellon, Spain,
Tel:+34-964-728293,
pobil at icc.uji.es

John Hallam,
Board Member European Robotics Network EURON
Co-Chair Euron SIG on Good Experimental Methodologies and Benchmarking
Mærsk-McKinney Møller Institute,
University of Southern
Denmark,
Odense, Denmark,
Tel: +45-65503546,
john at mmmi.sdu.dk

With the support of : Raj Madhavan, Chair IEEE-TC PEBRAS, raj.madhavan at nist.gov

List of topics:
* Design of Experiments in Robotics;
* Execution of Experiments in Robotics;
* Reporting experiments in Robotics;
* Epistemological issues;
* Examples of Good Practice;
* Evaluation of Experimental Robotics Work;
* Proposals for Promotion of Good Experimental Work;
* Metrics for sensory motor coordination and visual servoing
effectiveness and efficiency; * Metrics for SLAM effectiveness and
efficiency;
* Metrics for grasping effectiveness and efficiency:
* Benchmarking autonomy and robustness to changes in the environment/task;
* Scalable autonomy measurements;
* Requirements for robots in terms of performance, the approaches to
meeting these requirements, the trade-offs in terms of performance;
* Experimental scenarios to evaluate performance, demonstrate
generality, and measure robustness
* Performance modeling of the relationship between a task and the
environment where it is performed
* Relationship between benchmarking and replication of experiments with robots

Motivation and objectives:

In most cases the measurement robotic system performance, their
evaluation, comparison, characterization etc. involve practical
experimentation, which has to be carried out responsibly and reported
well.
Recently, the interest in experimental methodologies increased
dramatically within
the robotics community, both from researchers, aiming at more grounded
and fast research advancement, and from public funding agencies,
according to the idea that good experimental activities could reduce
the gap between research and industrial applications.
Some projects have been funded by the European Commission and a series
of workshops (see below) have been held in the latest years. While the
definition of proper benchmarking procedures has become a key
requisite of European project funding application.
The aim of this workshop is to
discuss fundamental issues about the role of experiments in robotics,
such as how can results be replicable and refutable on the one hand,
and quantitatively comparable according to community endorsed metrics.
A major output of the EURON SIG GEM is a set of guidelines for good
experimental and reporting practice:
http://www.heronrobots.com/EuronGEMSig/Downloads/GemSigGuidelinesBeta.pdf
An open and deep discussion of these issues, with regards to some
general principles about experiments in science and engineering, will
lead us to derive some insightful considerations on practical issues
and best practices, as well of theoretical natures on the role of
experiments in robotics.
We also aim to lay some foundations toward a replicable robotics
research publishing thread based on the publication of fully
replicable experiments.

* We believe it is now possible to draw some conclusions on a number
of subfields to the benefit of the community. *
The best contributions will be invited to submit to a refereed edited
book summarizing the activities carried on in the latest years in this
perspective.

Primary/secondary audience:

Primary: Robotics researchers from any subfield of the discipline from
both academia and industry.
Secondary: industry members interested in the exploitation of research
results, others interested in methodologies in scientific and
engineering disciplines.
Relation to previous ICRA or IROS workshops

This workshop is a joint initiative of the IEEE-TC on Performance
Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS)
and the EURON Special Interest Group on Good Experimental Methodology
for Robotics (SIG GEM). The proposers are co-chairs of IEEE-TC PEBRAS
and/or EURON SIG GEM and have jointly co-organized over 10 successful
related events in the last 5 years, such as 5-in-a-row workshops on
Benchmarking and Performance Metrics at IROS (2006-2010), three
workshops at RSS (2008-2010) and the Performance Metrics for
Intelligent Systems Workshop (PERMIS) series (2006-2010) and two
organized at ICRA 2010 and 2011 on replication of experiment in
robotics research.

Support
This workshop is supported by IEEE RAS TC-Pebras and the Euron GEM Sig


More information about the robotics-worldwide mailing list