Measuring cooperations scientific collaboration has been repeatedly mentioned as a necessary ingredient of science in the making. Science, as a human endeavour, has always been based on international collaborations, but it is only recently that they have been called cooperation. Scientific cooperation appears when support programmes are defined that promote scientific collaboration at the international level.

Scientific cooperation activities are frequently framed by some international, usually state-level, agreement in order to accompany financially the development of international collaborations. Additionally, international programmes and national agencies working at the international level, design and sustain their own cooperation programmes.

As a first result of preliminary studies, the dynamic of international research collaborations shows a large variety of situations, according not only to the countries, but also the sectors or disciplines involved. More analysis is needed, but we can underline that any measurement of international collaborations needs to take into account this diversity.

Most scientific collaborations are measured by the publication output, in particular by measuring co-authorship. This involves a series of technical steps in order to “clean” the data but also to maintain a validated database. More work needs to be done both on the significance of the co-authorships, which involves better knowing the authoring practices in many different disciplines, but also the relation between the actual collaborations and the publication habits.

At last, it is important to measure some aspects such as the ability to design a project, submit to a competitive call with cooperation obligations, establish and maintain a research team which is partly based in different countries, engage in cooperations with research personnel previously not interested in such international activities, and so on. In other words, better measurement depends largely upon a better knowledge of the way the international cooperations are actually engaged and developed.

Indicators that are needed to measure international scientific collaborations go beyond the usual input-output model and do not always relate to the traditional input statistics – such as equivalent full-time researchers, or budgets dedicated to R&D –that are the usual objects of the Frascati-type surveys.

The usual R&D indicators (collected by UIS and national authorities that allow to measure the human and financial resources) or the standard innovation measures (which is the purpose of the NEBIP network and partly taken in charge by the EMIS group inside MIRA), should be distinct from international cooperation indicators, which are quite rarely discussed – except on bibliometric ground. Measuring the impact of cooperation programmes The usual measurement through co-authorship of international collaborations is not enough when one wants to measure the impact of cooperation programmes.

It is necessary to develop a framework that involves a reference (which population is concerned?), a metric (which indicators?) and a temporality (a time frame that permits to oppose “before” and “after”). The impact measurement should be discussed in such a manner that permits to assess the meaning of a general cooperation policy. The framework should be developed that permits the measurement of impacts at different levels: at the national level, at the institutional level of the research institutions (performing institutions, like universities and public research institutes that received the funds), and at the programme level.

The effects can be empirically observed at the individual level, at the group level, at the institutional level or at the level of the scientific institution as a whole. Apart from this diversity, “impact” is a concept that has several meanings at different levels (regional, national or international level):

• Achievement of the programmes objectives;
• Implementation of national objectives/priorities;
• Consolidation of a research area – at the national & international level;
• Promotion of researchers;
• Strengthening of the National scientific community;
• Consolidation - or creation - of a research group;
• Strengthening of a research performing institution;
• Creation of a research network at the national level;

Same, at regional/international level All these impacts depend upon the objectives announced by the programme to be assessed. They also depend upon the capacity we have to identify clearly the universe that we try to assess. Setting ambitious objectives Up to a certain degree, a full and complete evaluation in large cooperation programmes is rather difficult because of the multi-level effects of programmes, the multi-actor nature of these programmes (many distinct populations are impacted by the programme) and the lack of “reference” groups, that is a sort of “control group” that is not affected by the programme. But there are ways to have proxies that permit to overpass these constraints.

They suppose specifically tailored tools, databases and instruments build to collect information:

1. surveys of beneficiaries.
2. clean bibliometric indicators.
3. database of projects/programmes.
4. on-going understanding of social insertion of scientific activities.

hese tools present difficulties and are expensive not so materially –expect maybe some bibliometric databases- but intellectually and in terms of competencies needed to manage them.

The first and foremost difficulty (and cost) relates to competencies needed: it is necessary to have stable competencies and trained personnel who have experience of the uses and benefits from these databases, either bibliographic or other. These people need to have an experience in the management of the database and of the routines needed to interrogate and build meaningful reports.

It is important also to keep in mind that there is a need to assess a more qualitative aspect of international cooperations which relates to the ability of researchers to be accepted socially, to be inserted both into the scientific community – something that goes beyond the sole relations of a research group with its partners in an internationally funded project – and also into society.

A second difficulty lies in the fact that most survey research is not liable to benefit from economies of scale: its cost grows as the actual size of the samples grow.

Finally, the actual cost of a large database is not easy to relate to the benefits one gets from its management. We need thus to build a stable relation between the managers of the data and the uses of the data. This goes beyond the sole creation of the group of experts that know how to manage the data: it relies on the creation of a linkage between the users of the data – policy-makers, public in general, programme managers, etc. – and the producers of the data.

Building the relation between the users of the data and the data producers is an ambitious goal, and it needs for MIRA to be thought of in terms of a large framework, on a large multi-year horizon. The workshop The workshop is designed to produce two separate documents, both part of the White paper: - a validated list of indicators that are needed in order to measure the scientific cooperation;

For this purpose, we will use the first list of indicators elaborated by the ONST from Tunisia which is exposed in the annex table - a preliminary framework for the assessment of impact of programmes. The first objective is to validate the list of indicators that the MIRA Project will use. This will be achieved by a small group of experts gathered during two days.

The workshop will also have to identify resource persons in the MIRA project able to produce indicators in their respective countries. By resource person, we mean people with some previous experience such as participants in the UIS workshops (UIS is the Unesco Institute of Statistics in Montreal) or in other European projects (ESTIME, ASBIMED, NEBIP, ERAWATCH, etc...).

During the first day, a series of lectures will examine different aspects and dimensions of these indicators, as well as the general context of production and use of indicators (institutions producing the data, availability, manageability of indicators, comparability, etc.). Some of the issues of debates (that go back to the Paris International Conference on Science Indicators, 1990) will be re-examined in the light of new sources of information and new methodologies. Apart from listing the indicators, this session will have to identify potential producers of the indicators.

A second objective will be to present and discuss a preliminary framework for the assessment of impact of programmes. It will be based upon previous work, such as the evaluation of INCO programs, assessment of national scientific policies (eg. Morocco evaluation), the assessment of impact of international organizations devoted to international scientific cooperation (eg. the International Foundation for Science), and previous work made on research systems (eg. ESTIME methodological guidelines).

Basic concepts exposed above will be discussed in the Workshop.

Collaboration MIRA – IFRIS

IFRIS (Institut Francilien Recherche,Innovation et Société) is a French public network institute (with its head located in Université Paris-Est Marne-la-Vallée). It is dedicated to basic research on science, innovation and society. IFRIS has been created in 2007 and networks about one hundred full time senior researchers, postdoctoral and doctoral students who are located in various research centers within the Paris-Ile-de-France region, including IRD at Bondy.

IFRIS pools skills and develops common activities in research and training, based on a shared vision of issues pertaining to science, technology and society, scientific policy and innovation, and higher education.

Additional information on IFRIS can be looked at here: http://www.ifris.org A specific group in IFRIS has been set up to design indicators for science policy and many specialists of database management are working around the creation of a technological platform. As a consequence, IFRIS will back up our workshop by offering intellectual resources of high value. Three presentations will be made under the auspices of IFRIS, which will guarantee a lively discussion and interesting interventions for the MIRA project.

For further information, please, download this document (click here)