In this meeting note, Sherin Maria Saji reflects on her participation in the workshop on systematic reviews and evidence and gap maps organised by the Evidence Module of the CGIAR GENDER Platform and the Campbell Collaboration.
CONTEXT
A 3-day training programme for researchers and postgraduate students was organised by the Evidence Module of the CGIAR Gender Platform in partnership with the Campbell Collaboration. The Evidence Module of the CGIAR Gender Platform primarily concentrates on co-creating, assembling, and disseminating credible evidence, identifying emergent issues, and filling data gaps in the area of gender in agricultural and food systems. The Campbell Collaboration works to advance good social and economic change by creating and utilising systematic reviews and other types of evidence synthesis for evidence-based policy and practice.
The program aimed to introduce the concept of Systematic Review (SR) and an Evidence and Gap Map (EGM) tool. Participants included researchers and scientists from the ICAR-Central Institute of Freshwater Aquaculture (ICAR-CIFA), International Livestock Research Institute (ILRI), World Fish, and Masters and PhD students from ICAR-Indian Agricultural Research Institute (ICAR- IARI) and Anand Agricultural University (AAU).
TOPICS COVERED
EXPLORING EVIDENCE AND GAP MAPS
Evidence Gap Maps (EGM) is a matrix of intervention categories (rows) and outcome domains (columns) with bubbles showing the type and number of studies. An EGM systematically presents all available, relevant evidence for a particular sector or sub-sector. If we compare EGM and Systematic reviews, EGMs are broader in scope than systematic reviews. Systematic reviews inform policy, and EGMs primarily inform research priorities. Dr Bhumika TV led this topic and at the end she provided links to various EGM types. The maps were made available for the participants to examine. They were then asked to rate them according to four criteria: ease of use and understanding, achievement of the purpose, and need for updating.
Participants were also asked to identify policy implications from these maps. In this we reviewed 4-5 papers under the category of high confidence. We glanced through the content, especially the abstracts and conclusions. Based on our interpretation of it we discussed some of the policy implications. So, by analysing pertinent studies in the evidence and gap map, policy implications can be drawn and this aids in making evidence-based decision making and informed judgements.
Figure 1: An Evidence and Gap Map on agriculture innovation
FORMULATING RESEARCH QUESTION FOR SYSTEMATIC REVIEWS & EGMS
Dr Howard White led this topic wherein he strongly emphasised narrowing the umbrella of any subject for conducting a systematic review. A systematic review is a research paper, also known as a report, that seeks answers to a pre-defined question using a method known as ‘evidence synthesis’. A systematic review’s goal is to compile the most relevant studies on a given topic. This is accomplished by combining the findings of various studies. A systematic review retrieves, assesses, and summarises the findings of pertinent research using a transparent methodology.
The title of the systematic review was therefore created using the PICOS method. PICOS stands for ‘Population’, ‘Intervention’, ‘Comparison’ ‘Outcome’, and ‘Study design’. For instance, if one is doing a systematic review on ‘Impact of agricultural insurance schemes on farm income of smallholder farmers in South Asian developing countries: A systematic review’, the details could be as follows:
Population: Farmers
Intervention: Agricultural Insurance
Comparison: Beneficiaries and non-beneficiaries of agricultural insurance schemes
Outcome: Impact of insurance on farm income
Study design: Quasi experimental.
The attendees were divided into teams for this session (4-5 people/team). Each team was given a team code and instructed to brainstorm any pertinent research issues and create a title using the PICOS format.
SEARCH STRATEGY
Ashima Mohan, Campbell South Asia’s Manager of Research and Communication, led this topic. She specifically covered various search sources and search tactics (e.g., Boolean Operators such as AND, NOT and OR) to extract relevant and focused literatures from the pool of online materials, for example, Agriculture Insurance AND (Farmer OR Farm woman). The Boolean operators mainly add value to the search process where one can include or exclude key terms while searching and it helps to avoid human errors in the search process. A search strategy is important to save time and to achieve precise search results. Sources of information are grey literature, databases, physical search of journals, snowballing references or citations, and contacting experts. One can use filters and other add-ons to make the search process more focussed and convenient.
SCREENING FOR EVIDENCE AND GAP MAPS AND SYSTEMATIC REVIEWS
Sabina Singh, Director, Research Campbell South Asia, led this session. The discussion focused on developing screening tools for gathering pertinent information for EGM and systematic reviews. The screening tool can be developed from PICOS. We can screen the relevant articles based on the intended language, region, population, intervention and study design. The screened articles are subjected to review so as to address the formulated research problem.
CRITICAL APPRAISAL
This session was conducted by Suchi Kapoor Malhotra, Evidence Synthesis Specialist of Campbell South Asia. The discussion centred on the idea of critical appraisal and how it can be a valuable tool for gauging the level of confidence in study findings. Critical appraisal of literature may help distinguish between useful and flawed studies. The articles need to be critically evaluated by rating each article on a scale of low, medium and high, providing relevant justifications for each confidence level of the appraised articles.
EFFECT SIZES AND FOREST PLOT
Dr Neha Gupta, Evidence Synthesis Specialist of Campbell South Asia, led this topic that focussed on calculating and comprehending effect sizes. The difference in mean outcomes between the intervention (treatment) group and the control (comparison) group is measured by the effect size. Forest plots are graphical displays of findings of systematic reviews and meta-analyses which also help in the interpretation of the analysed data.
EPPI REVIEWER-SOFTWARE
This topic was again led by Suchi Kapoor Malhotra, Evidence Synthesis Specialist of Campbell South Asia. She focused on using EPPI Reviewer-Software which is an all-inclusive package for conducting systematic reviews. It helps in managing and analysing the systematic review literature and for meta-analysis and report preparation. It is a paid web-based application. It makes the analysis work easier. There are other free softwares such as RevMan and Rayyan, which is useful for screening or meta-analysis. But these cannot cover all the aspects in one place as EPPI Reviewer does.
CAMPBELL REPORTING STANDARDS AND TITLE REGISTRATION FORM (TRF)
This topic was led by Ashrita Saran, Senior Advisor (Consultant) of Campbell South Asia. She emphasised the essential standards for systematic reviews and EGMs. The standards are meant to maintain the quality of the reviews. For instance, one should publish protocol before beginning a review. We should use structured data extraction and coding forms. PRISMA flow chart is yet another standard of SRs. I have added a reference link to review the policy and guidelince for Campbell reviews.
To produce Campbell reviews, the first step one needs to take is to submit a title registration form online, second step is to develop a protocol, and third is to start the review process/EGMs. The intention is to have a clearly stated research problem along with the scope of the review, before starting the review process. It adopts PICOS format to fill in the TRF. The scope of Title that will be registered should be determined in consultation with key stakeholders including the intended end users.
USE OF SYSTEMATIC REVIEWS IN EVIDENCE-BASED RESEARCH AND DECISION-MAKING
Dr Howard White led this topic that discussed the value of systematic reviews and how they have evolved as tools for sound decision-making. Evidence maps serve as the base of the pyramid to determine how strong the foundations are for producing top-level evidence products. It can be used to find research gaps and research priorities. So EGMs are not usually an end in themselves but a step towards further work. The elements of building evidence architecture is given in Figure 2.
Figure 2: Elements of building evidence architecture
MY IMPRESSIONS
All the topics covered in the three days were meticulously planned and the entire programme was very engaging. The workshop had the ideal balance of lectures, group projects, and hands-on activities. As a result, it was possible to retain the novel information. Groups were used for all activities. Over the three days, each group specialised in tackling one research issue. All of the sessions are linked together to form a whole. This workshop was introductory. There are two more workshops coming up for advanced learning.
A lot of research work that we undertake may turn out to be repetitive in terms of the outcome it delivers. Systematic reviews will aid in understanding the kind of research work already done on a particular aspect. It reveals the extent of research work already undertaken. Hence, we can interpret and extend our primary research based on these reviews.
The Evidence and Gap Maps are a step further where we consolidate the reviews and primary studies based on the intervention and outcomes. It is an interactive map which helps us identify gaps in the research. You may explore some of the maps linked below to know more about Evidence and Gap Maps. It will aid in prioritising research areas and subjects in particular. At present I am discovering more on this subject. Working with teams to develop evidence, gap maps, and systematic reviews will be particularly interesting. So, I would be interested to aid teams working in this field and explore more of these aspects. For those interested in exploring more on these topics I recommend the links given below:
EGM
- Agricultural innovation: an evidence gap map. https://gapmaps.3ieimpact.org/evidence-maps/agricultural-innovation
- Evidence map https://beamexchange.org/
- RESULTS – Evidence Map of Mindfulness – NCBI Bookshelf https://www.nih.gov/
- https://www.unicef-irc.org/evidence-gap-map-violence-against-children/
- Cataract evidence gap map https://www.sightsavers.org/
Systematic Reviews:
Campbell Standards for Systematic reviews
This technique is currently used by international institutions such as Campbell Collaboration and 3ie. It is frequently used to draw attention to methodological issues in research studies that can enhance ensuing research. For researchers, learning how to conduct SRs and create EGMs is thus a worthwhile endeavour because it can help them develop new research ideas, improve their understanding of their area of interest, and develop critical synthesis skills.
Sherin Maria Saji is an MSc student (Agricultural Extension and Communication) from Anand Agricultural University, Anand, Gujarat, India. She can be reach on Email: sherinmariasaji@gmail.com.
It’s appreciable that Master’s students are not only attending such learning events but also sharing what they have learnt in the form of meeting notes-we call this good practice! Congratulations Sherin and AESA for this well written Meeting Note. Two PhD students from my Division were also selected for this programme, but couldn’t join due to clash of dates with their final practical exam dates.
Thank you for your encouragement Sir.
Congratulations Sherin!!
It’s encouraging to learn that the programme is extremely beneficial and essential to researchers. Even some of the platforms for reviews are quite new to me. Thank you for sharing your insight, and I’m also happy to have you as a student at Anand Agricultural University, Gujarat.
I must also recognise the CGIAR group’s work in support of such initiatives.
Best wishes for your further endeavours.