Literature Searching for Realist Synthesis

Literature searching for realist synthesis is complex. It is required to be sufficiently systematic to pass the peer review process but not so mechanical as to impair useful insights. This Wiki puts together some useful resources to help in the literature search process.

In my view literature searching for realist synthesis has four main components. These may be iterative and interactive so the following is not necessarily a sequence.:

A. Constructing a Sampling Frame

B. Searching for Theory

C. Identifying Clusters of contextually rich papers

D. Purposively sampling to generate and test programme theory

After a brief selection of overview materials I highlight the different stages in turn before concluding with guidance from the RAMESES reporting standards:

Literature Searching Overviews

Here is my CARES Conference (November 2014) Powerpoint presentation on In search of Conceptual Richness and Contextual Thickness: Implications of Realist Synthesis for Information Retrieval

Abstract

Realist Synthesis presents an exciting fusion of principles derived from qualitative inquiry and from systematic review techniques, amongst other more overt influences. As such it offers a provocative challenge to established methods of information retrieval as developed over the last two decades. Comprehensive sampling, so often the mainstay of conventional systematic reviews, is clearly unfit for purpose with realist synthesis methods requiring a more theoretical sampling approach. A focus on the individual study report, which is often hamstrung by the constraints of reporting placed by journal word limits, challenges the requirement to develop a rich and full understanding of the theory underlying an intervention programme (conceptual richness) and sufficient detail for a nuanced sensitivity to context (contextual thickness). These twin demands place a premium on informed and systematic application of what are almost dismissively labelled "supplementary search techniques". In fact such search techniques as citation searching and retrieval of "sibling" and "kinship" studies appear to offer a realistic alternative to textword based bibliographic strategies.

This presentation will review a rich variety of reports to date of search methods as used in published realist syntheses. It will identify some common themes and considerations together with associated information retrieval challenges. Drawing from experience of reviews in which he has participated or those he has supervised, the presenter will offer a range of search techniques that can used, either individually or when combined, to meet the exacting demands of realist synthesis. The presenter will update thinking on the previously published CLUSTER method of searching for conceptual richness and contextual thickness [1]. He will rehearse the strengths and limitations of citation searching and discuss the controversial role of systematic searching for theory. He will also explore the potential to enhance the existing RAMESES reporting standards with a fuller acknowledgement of information retrieval considerations [2].

This presentation is intended to form an initial basis for informed discussion about search methods for application within all realist syntheses. The intention would be neither to prescribe nor proscribe the search methods to be used but to offer a toolbox from which methods can be selected judiciously. Above all these methods would seek to remain true in implementing the underlying requirements of the realist synthesis methodology.

[1] Booth A, Harris J, Croot E, Springett J, Campbell F, Wilkins E. Towards a methodology for cluster searching to provide conceptual and contextual "richness" for systematic reviews of complex interventions: case study (CLUSTER). BMC Med Res Methodol. 2013 Sep 28;13:118. doi: 10.1186/1471-2288-13-118. http://www.biomedcentral.com/1471-2288/13/118  
[2] Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: realist syntheses. BMC Med. 2013 Jan 29;11:21. doi: 10.1186/1741-7015-11-21.  http://www.biomedcentral.com/1741-7015/11/21. 

Those clever chaps at PenTAG (Simon Briscoe and Chris Cooper) have produced very useful presentations on Literature Searching for Realist Reviews [Briscoe]:

https://medicine.exeter.ac.uk/media/universityofexeter/medicalschool/research/pentag/documents/PenSR_April_slides_final.pptx

From CARES Conference [Cooper]:

Realist literature searching – towards processes and standards

Authors: Chris Cooper (Christopher.Cooper@exeter.ac.uk - University of Exeter Medical School), Harriet Hunt, Mark Pearson, Rob Anderson

Abstract

Realist approaches to primary evaluation and evidence synthesis are increasingly advocated as a method for explaining the effectiveness of complex interventions. Methods for conducting a Realist evidence synthesis ('Realist review') differ from those conventionally considered to be 'systematic' and they are being substantively developed and tested. This point is markedly clear as it relates to literature searching, where the difficult issue of designing, conducting and documenting pragmatic literature searches for a Realist review remains un-addressed. 

Objectives of this session
• to explore the role of literature searches in Realist reviews (what are we trying to do?);
• to present a model of pragmatic literature searching in Realist reviews, drawing from a range of systematic literature search techniques (matching search techniques to information needs); and
• to discuss the need for transparency in the evidence gathering process (why we should always show our workings). 

Background to this work: Twelve Realist reviews and the RAMESES publication guidelines were analysed in the preparation of a literature searching protocol for a Realist review of Intermediate Care. The literature searching methods used in these twelve reviews were posited alongside traditional 'systematic' literature search techniques and approaches (e.g. database searching, citation analysis etc.) to develop a literature search protocol for our Realist review. 

Building on our experience of searching in Public Health systematic reviews, and systematic reviews of complex interventions, we structured a search protocol which matched our review's aims with the complex information needs of a Realist synthesis and the evidence base we had to work with. 

What we found:
• various literature search techniques have been used in Realist reviews and they have been used in a variety of ways to meet the complex information needs of Realist review processes; 
• there is no guidance or methodological work on literature searching for Realist reviews; and
• existing Realist reviews typically take a less rigid approach to recording their literature searches, making it harder to trace the development of the review's evidence base. 

Conclusions: Any form of evidence which provides insight is valid in a Realist review and it may be used at any time within a review. Capturing relevant evidence, at the right time, requires the development of new search techniques – or new combinations of searches - so the evidence can be pragmatically located. 

This session will propose a pragmatic model to literature searching in Realist reviews. Using multiple searches, each planned to satisfy different information needs, we will describe how literature searches can be pragmatically integrated and transparently recorded in Realist synthesis.

See Also: RAMESES Training Materials: Section 5 - Developing a search strategy (Pages 29-33)

http://www.ramesesproject.org/media/Realist_reviews_training_materials.pdf 

Because elements of realist searching are shared with searching for qualitative evidence the following may be useful Background:

Booth A. Chapter 3: Searching for Studies. In: Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, Lockwood C (editors), Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Version 1 (updated August 2011). Cochrane Collaboration Qualitative Methods Group, 2011.

http://cqim.cochrane.org/sites/cqim.cochrane.org/files/uploads/DG%20Chapter5_Searching_for_Studies%20(2).doc

Finfgeld‐Connett, D., & Johnson, E. D. (2013). Literature search strategies for conducting knowledge‐building and theory‐generating qualitative systematic reviews. Journal of advanced nursing, 69(1), 194-204.

Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010). Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Information & Libraries Journal, 27(2), 114-122.

http://www.researchgate.net/publication/44690064_Literature_searching_for_social_science_systematic_reviews_consideration_of_a_range_of_search_techniques/links/00b4951e9ac9e5995e000000

A. Constructing a Sampling Frame

Jagosh, J., Pluye, P., Wong, G., Cargo, M., Salsberg, J., Bush, P. L., ... & Macaulay, A. C. (2014). Critical reflections on realist review: insights from customizing the methodology to the needs of participatory research assessment. Research Synthesis Methods.  5(2):131-141 

B. Searching for Theory

This is not covered very extensively in the literature. We have included it alongside other things in the following paper:

Booth A, Harris J, Croot E, Springett J, Campbell F, Wilkins E. Towards a methodology for cluster searching to provide conceptual and contextual "richness" for systematic reviews of complex interventions: case study (CLUSTER). BMC Med Res Methodol. 2013 Sep 28;13:118. doi: 10.1186/1471-2288-13-118. http://www.biomedcentral.com/1471-2288/13/118 

We have published an article on a methodology for searching for theory:

Booth, A., & Carroll, C. (2015). Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable?. Health Information & Libraries Journal, 32(3), 220-235. https://www.researchgate.net/profile/Andrew_Booth/publication/279067860_Systematic_searching_for_theory_to_inform_systematic_reviews_Is_it_feasible_Is_it_desirable/links/55f2952008ae0af8ee1f8f48.pdf

Not realist synthesis but useful insights on searching for theory:

Challenges facing a BEME systematic review of the contribution of theory to the development & delivery of  effective interprofessional curricula in health professional education

http://eprints.bournemouth.ac.uk/21049/2/BEME_._HEAN_2012.pdf 

Pound, P., & Campbell, R. (2015). Locating and applying sociological theories of risk-taking to develop public health interventions for adolescents.Health Sociology Review, 24(1), 64-80. http://www.tandfonline.com/doi/full/10.1080/14461242.2015.1008537#.VmsS6kqLTcs

Campbell, M., Egan, M., Lorenc, T., Bond, L., Popham, F., Fenton, C., & Benzeval, M. (2014). Considering methodological options for reviews of theory: illustrated by a review of theories linking income and health. Systematic reviews, 3(1), 1-11. http://www.systematicreviewsjournal.com/content/3/1/114

C. Identifying Rich Clusters

A single article is unlikely to contain sufficient conceptual and contextual detail to sustain a synthesis. We suggest that you look for sibling papers (i.e. multiple reports from the same project) or kinship papers (antecedent and descendant initiatives, overviews and reviews of projects etcetera:

Booth A, Harris J, Croot E, Springett J, Campbell F, Wilkins E. Towards a methodology for cluster searching to provide conceptual and contextual "richness" for systematic reviews of complex interventions: case study (CLUSTER). BMC Med Res Methodol. 2013 Sep 28;13:118. doi: 10.1186/1471-2288-13-118. http://www.biomedcentral.com/1471-2288/13/118  

D. Purposively Sampling to Generate and Test Programme Logic (Theory)

Librarians know very little about sampling, except comprehensive sampling for systematic reviews and opportunistic sampling for quick queries. Here is a paper that can be used to share learning on searching and sampling:

Suri, H. (2011). Purposeful sampling in qualitative research synthesis.Qualitative Research Journal, 11(2), 63-75.

http://www.emeraldinsight.com/doi/abs/10.3316/QRJ1102063 [May require subscription]

Also see Chapter 8 - Searching for Studies (p.88-107) in: Suri, H. (2013). Towards Methodologically Inclusive Research Syntheses: Expanding Possibilities. Routledge.

Typically you would test your theories from within your initial sampling frame first. However it is possible (indeed highly likely) that theory-relevant materials may exist in areas outside this sampling frame. In which case you would typically conduct very targeted focused searches of the literature ("literature biopsies") to identify specific evidence to confirm or disconfirm your emerging theory.  

According to RAMESES 

Jeannette Buckingham from Alberta, Canada-  a very wise librarian and good friend, input into the search methods for the RAMESES Publication Standards [See Ref below: http://www.biomedcentral.com/1741-7015/11/21]. They remain valid in the light of our own experience>

•Searching should be guided by the objectives and focus of the synthesis, and revised iteratively in the light of emerging data. Data…may lie in a broad range of sources that may cross traditional disciplinary, program and sector boundaries. The search phase is thus likely to involve searching for different sorts of data, or studies from different domains, with which to test different aspects of any provisional theory. ü

•Search methods using forward and backward citation tracking may be particularly valuable in finding the documents necessary to develop and then test provisional theories. ü

•Realist syntheses do not exclude sources solely on the basis of their study design; hence, 'methodological filters' …may add little to the search and could potentially miss relevant papers. ü

•Searching is likely to be iterative because, as the synthesis progresses, new or refined elements of theory may be required to explain particular findings, or to examine specific aspects of particular processes. ü 

•As new elements of theory are included, searches for evidence to support, refute or refine those elements may be required…the process used for any such additional searches should be clearly documented. A single pre-defined search is unlikely to be sufficient and may suggest insufficient reflection on emerging findings. ü

•Sufficient detail should be given to enable the reader to judge whether searching was likely to have located sources needed for theory building and/or testing. ü

Source: Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: realist syntheses. BMC Medicine,11(1), 21. 

 http://www.biomedcentral.com/1741-7015/11/21

This resource is designed to help in the literature searching process. Further resources will be added as they are identified or become available. Suggestions welcomed. Email them to A.Booth@sheffield.ac.uk