DHQ: Digital Humanities Quarterly
Preview
2020
Volume 14 Number 
Preview  |  XML |  Discuss ( Comments )

A Case Study Protocol for Meta-Research into Digital Practices in the Humanities

Maciej Maryl  <maciej_dot_maryl_at_ibl_dot_waw_dot_pl>, Institute of Literary Research of the Polish Academy of Sciences, Poland
Costis Dallas  <costis_dot_dallas_at_utoronto_dot_ca>, Faculty of Information, University of Toronto; Digital Curation Unit, IMSI-Athena Research Centre
Jennifer Edmond  <edmondj_at_tcd_dot_ie>, School of Languages Literatures and Cultural Studies, Trinity College Dublin, Ireland; DARIAH -EU
Jessie Labov  <labovj_at_ceu_dot_edu>, Center for Media, Data, and Society, Central European University, Hungary
Ingrida Kelpšienė  <ingrida_dot_kelpsiene_at_kf_dot_vu_dot_lt>, Vilnius University Faculty of Communication
Michelle Doran  <doranm1_at_tcd_dot_ie>, Trinity College Dublin
Marta Kołodziejska  <marta_dot_kolodziejska_at_ifispan_dot_waw_dot_pl>, Institute of Literary Research of the Polish Academy of Sciences, Poland
Klaudia Grabowska  <klaudia_dot_grabowska_at_gmail_dot_com>, Institute of Literary Research of the Polish Academy of Sciences, Poland

Abstract

This paper presents a multicase study protocol for meta-research in Digital Humanities, prepared by Digital Methods and Practices Observatory (DiMPO) Working Group of the Digital Infrastructure for the Arts and Humanities in Europe (DARIAH). The protocol is intended to help researchers in conducting meta-research and adopting this methodology for different purposes, disciplines and approaches. As many of the issues raised here are already covered in manuals for social research, our focus is the specificity of meta-research in the DH environment. The specificity of DH in this respect relies on an intrinsic challenge of bringing together generally undertheorised approaches of the humanities with very formal and process-driven ICT approaches.

The main assumption behind this research is that a meaningful change in scholarly practices is taking place and is worth investigating. Moreover, a full assessment of this transformation should not focus exclusively on pioneering research, but rather on the selective uptake of digital practices and methods by researchers in the humanities: those who do not necessarily affiliate themselves with DH but simply use digital tools to explore particular problems. Hence, this paper should be of interest not only to researchers willing to conduct meta analysis, but to all DH practitioners willing to gain critical perspective on their work, as well as for those working on funding, evaluation and research policy.

Three pilot studies are discussed in this paper, as they served as a basis for the protocol. They focused on different “units of inquiry” (individual researchers, projects, research communities) and varied in methodological directions: (a) individual interviews with Polish DH researchers; (b) mixed-methods analysis of digital practice in E-CURATORS, a multicase SSHRC Insight project focusing on archaeological research sites or projects, integrating individual interviews, document analysis and naturalistic observation; and, (c) group interviews with historians and literary scholars conducted within the framework of NEP4DISSENT COST Action. The resulting protocol is discussed in detail and some directions for further research are suggested.

Humanities work is changing.

A historian in Northern Europe uses Facebook, Youtube and other social media to study memory of dissent. She sees social media as a valuable resource for the study of memory shaped by public access and personal interactions.

A memory studies scholar from Bulgaria admits that the starting point for her research is often a term written into a Google search tab, which later develops as a ‘snowball’ helping to find essential websites for the study of cultural memory.

A historian of the early modern Ottoman court re-purposes a popular note-taking software to create a linked data system hierarchizing all of her archival notes. In her efforts to track the contact zones and cultural intersections of her subjects, she creates a valuable rich, structured dataset.

A media studies researcher who has spent over two years compiling a database of more than 9,000 articles on the Spanish Civil War in the Norwegian press, entirely manually coded, asks what text mining tools might help him further explore his materials. He also economises his time in the archive through the use of a tablet device, but keeps both microfilm and microfiche readers in his office to access some of his sources.

An archaeologist collaborates with first nations communities in the Arctic to record archaeological sites using smartphones and uses drone photography with digital 3D panoramas to recreate indigenous landscapes.

These examples, selected out of many stemming from our meta-research into digital practices in the humanities (i.e. the study of research carried out with the help of digital technologies), illustrate how technology is penetrating unevenly into different stages of the research process: the discovery phase (search & access to research data); gathering activities (data collection); conceptual work and organization of one’s research assets (data curation), and computer-assisted analysis of the research assets (data analysis). Invariably, these examples also problematise the institutional, custodial, and disciplinary nature of humanities research conducted within the confines of the physical archive, the geographically bounded community of scholars, and the materially constrained toolset of a single discipline. If scholarly research often now appears to transcend locality and place, methodological approaches to studying how scholars work should also adapt accordingly.
As we come to reflect on how humanities research is shaped today by the physical and material factors that condition the work of researchers, as well as the new capabilities that the digital ecosystem affords, we see that there are micro-practices — visible in those examples described above — that fall outside both the traditional humanistic methods and what is usually considered Digital Humanities (DH). We understand DH in the broadest possible sense, as the “big tent” [Svensson 2012] of digital practices, methods, tools and resources which inform various disciplines in the humanities. Moreover, we do not limit its scope to the core DH community of researchers advancing cutting-edge methodologies, but also look at scholars who would never call themselves digital humanists yet do take advantage of the digital to answer their non-digital research questions. “Big tent” DH recognizes that the importance of the digital is not limited to the use of computational, algorithmic methods for research, but has to do with the broader shift of the whole paradigm of humanities proper: the centrality of digital resources and access to them for all kinds of humanities work; changes in scholarly communication, peer review and publication where digital and online systems are central, inter- and cross-disciplinarity; as well as working with open content and open data, both digitised and born digital.
In this way, we can see how research practices are fundamentally unpredictable: they do not fall under the umbrella of a single custodial environment but are rather assembled as a local response to a single project’s needs. To assemble them involves repeated acts of bricolage, tinkering with moving parts that were not designed to work together. Nowhere is the impact of this status more present than in the situatedness of humanities research, that is in the (self-) management of its physical, virtual and mental spaces. SPARKLE (Scholarly Primitives And Renewed Knowledge-Led Exchanges), one of the studies we reference below, showed in particular the degree to which location and spatial configuration were felt by the researchers interviewed to be integral to the productivity of the humanistic scholar [Edmond et al. 2016]. We are perhaps used to this aspect of, for example, archive-based work, due to the external constraints of opening hours, reading rooms, and all the rest of what Arlette Farge refers to as their “absurd rules of operation” [Farge 2013, 52]. But what perhaps has its roots in a time-management challenge progresses into what manifests as a highly individualised and expertly tuned set of spatial strategies. Even in the relatively small SPARKLE sample, there was essentially no agreement, of where the ‘best place’ to work might be. In part this was a reflection of adaptation to different life circumstances: senior academics with teaching and administrative responsibilities seemed generally to find their offices more rife with distraction than their home spaces, for example. It also sometimes indicated a separation, or multiple separations, between administrative work, personal life and research work. But even these general rules had exceptions, where the desire for work-life balance created yet further variation (libraries or coffee shops, for example, or working in the office but out of business hours, on weekends or in the evening).
The nature of humanities research as being physically unbounded, or perhaps, more precisely, loosely and alternatingly bound to multiple places (the office, the archive, the classroom, indeed even the virtual environment or page of writing etc.) places it in stark contrast to laboratory science, in which the nature of the work space is very specific and determined, forming (at its most extreme extension) a guarantor of scientific credibility, that is a “clean” or “controlled” environment. The contrast between the closed physical space of the laboratory and nomadic nature of the humanistic knowledge-creation process, with its constant refreshing of sources, backdrops and inspirations, is reminiscent of the contrast between farmed agriculture and the hunter gatherer, with the pre-agrarian mode of subsistence being perhaps more dependent on shifting availability of resources in a wider space, but benefitting instead from a more multisensory, kinetic and synthetic mode of cultivation.
The idea that objects should be organised in physical and mental space also interconnects with another theme that appeared across our study, which is that knowledge creation is an embodied process, encompassing far more than just the brain and its focussed sense of sight as an organ for processing source inputs. This is no less true in virtual than in physical spaces. Separation (of tasks, of ideas) and integration (of source material, with a community) are constantly in balance in the epistemic process, to the extent that one interviewee stressed the importance of technology for him in making it possible to access his entire personal source library from his computer at all times. These mental and physical environments, as well as the broader sensory world, of those engaged in historical humanistic research leads to an enduring dominance of multimodal personal research environments. In spite of the rise of digital humanities (which some, but not all, of the interviewees had embraced), historians still prefer to have notes, notebooks, pencils, post-its, books, printouts, and all manner of stimuli around them, in addition to their laptops and digital tools. This set of hybrid practices, spaces and tools are therefore customised to the questions and the knowledge base of the researcher(s) using them, and yet must also be flexible enough to accommodate research teams that evolve over time, software that is continually versioning up, and workflows that must be reinvented with every new experiment, even as they expand to incorporate new technological affordances. How can we study this kind of subtle and evolving phenomenon?
This paper proposes an approach to qualitative meta-research into contemporary scholarship which enables us to track and understand novel research practices and assess their impact on scholarly activity as a whole. Our approach is based on several pilot studies, and our goal has been to use these studies as “ground truth” to support a theoretically-informed, highly original methodological framework for qualitative research on DH practice: a self-reflexive approach to DH practice which is often overlooked in existing empirical studies. The primary audience of this paper, given its methodological scope, are DH researchers willing to either conduct meta-research, or to critically evaluate their own work. We believe that this kind of research should be of interest to researchers willing to understand their own method and be empowered to be critical about their work (in particular people embarking on large-scale collaborative projects). Additionally, anyone interested in interdisciplinarity should find much to think about here as well. Beyond the academic, anyone looking to fund, evaluate, promote or integrate DH (whether as a Dean or funder or research manager or policy maker) could make use of this perspective.
The rationale for meta-research in DH, viewed as an integrative framework, could be explained in four dimensions: ontology (how do we conceptualize the domain of research), epistemology (what is the knowledge we are seeking to produce), axiology (what values shape the object and process of our research, i.e., the infamous “so what” question), and methodology (research design and method selection to address these issues).
At the ontological level we conceive of DH research as a practice based on symbolic interactions involving purposeful action by researchers. Hence DH is treated not merely as empirical reality (accessible only through empirical study) but also as consisting of such elements as motivations and norms of researchers, or goals and objectives served by their research choices, or the criteria, often implicit, of what constitutes “good research.” These ontological commitments bring our work close to the model of purposeful human activity established by cultural-historical activity theory [Leont’ev 1978] [Engeström 2000]. Figure 1 offers an informal model representing this ontological view.
As for epistemology, our approach relies on access to diverse evidence drawn from interviews with actors, but also from examination of documents, records and other traces. At the same time, building from the ontological assumptions, our protocol does focus on understanding what motivates and what shapes DH researchers' work, i.e. it seeks to prioritize objects of our study from the viewpoint of research participants.
As for the axiological level, we advocate for DH researchers (not just meta-researchers) to engage in what Pierre Bourdieu calls “epistemic reflexivity” [Bourdieu and Wacquant 1992]. Central to Bourdieu's idea is the notion of “two steps back” in research: the first step back is to look at the research field itself. In other words, we “zoom out,” to use Nicolini's (2010) different formulation, to view more clearly the main entities and processes-relations in the field of study. The second step back is to look from the outside at the approach, methods, and instruments of research (i.e., for a DH researcher, to consider the “epistemic framework” or commitments of their own research, or the research of others DHers). This kind of “double objectification” is what we suggest through our call for meta-research.
Finally, our methodological plane coheres with the commitments of the study detailed above. We employ our multi-case studies meta-method within a critical realist grounded theory, i.e. we neither adopt the notion that we start tabula rasa seeking to "hear" the data speak to us, nor do we believe that it is enough to just start with hunches and “sensitizing concepts” and take it from there.
In some ways our meta-research on DH can be equated to the kind of methodological work that goes on in any community of practice (bioinformatics, for example), or indeed is akin to the work of Science and Technology Studies (STS) and the Sociology of Scientific Knowledge (SSK), but there are a few key differences. Primary among them would be the gap in epistemic vocabulary and different levels of methodological coherence between the diverse fields we examine. Unlike the relative coherence of work within specific sciences that was noted in STS/SSK studies, e.g. the existence of coherent epistemic cultures in molecular biology and in high energy physics identified by Knorr-Cetina (1999), humanities disciplines tend to be less coherent methodologically, especially when contrasted with information and computer sciences. In other words, humanities are generally undertheorised in terms of their methods and epistemic strategies, while software development seems at times perhaps overly process-driven in this respect. There is a distinct challenge in bringing these approaches together. In addition, the starting point for this work has not been entirely disinterested: this is an investigation of DH, but also performed — to some extent — by those trained primarily as (digital) humanists, not as social scientists. Because of this, even when we use social science tools and instruments, we may use them differently, drawing on the strengths of what we were taught to do (read texts, looks for patterns, use a peripheral view, be sensitive to subtle, trace indicators of bias or habitus, be unwilling to accept that language use is neutral, focus on cultural norms, values etc). The resulting positioning of our approach to understanding how DH research works, brings certain restrictions on its scalability or universality. While it would be possible for observers of other inter-disciplines and methodological communities to take inspiration from the manner in which we have leveraged our perspective on the work of those in our immediate environment, the specificities of how that would optimally be deployed could be quite different and lie beyond the scope of this paper. That said, our approach would be quite extensible to most DH research and researchers seeking a framework from which to observe their fields.
In what follows, we present a review of previous studies on digital practices in the humanities, before moving on to discuss the steps taken by the Digital Methods and Practices Observatory (DiMPO) Working Group of the Digital Infrastructure for the Arts and Humanities in Europe (DARIAH) to develop a protocol for DH meta-research. Included in that section is a critical overview of the notion of meta-research, as it applies to the DH field, and an account of earlier DiMPO work conducted with a meta-research orientation. The pilot studies that follow point to promising directions in the area of DH meta-research and lay the groundwork for the development of our protocol which is presented in the penultimate section of the article. The DiMPO research protocol for DH meta-research consists of a set of specific methodological guidelines for researchers wishing to study research practices in the humanities. It includes: (a) the proposed research workflow (summarized in Figure 3); (b) a conceptual model accounting for both external and internal dimensions of DH activity; (c) a core code system (taxonomy of descriptors) amenable to the qualitative data analysis (QDA), which could be expressed in XML as a SKOS structure if necessary; (d) a qualitative interviewing and document analysis codebook identifying themes for eliciting evidence in situ (elaborated in Table 1); (e) an exemplary case study structure to shape analysis and writing-up. The final section sets forth the next steps for the protocol and the DiMPO working group.
We believe that broader adoption of this protocol by the research community could form the basis for the improved exchange of qualitative research results in this area, and a better understanding of DH and its penetration beyond the scope of those already committed to these methods. In establishing the protocol we hope to illustrate diversity and contextual dependency across different disciplines, schools of thought, typologies of research project and team, and individual practices. Taken together, the article and protocol map out an approach to meta-research of digital practices in the humanities which goes much further than the state of the art, in that it asks us to consider why and how we can comprehensively chart the impact of the digital on our knowledge production practice.

Earlier Studies on Digital Practices in the Humanities

Considerable international research has been conducted on digital scholarship, digital research infrastructures, the organizational and disciplinary structures enabling digital work in the field of humanities research, and the information seeking and use patterns of researchers in the human sciences (for an overview, see Borgman 2007, Case and Given 2016, 287–96, Schreibman et al., 2016). Within information science, researchers in the subfield of information behaviour have been trying to understand research practice as far back as the 1950s. While this work was initially limited to information seeking – how scholars sought information relevant to their research – it gradually expanded to encompass processes of information management, curation, collaboration and communication [Julien and Duggan 2000]. Information behaviour studies addressed questions of computer use, comparison between the humanities and other disciplines, resource discovery, primary and secondary sources, and finding aids and archival services used by humanities researchers [Dalton and Charnigo 2004] [Delgadillo and Lynch 1999] [Duff and Johnson 2002] [Stone 1982].
One strand of earlier information behaviour research in particular remains relevant today: the abstraction of processes of scholarly information work, such as chaining, browsing and extracting, as described by Ellis in his comparative study across the sciences, social sciences, and the humanities (1993). This comparative approach was enriched with further processes [Meho and Tibbo 2003], and later converged with the notion of scholarly primitives introduced by John Unsworth (2000). Carole Palmer and her colleagues further developed the notion of scholarly primitives into a fully-fledged classification of twenty granular, standardised and recombinant scholarly activities, within broader processes of searching, collecting, reading, writing, and collaborating [Palmer et al. 2009]. On a parallel trajectory, the work of the Arts and Humanities Data Service (AHDS) in the 2000s took the abstraction of the research process further by establishing the complementary notion of computational methods. This led to the development of an extensive taxonomy of methods for the arts and humanities, documenting the context, needs and scenarios of use of each method in particular disciplines – from history, performing arts and archaeology to theatre and linguistics – across complementary contexts of digital data creation, processing and use [Hughes 2008] [Reimer 2009]. The AHDS methods taxonomy has been used to structure the ICT Guides database of digital arts and humanities projects in the UK [Grindley 2007a] [Grindley 2007b] (cf. 2006a, 2006b), and governed the information architecture of the arts-humanities.net portal, hosted for several years by the Centre for e-Research (CeRch, King's College London). Both the repository and the taxonomy were furthered by the DH@Oxford initiative in the UK, and the Database of Research and Projects in Ireland (DRAPIer, Digital Humanities Observatory).
The Digital Research Tools (DiRT) directory originally appeared under the auspices of the Bamboo project [Loesch 2013], and created a different classification of digital tools for humanities research which, implicitly, reflected an underlying categorization of scholarly research activities. An initiative within DARIAH-DE (the German tier of the DARIAH infrastructure) drew from such earlier work in the pragmatic context of building a tagging scheme for a Zotero bibliography on “Doing Digital Humanities”[1] to develop an updated Taxonomy of Digital Research Activities in the Humanities (TaDiRAH) in collaboration with the DiRT directory [Borek et al. 2016] [Perkins et al. 2014]. Echoing Ellis and Unsworth, DH researchers involved in the development of methods taxonomies turned to modular, standardized procedures underlying humanities research enabled by ICT, and usable in hugely diverse contexts. One driving idea was to advocate for the development of “a system that would span disciplines and facilitate knowledge transfer around digital methods, could help prevent the re-inventing of the wheel, encourage re-use of resources and contribute to a greater awareness of the importance of digital research” [Reimer 2009].
In spite of all of this work, however, our knowledge of the information behaviour of humanities researchers in the digital environment remains incomplete, largely due to the fact that this remains a diverse field and has been a moving target, especially as the tools of scholarship have changed radically with the advent of ICT. The complexity and fluidity of the research environment calls for flexible methodologies, capable of capturing the nuances and accounting for the variety of research practices. Hence, the emergence of qualitative inquiry approaches towards DH, which tend to investigate digital practices in a broader epistemic and organizational context, very much in-line with the methodologies presented in the following sections (e.g., Antonijević 2015).
There are a number of recent studies which either directly or indirectly address digital research practices in the humanities. An important subset of this research has been conducted in the context of defining user requirements for digital research infrastructures and services for the arts and humanities. Questionnaire survey and qualitative interviewing research in the context of the Preparing DARIAH and European Holocaust Research Infrastructure project [Benardou et al. 2010] highlighted, among other findings: the persistent use of traditional, non-digital formats to access textual archival resources and books; the high importance attributed to collecting and managing references, as well as to storing both digital and paper copies of both published and unpublished materials; the perceived value of highlighting relevant text passages and storing notes with them; and the importance of named entities for content-based retrieval of primary and secondary sources. Further analysis of interviews under the same project formally corroborated a number of important intuitions, such as a widespread tendency of researchers to use primary data and secondary sources at the same time, or to forge links between objects on the basis of their conceptual content [Benardou et al. 2013]. The Scholarly Research Activity Model, resulting from this work, drew from mixed methods research on the scholarly practices of researchers in the DARIAH and EHRI communities and was used to model, represent and analyse the findings of such analysis [Benardou et al. 2010]. Some recent work within the DARIAH Community Engagement Working Group (CE-DARIAH) continues to analyse the work practices of humanities researchers and how these researchers might be encouraged to better engage with Research Infrastructures (RIs) such as DARIAH [Garnett and Papaki 2019].
Likewise, numerous projects have tackled research on DH from different perspectives. The CENDARI Project (2012-2016) was funded by the European Union’s Seventh Framework Programme to pilot the implementation of a virtual research infrastructure for scholars of medieval and modern history. A part of the project dedicated to the Domain Use Cases aimed to capture the different research practices in transnational history, challenges posed by the fragmentation of archival sources, and how a virtual research infrastructure can help address these challenges (CENDARI WP4 team, 2013). The DESIR project (INFRADEV-03-2016-2017; 2017-2019), created a very informative series of video interviews with DH researchers, focusing in part on their own career paths.[2] The KPLEX project (2017-2018), funded under the European Commission’s Horizon 2020 research programme, conducted surveys and interviews on how big data research might be better informed by humanities research practices. The most relevant pieces of work for DH meta-research are the interviews of WP2, which looked at computer scientists’ attitudes toward the definitions of certain key terms in their work (especially “data”), provenance and data cleaning as productive or destructive processes, and the role of uncertainty in humanities research datasets. The protocol used, background information and analysis of the interviews were published as a report.[3] The PROVIDEDH Project (CHIST-ERA, 2017-2020) prepared User Stories and Scenarios which present eight cases of how uncertainty arises and is managed in early modern historical research, drawn from interviews with four expert researchers in the field. Finally, The Scholarly Primitives and Renewed Knowledge-Led Exchanges Project [Edmond et al. 2016] was inspired by ethnographic methods, in looking at the practices of knowledge creation in the humanities. Nine interviews were carried out with the aim of pinpointing the key moments and milestones of this process [Edmond 2018]. What we can extract from this distillation of almost three decades of research in digitally-enabled humanities scholarship is that any broad framework for study has to be expansive and customizable for a field-specific study. It needs to account for the specificity of the sources, the research team and funding structure, versioning software — in short, the radical instability of the rapidly changing research environment as well as knowledge production itself. Even a cursory examination of how researchers work with digital technology reveals that the application of specific research methods is dependent on the researcher’s ability to use particular digital tools and services (as well as on the stage in research lifecycle, the disciplinary context, etc.). Those studies form the background for the investigation on the scholarly practices, digital needs and attitudes of European researchers in the human sciences which is the focus of the present discussion, and of the meta-research approach that has been developed within DARIAH DiMPO Working Group.

Towards a Protocol for DH Meta-Research

The number of studies presented in the previous section brings to the fore how complexity is inherent not only in the human cultural record that is the object of humanistic inquiry but also in the very practices that constitute scholarly work. In the last five years, the work of DiMPO has aimed at studying the purposeful activity of humanities researchers as they engage with everyday processes of scholarly work and to document and account for the role of the digital in their workflows. Its first research endeavour was the European survey on scholarly practices and digital needs in the arts and humanities, disseminated in ten languages during winter 2014/2015, which attracted 2,177 respondents from sixteen European countries. The survey aimed at providing “an evidence-based outlook of scholarly practices, needs and attitudes of European humanities researchers towards digital resources, methods and tools across space and time” [Dallas et al. 2017].
The DiMPO survey results depicted a landscape in flux, with rapid change in some places and contexts, and surprising stagnation in others. To capture this changing landscape, the survey was designed as a longitudinal research project, to be repeated every few years. However, while a survey is a good instrument in identifying “what” questions through quantitative summaries and statistical associations between different variables, it provides little context to help elucidate holistically the “how” and “why” of DH research. For this reason, DiMPO agreed that a process of qualitative research is also necessary to allow a fuller understanding of how humanities researchers work while engaging increasingly with digital resources, tools and methods. Such a qualitative approach would complement the quantitative while allowing for the interpretations to the multiple intentionalities, disciplinary affiliations and contextual dependencies of particular developments in the field.
Before the next iteration of the survey, DiMPO initially adopted a case-study approach, inspired by a multicase study model [Stake 2013], for the following reasons:
  1. it would provide an insight into researcher’s motivations and their reasons for choosing (or refusing) certain digital methods while providing a perspective on the epistemological influence of tools on the research process;
  2. it would be better suited to capture and understand the moment of digital transition in the humanities, especially how earlier methodological or theoretical assumptions are negotiated when adopting digital methods;
  3. it might provide some hints on the evaluation of research work and new methods in DH;
  4. it might shed more light on who are digital humanists, or, more precisely, what it takes for researchers to consider themselves digital humanists;
  5. it accounts for the holistic, contextual and situational dimensions of human culture
Discussions of the rationale and priorities that should govern research on DH research practice led DiMPO to consider a broader epistemic framework for its activities, based on the notion of meta-research. Stemming from concerns on reproducibility and waste in the biomedical field, meta-research emerged recently as an important orientation to the study of research practice. While the term was originally used as a mere synonym to meta-analysis, i.e. the consolidation of aggregated findings of published studies such as different clinical trials on the same research hypothesis, the emerging new field of meta-research now seeks to find evidence-based improvements to scientific work through “research on research” [Wikipedia 2018] by looking more broadly at “thematic areas of methods, reporting, reproducibility, evaluation, and incentives,” combining theoretical and empirical investigation, and extending beyond retrospective studies to planned, prospective inquiry on scientific practice [Enserink 2018] [Ioannidis et al. 2015]. DiMPO shares the pragmatic goal of meta-research, as it has been adopted in the natural and hard sciences, to make a positive difference in the way researchers work. This commitment underlies the mixed methods research it adopted, consisting of a combination of longitudinal questionnaire survey, literature review, and qualitative multicase study investigations. The scope of DiMPO work, driving the construction of the DiMPO protocol for case study meta-research, is therefore adjusted to studying DH scholarly work by zooming in to a granular study of particular research activities and operations and zooming out to considering broader socio-technical and cultural factors. Beside contribution to knowledge, an important objective of DH meta-research is to assess and help overcome challenges currently faced by humanities researchers in creating, curating and using digital resources, taking up and building useful digital tools and services, and developing and adopting appropriate digital research methods in practice.
During a series of working group meetings in 2017-18, DiMPO planned for harmonization between three pilot studies conducted by different group members and discussed their outcomes, paving the way for the preparation of the final version of the DiMPO protocol. These pilot studies were not designed for the purpose of this meta-research endeavour: they rather represent active, actual research on DH scholarly practices and knowledge work conducted by different co-authors of this study. These span different disciplines and fields within the “big tent” of DH, from literary studies to history and archaeology, different units for inquiry from individual researchers to research projects and whole fields, and diverse kinds of research questions from digital literacy to the effect of digital tools on research activity to the formation of research communities. Collectively, and through their complementarity and diversity, these pilot studies were selected to satisfy, through the principle of saturation typical of qualitative research, a criterion of generality for our proposed methodology.
Based on input from these pilot studies, a preliminary case study structure was produced, including a common template for outcomes and reporting, instructions on construction of each case study dossier and how it should be situated in the context of multiple case studies. This framework was elaborated according to the particular research needs of the pilot studies. Three studies, presented in brief in the next section, account for three complementary research perspectives adopted by DiMPO, focusing on a different level of “unit of inquiry” and adopting methodological directions from this orientation: (a) individual interviews with Polish DH researchers; (b) mixed-methods analysis of digital practice in archaeological research sites or projects, integrating individual interviews, document analysis and naturalistic observation, and, (c) group interviews with historians and literary scholars conducted within the framework of NEP4DISSENT COST Action. A secondary dimension, on which the pilots contribute differently, was to balance between a “grounded theory,” inductive approach based on open coding, and a theory-laden, conceptual model based approach based on practice studies, activity theory etc. Finally, the choice of case-studies was motivated by the need to gather input on digital practices in multiple disciplines (literary studies, history, archaeology), reflecting our broad understanding of the humanities and the “big-tent” characteristics of DH. In all, a total of twenty-seven researchers were included as study subjects in these pilot studies. The following section briefly contextualises the presentation of the protocol that follows and its main purpose is to give the overview of research questions, methods and possible outputs that can stem out of meta-research.

Pilot Study 1: Individual Researchers

This pilot study involved Polish humanities scholars and aimed to gain a better understanding of the DiMPO survey results with regard to how digital practices change the work of researchers in various fields, and how these researchers define and understand digital practices in the context of their needs. It was assumed that a series of interviews would provide better information on the values ​​assigned to digital technologies in the context of their use in research, to identify particularly useful existing and missing digital tools, as well as potential obstacles. Other research questions included the influence of digital tools on communication between researchers and disciplines, dissemination of results, the perception of what constitutes research data, and researchers’ assessments of the level of digital competence among their peers.
The pilot study employed a relational approach to digital literacy, which defines the competent use of the digital in terms of a contribution to one’s life quality and performance [Filiciak et al. 2013]. This relative perspective allows us to view digital competence in relation to other spheres of activity, allowing a subjective assessment of respondents’ competences with regard to their actual needs. In order to fruitfully explore the subject in the relational approach, the research team employed the technique of episodic interviews [Flick 1997], as it encourages the exploration of both semantic and episodic dimensions knowledge. The former entails internalised social knowledge, whereas the latter expresses its particular expressions based on specific actions. The method provided insights not only into interviewees’ assumptions on digital research technologies and values ascribed to them, but also into particular manifestations of these assumptions and values in research.
The interview questionnaire consisted of three key sections, each containing up to 9 questions. Section one, aimed at eliciting semantic knowledge, included general questions about the usefulness of digital technologies in research, along with introductory questions identifying the interviewee’s main research interests, career focus and current work situation. Section two concentrated on the episodic knowledge, which regards to the following dimensions: communication within the scientific community, disseminating research results, doing research, academic teaching. The interviewees were very willing to share their experiences with particular devices and software with regard to those dimensions. Section three was dedicated to consolidating and juxtaposing both dimensions of knowledge. The questions focused on the interviewees assessment of their individual digital skills as well as the skills of their scholarly community. This section also assessed the perceived effectiveness of using digital technologies in research in the context of the non-digital practices.
There were five interviewees in total. The subject group consisted of one female and four male researchers, representing different career stages (three professors and two PhDs) and disciplines (two historians and three literary scholars). Two interviewees were leading grant projects at the time of the interview. Three interviews were conducted face-to-face in Warsaw, and two remotely via Skype. Each interview lasted between 45 and 65 minutes. The interviews were recorded, live transcribed and coded in vivo by the team members, i.e. new codes were generated inductively from interviewees’ statements by two researchers separately listening to the recordings and taking notes. To decrease the bias, the material was cross-analysed by two researchers. Although this was manageable in the pilot study due to the volume of material (five recordings, 60 minutes each on average), for a full-fledged study full transcription and qualitative coding would be advised.
The study enabled investigators to answer the majority of research questions, and proved effective in eliciting both the general attitudes and experiences concerning the use of digital technology in research, and the experiences and assumptions connected to particular instances of using such technologies on the various stages of the research process. This short discussion of the study outcomes is by no means comprehensive. Two key conclusions from this research were that digital practices enable, facilitate, and in some cases demand the emergence of networks of cooperation within the academia (i.e. researchers from various fields), as well as between academia and other groups (amateur science enthusiasts for instance). Furthermore, digital technologies and practices can help bridge the gap between academic centres and peripheries (although limited access to digital technologies may have a negative impact on this process), as it facilitates cooperation between units, opens access to various resources, and offers state-of-the-art solutions to methodological or theoretical issues.
As for more detailed conclusions, interviewees reported a major qualitative and quantitative change with regard to possible research avenues, quoting perspectives enabled by big data to conduct large-scale comparative studies. Digital research is perceived as more cost- and time-efficient, facilitating the coordination of various elements of the research process. However, the interviewees highlighted the need for the conscious use of the new tools, which may give great opportunities but should always serve as a means for answering the research question, so that the mere use of tools on new material does not become a goal in itself. The writing was reported least affected by digital technologies, as standard, widely available word processors were not replaced in the studied sample by new tools. Interestingly, what was further confirmed in the community research described below, researchers incorporate into their work the non-scientific resources as, for instance, data provided by amateur genealogists or individual enthusiasts. As for the physical working space, interviewees tend to work remotely in different settings (home, university, library), using mostly their own equipment.
In sum, this pilot study informed our thinking about the DiMPO protocol in terms of introducing the methodological orientation of the episodic interview approach, and of defining the research questions and questionnaire construction, and the workflow for the individual interviews.

Pilot Study 2: Research Projects and Sites as Activity Systems

The second pilot study was conducted as part of E-CURATORS, a research project funded by an Insight grant from the Social Sciences and Humanities Research Council of Canada, seeking to advance scholarly knowledge on the growing complexity and diversity of digital research practices in archaeology in the context of increasingly mainstreamed, pervasive, and invisible digital infrastructures. The study looks at practices such as: adoption of mobile devices to capture and document excavation and survey data; use of off-the-shelf mobile apps to construct three-dimensional models of archaeological artefacts, or to geo-locate archaeological information resources; instant online aggregation of captured data and resources in research archives, databases and repositories at the time of capture; use of synchronous and asynchronous communication technologies to connect researchers with data and enable interpretation “at the trowel’s edge” [Hodder 1997]; collaborative annotation, enrichment and interpretation of archaeological data using Web 2.0 technologies, crowdsourcing and social tagging; adoption of virtual reality (VR) and augmented reality (AR) visualization methods and equipment imported from fields such as the media and gaming industries for the generation and assessment of archaeological knowledge; use of blogs, wikis and social media networks to co-create and co-curate archaeological information objects; open access provision of data outside established archaeological data infrastructures; and, the use of gamification, storytelling and social media networks for public communication, learning and mediation.
Inspired by activity theory [Allen et al. 2011] [Engeström 2000] [Leont’ev 1978] and its application to digital curation [Dallas 2007] [Kaptelinin and Nardi 2006], E-CURATORS focuses on studying archaeological research in the situated framework of bounded research processes considered as activity systems, which involve a group of individuals purposefully engaging in specific, hierarchically organized, activities prompted by multiple motivations, pointing to specific goals, and made possible through the affordances of a diversity of norms, established procedures, methods, competencies, as well as digital and conceptual mediating tools. E-CURATORS is thus construed as a multicase study on archaeological projects, organisational units performing situated archaeological work, or sites of archaeological activity. Following a mixed methods approach, for each case study, the project collects a combination of three kinds of research data: (a) audio or video recorded testimonies and stories, acquired through a combination of life history and semi-open episodic interviews, and covering historical (factual), attitudinal and normative dimensions of archaeological activity; (b) documentary evidence, including captured field data, digital photos, models, notes, recordings, logs, database entries, maps, cataloguing texts, reports, web content and publications, and (c), optionally, naturalistic observation of study subjects. While the focus is on the overall activity system constituted by interacting actors, objects and mediating tools, E-CURATORS aims to examine different levels of granularity of scholarly activity through a process of zooming-in to the specifics of digitally-mediated actions and operations, and zooming-out to the contextual factors of sociotechnical infrastructures and epistemic culture [Nicolini 2013].
E-CURATORS adopted a multi-step research activity design, based on connecting three interrelated processes: conceptual modeling of digital archaeological practice, data constitution, and analysis and theory building (these were integrated into the DiMPO protocol, which is presented in more detail in Figure 2).
Work in the E-CURATORS project involves frequent iterations between activities, as well as interaction between activities executed in tandem across processes. So far, the E-CURATORS team developed a preliminary, extensible model and “code system” for QDA, drawing from an earlier model on archaeological digital curation [Dallas 2015], and based on integration between relevant parts of the NeDiMAH Methods Ontology [Hughes et al. 2016] [Pertsas and Constantopoulos 2017], the CIDOC Conceptual Reference Model [Le Boeuf 2015], and the gIBIS practical argumentation and design rationale schema [Conklin and Begeman 1987] [Shum et al. 2006]. The model aims to represent evidence not only on (a) activities, actors, their goals and motives, archaeological entities, information resources, methods, procedures, digital devices, tools and services, but also on (b) issues (problems, questions), positions (solutions, ideas), and arguments (criteria, pros and cons) elicited from study subjects on their archaeological practice. A preliminary schematic representation (Figure 1) identifies the functional structure of digital archaeological practice through three overlapping facets, or views: (a) intentionality, relating activities with research actors, beliefs they hold, collectivities they belong to and shared drivers such as motives, goals and norms; (b) process, relating activities with procedures guided by drivers, tools implementing them, and information objects involved in them; and, (c) knowledge work, relating activities with archaeological entities, information objects, and beliefs they support or challenge (Figure 1). The code system consists of extensible code stub hierarchies for all entities in the model, and the relationships between them.
Figure 1. 
E-CURATORS schematic model of archaeological digital research practice based on activity theory [Leont’ev 1978] [Engeström 1999] and scholarly activity models (SRAM) [Benardou et al. 2010]. The model represents archaeological activity though actions such as surveying, recording, and classifying, performed by a diversity of actors (e.g., excavators, finds specialists), who hold, establish or adopt claims (positions, arguments) on archaeological questions through knowledge interaction with archaeological entities (artefacts, ecofacts, sites, ancient people, events, cultural meanings etc.) as data and information resources accessed by means of diverse (physical, digital, conceptual) mediating tools (methods, procedures, workflows, recording devices, and analytical software).
Fieldwork on the first case study, the Archaeology Data Service (University of York, UK) and its Internet Archaeology journal, was completed in December 2018. Interviews conducted with seven key members of the ADS team are currently being transcribed, coded and annotated using the MAXQDA computer-assisted qualitative data analysis software (CAQDAS) [Woolf and Silver 2017]. In tandem, a Neo4J property graph database [Vukotic 2015] is being developed to host representations of research data (coded transcripts of interviews, annotated and categorised documentary data), so that they can be visualised and further analyzed through graph browsing, filtering, query, summarization and synthetic views of main activities and ideas expressed by study participants, and related motives, goals, norms and competencies, as well as digital infrastructures and tools, and archaeological entities related to research practices.
An initial evaluation of results from this single-sited case study demonstrates themes and underlying factors, such as shared norms and metaphors about digital preservation work, the nature of the archaeological record, the role of digital repositories such as ADS, and the value of specific knowledges and capabilities, across study participants. Positional roles and relationships between researchers, trans- and interdisciplinary interactions and “translations”, emerged from interviews with different team members and documentary evidence they pointed us to, and helps build an account of digitally-enabled archaeological activity at ADS that would be impossible without attention to the collective work of study participants within multi-year projects, and to the set of arrangements, presuppositions, methods and norms shared by the ADS team. Individual competencies and orientations play nevertheless as important a role as the broader context of “client” organisations and people in British archaeology, peer organisations and networks active in archaeological data archiving in the European or global context, and, last but not least, a transdisciplinary ecosystem of diverse platforms, research organisations in computer science and informatics, policy makers and funders whose activities affect, directly or indirectly, the work of the organisation. Sensitizing concepts such as objectual practice [Knorr-Cetina 2001], thought collective [Fleck 1979], and semiosphere [Lotman 2005] seem to be promising in providing fruitful theoretical insights on this study.
The ADS case study raises particular questions regarding the locatedness of archaeological curation practice. ADS activity is predicated upon localised primary archaeological fieldwork, but, by virtue of how communications between archivists and “client” archaeologists unfold, and how information objects (such as datasets, and archaeological reports) are produced and curated, the ADS work itself detached from the location where it takes place: ADS staff work mostly in front of computer screens, using information systems and services accessible via digital networks, and interact with stakeholders over the phone, email and other digitally-enabled means of communication. Viewed as an archaeological semiosphere [Laužikas et al. 2018], the field where ADS research, curation and data management work is enacted may be conceived as a (conceptual) landscape punctuated by diverse disciplinary traditions, data management methods and rules-of-thumb, norms of best practice and acceptance criteria, competencies and skills, systems and services, and data standards, among other things. While place is not one of the entities represented explicitly in the E-CURATORS schematic model (fig. 1), a topological structure is implicit in its overarching conceptual organisation. Besides, the Neo4J graph database was selected to represent E-CURATORS data exactly because of is ability to allow visualisation, navigation and traversal of topological structures representing curation practices such as found at ADS. Aspects relevant to the ADS case include the emergence of practices of creolisation and translation emerging at the boundary between semiospheres of scholarly archaeology and data management. Other cases of archaeological practice investigated by E-CURATORS, related to digitally-enabled archaeological excavation and survey, may call attention to the boundary between material-tangible and digital-intangible archaeological work, and the processes of creolisation and translation between competing norms, methods and regimes of value among these two semiospheres.
The E-CURATORS pilot is special in being part of a funded project, with an international team of senior collaborators and a dedicated team of paid research assistants, hence some of its characteristics are not transferable to the mostly unfunded, in-kind contribution model under which DiMPO operates. From E-CURATORS, our DiMPO protocol inherits the appreciation of, firstly, the methodological value of zooming-in/zooming-out between different levels of DH research activity, of attention to interactions between different actors in the context of a situated research process - which may be identified as a research site, an organisational unit, or a project - and between research actors and digital resources, tools and methods. Secondly, of attention to the issues, positions and arguments digital humanists identify as they consider challenges and opportunities, and obstacles and drivers to purposeful DH research activity, as well as the importance of norms, motivations and shared orientations within different kinds of “thought collectives,” especially as DH changes through interdisciplinarity and osmosis with pervasive digital infrastructures, diverse stakeholders, and bottom-up practices “in the wild” [Dallas 2016]. And, thirdly, of the shifts introduced by digital tools, services and infrastructures to the spatial enframing of scholarly work, when activities, methods, thought collectives, norms and other elements of scholarly practice collectively form a conceptual rather than physical landscape, a topological structure amenable to traversal and navigation as researchers cross disciplinary and methodological boundaries.

Pilot Study 3: Research Community Profiles

This pilot study features our engagement with “New Exploratory Phrase in Research on East European Cultures of Dissent” (NEP4DISSENT),[4] a research network of more than two hundred researchers from thirty-seven European countries, funded by the COST Association as a 4-year Action. The Action includes a diverse group of scholars, including historians and art historians, anthropologists and scholars of media studies, librarians/archivists and curators, information technologists and digital humanists.[5] One of the main goals of NEP4DISSENT is to introduce DH methods to scholars who are already immersed in digital materials and tools but not yet self-consciously carrying out DH projects. Interviewees were selected who seemed “DH-curious” and were open to learning new methodologies, but for the most part not yet fluent in DH terminology and major approaches. Therefore, the group interviews conducted at this early stage in the project (March 2018) served several purposes at once: (a) to measure the degree of familiarity with DH tools within the research group, as well as probing attitudes and beliefs about digital tools for research; (b) to provide a snapshot of scholars’ current practices and methods for collecting and curating their data (in some cases born-digital but in most cases analogue); and, (c) to identify the key needs of this group, in terms of areas for possible DH collaboration and training in DH methods within the framework of the Action.
The interview protocol was developed by DiMPO members who are also taking part in NEP4DISSENT under Working Group 5 (“Mediating Research Through Technology”), and based on the paradigm developed earlier for the SPARKLE project (discussed in section two above). The eleven selected interviewees, seven male and four female, represented five disciplines: history (6), art history (2), media studies (1), sociology (1), and tourism studies (1). They were interviewed in groups of two to four, over videoconference, with at least two interviewers taking part in each session. At times the subjects interacted with each other during the interviews, but for the most part, the dialogue was one-to-one between interviewer and interviewee. The interviews were coded in vivo by the interviewers and later distilled into a table reflecting key aspects of the study: sources, methods, tools, privacy, publication, barriers, and examples.
The question of barriers to digital research revolved largely around issues of access (digitization priorities of national archives and libraries) and funding, with one respondent pointing to a cultural issue in the region which scholars are sometimes averse to: open-access. Another issue that came up repeatedly was the difficulty of creating datasets that could be easily integrated with those maintained by other scholars – sometimes because of an investment in a single platform from which it is not easy to extract data (or at least not obvious to the researchers using it how to do so). This does not necessarily point to any inherent problem with the platform, as much as to the researcher’s ability to leverage its use and value. “Working with quantitative data is very time-consuming,” a historian commented, “even though there are a lot of tools out of there, but the risk is big because you need to build up your entire data set to accommodate to one tool, and then you realize it’s not the tool you needed (after you cut and pasted for days)” (Group Interview 3, 2018). These group interviews also revealed a promising line of engagement for NEP4DISSENT to introduce more advanced text analytical methods to those already working with quantitative analysis of written materials, but unsure of how to engage with semantic-level analysis. The main reasons scholars gave for using analogue over digital methods were that: (a) their sources were too mixed, (b) that they lacked faith in digital tools, and, (c) the scale of material collected was so small that they didn’t feel they need for the specific tools discussed or mentioned by colleagues.
A more systematic analysis was performed by applying full qualitative data coding of one of the group interviews. In the context of QDA, coding is defined as a systematic classification process that helps identify themes or patterns, thus forming a basis for the subjective interpretation of a document’s content [Hsieh and Shannon 2005]. There are different software options that enable computer-supported content analysis and coding, including MaxQDA, NVivo, Dedoose, and ATLAS.ti. These tools allow investigators to import various kinds of documents, usually in .txt .rtf .doc/x .pdf formats, and to perform different actions that help the content analysis process.
For this in-depth qualitative analysis, data was gathered from the group interview with two researchers working in the area of memory studies, both members of the NEP4Dissent Working Group 4 on “Cultural memory of dissent” (Group Interview 4). A full transcription of the interview was prepared and QDA was applied to the transcribed text by using the MAXQDA software. The creation of a code system followed an inductive (data-driven) approach by reading and summarizing raw data. The inductive approach uses open coding, where specific codes are derived from data and are subsequently classified in order to create the code system, usually by categorisation aiming to reduce the number of codes by grouping them. An advantage of the inductive approach is that it does not impose any preconceived categories or theoretical perspectives to the ongoing research and allows to gather direct information from study participants, which is grounded in the actual data [Hsieh and Shannon 2005]. No in vivo coding was applied during the QDA, meaning that the process of assigning a code mainly involved paraphrasing and generalization of selected text segments rather than verbatim codes. The inductive approach provided useful insights into an analysis based on raw data and helped to identify potential areas of concern expressed by researchers during the interview. Open coding resulted in the emergence of a code system which consisted of six broader definitions (codes) and their subcodes thus presenting the main findings of the interview (see Figure 2).
Figure 2. 
Code system for the group interview analysis on digital research practices. The inductive codes indicate five main categories related to digital research as mentioned by the researchers in the interview, while subcodes indicate specific details drawn from the discussion.
The findings of the case study showed that the evaluation of one’s familiarity with digital research or the role of the digital was mainly shaped by scholars’ attitudes and beliefs, as well as by personal experience working with digital resources and digital tools: “(...) in my case there is a disconnection extent to which I work with digital sources and huge amount of ignorance that I possess about them. So I don’t have proper tools and knowledge on how to work with them. As a cultur[al] historian I see a big need to work with digital sources” (Group interview 4, 2018).
Questions about digital resources revealed that it’s not only digital publications that are important for research, but even more so online research data, with different websites and social media platforms playing a key role. The Danish researcher admits that social media is a huge incentive for her in finding new ways of studying memory: “I also haven’t studied the memory of dissent as such, but I was looking into dissenting memories and what you can do with digital sources and social networks, mainly Facebook, and also Youtube. They are places where anyone can contribute and it’s a massive resource for the study of memory” (Group interview 4, 2018). Meanwhile, the Bulgarian scholar points to the importance of thematic portals for her research, e. g. “(...) websites for sharing memories of socialism (...) is another source which is born-digital, because people go there and they write their comments and memories, but sometimes they upload photos of objects from that period (...)” (Group interview 4, 2018).
With regard to questions of digital research methods, scholarly research processes were the most extensively discussed by both scholars, and particular codes were assigned to identify specific activities being mentioned, e. g. seeking of information, acquiring and organizing research data, publishing research results, collaborating with other scholars, and preserving data. In some cases, it’s fascinating to acknowledge how the research process might be ordinary and unformalized: “(..) we are very honest web users as most other people, the way we engage with net is unprofessional acting like ordinary people with tools that are easy available” (Group interview 4, 2018). Actually, the simplicity of research activities was observed in every step of research, where research data is usually acquired by copying, pasting or by printscreens and organised by using USB stick, folder system, Powerpoint or Word document: “(...) I have a Word file where I copy-paste links that I feel I want to go back [to]. It’s like [a] bibliography, but it’s a linkography - list of links” (Group interview 4, 2018).
Finally, the key issues identified in the analysis of the group interview were related to open access, lack of digital methods literacy and the complexity of online data. And while the first two perhaps are easy to anticipate, the third issue points to an increasing need to use online data in/for research, as well as methods and tools that could deal with it. As noticed by one of the researchers in the group interview, “(...) working with social networks is incredibly tempting and definitely dangerous, because it’s so easy and the feeling is quantifiable (you can count number of likes, comments). What I saw as a challenge in this connection is a super-fluidiness of so much of it, it’s so unstable, so easy, inexpensive, so you don’t know much what is going on” (Group interview 4, 2018).
This method was also tested in an environment of scholars with a fairly long record in DH, who identified with that field, through an interview conducted with four members of the Stanford Literary Lab,[6] a group representing the field of literary studies, and consisting of scholars on different stages of their careers, with a long history of collaboration. Hence, the interview collected individual responses but also became a platform for negotiating a collective approach towards methods, tools and research strategies adopted by the group. The NEP4DISSENT Group Interview Protocol was slightly modified in this study, allowing for questions inspired by the narrative component of the episodic interviews, discussed in the first pilot study, namely about how the researchers were introduced to digital methods, and how they see the field in the time-perspective (five years ago and five years from now). It should be noted that, as in the case of previous interviews, it was clear for interviewees that the application of digital tools and methods should be always driven by the humanities’ research questions, not the other way around. Moreover, the group tend to seek novel research problems to develop new tools, rather than to use the ones they have already developed on other material. This high methodological awareness also translated into the interviewees’ self-perception as DH scholars. Actually, in their words, being a DH scholar means having “two jobs,” or writing “one and a half dissertation(s),” as they need to master two fields: literary studies and computational analysis. They also pointed out to the collaboration model of DH inquiry, which differed largely from individualistic approaches of literary studies. They were also aware of some methodological criticism of DH methods, which they dubbed ‘the rolling backlash’, but claimed it was very productive for their work as it pushed them to rethink certain aspects of their mode.
The working space is an important factor for the project workflow. Although the Lab members often live in different cities and communicate with each other via electronic channels, the actual meeting space remains crucial for debating the current status of the project and exchanging the ideas that may emerge in the “first phase” dedicated to individual work. The big table in the lab, around which the team gathers for the project meeting, is, in the words of the founding director of the Lab, “as essential a tool as the really expensive ones ... All researchers bring to this phase their interests, and even fixations. At times, there is a lot of noise. But in a few magic moments, the group becomes truly more than the sum of its parts; it ‘sees’ things that no single pair of eyes could have” [Moretti 2016, 2–3]. Hence, the physical space becomes a platform for the integration of the individual perspectives and generation of genuinely collective outputs.
To sum up, this short overview of findings, interviewing DH communities may provide more knowledge about the DH field as such, as well as about overcoming methodological obstacles through collaboration with peers and negotiating DH status in the humanities departments.
Pilot studies on research community profiles contributed to the protocol with regard to establishing the research framework, desk research on actors, conducting group interviews with researchers, as well as transcribing them, coding and analysing.The resulting interpretations will not be designed for DH research as a whole, or even digital history, but rather customized to fit the needs, skills, research environment, and general methodological approaches of our project participants.

The DiMPO Case-Study Protocol

The outcome of the pilot studies is the DiMPO Case-Study Protocol for Meta-Research in Digital Humanities (Figure 2) and accompanying guidebook (under preparation). These materials are intended to help researchers in conducting meta-research and adopting these guidelines for different purposes, disciplines and approaches. As many of the issues raised here are already covered in manuals for social research, our focus is the specificity of meta-research in the DH environment. This section discusses only the main assumption of the protocol. For a more detailed overview consult the guidebook (under preparation; see Next steps).
Figure 3. 
The DiMPO Case-Studies Protocol for meta research in the DH

Background

The protocol should serve to bridge knowledge gaps about questions or issues that are relevant, consequential and potentially impactful. The very first step is to define the main area of the case study. As digital humanities tends to be conceived as a “big tent” for a plethora of approaches and disciplines (cf. Svensson 2012), there are many potential areas for a meaningful work into contemporary research practices, be it the uptake of digital encoding by scholarly editors in literary studies, cultural analytics for art history, or databases in religious studies. The main assumption behind scholarly practices meta-research is that a meaningful change in scholarly practices takes place and is worth investigating. Moreover, as we argued in the introduction, the full assessment of this change should involve not only pioneering computational research, but rather the broader uptake of the digital practices and methods by researchers in the humanities, even if they do not self-identify as digital humanists, as long as they use digital resources, tools and methods in order to explore particular problems. Hence, the investigator should choose the most promising aspect of the field, basing on her own expertise and knowledge.
That leads us to the identification of appropriate research questions. Sometimes, research questions may take the form of well-defined hypotheses, to be confirmed or refuted by operationalizing them as relationships among properties of the data collected in the study (deduction). Alternatively, they may merely consist in identifying relevant aspects of phenomena manifested in the data, which will be later elucidated through “inference to the best explanation” among alternatives (abduction). Finally, there are cases where the research questions are little more than informal ideas and “sensitizing concepts” on potential topics of interest, in the expectation that new knowledge will arise from the grounded, bottom-up examination of the data (induction). DH meta-research questions are diverse: for example, they may be how digital resources and tools change the way research questions are formulated and scholarly investigations are conducted, what is the status digital sources in humanities research, or what are the effects of digital collaboration practices and dissemination of knowledge.
On the basis of prior literature and pilot studies, DiMPO identified a number of research questions relevant for DH meta-research, categorized within the following facets: Research Activity, Resource Assets, Tools, Researchers, Place and Time. Facets and research questions are summarised in Table 1.
Facets Questions
Research activity
  • How do researchers seek, find and organize their research assets?
  • Are researchers’ method digital and could they be?
  • Are their research questions quantifiable and could they be?
  • How do digital practices influence the work of researchers? What aspects of this work are changing the most?
  • What are barriers for researchers and could they be removed with the use of digital tools?
  • How do digital practices influence communication between researchers and disciplines?
  • How do researchers assess the level of digital competence of their environment?
  • What are the biggest problems in using digital technologies by researchers?
Research assets
  • What is perceived as research data?
  • Are your research data digital?
  • What is shared and how?
Tools
  • What digital tools do researchers use?
  • Are those tools public or private?
  • Which analog tools prevail?
  • How do digital tools influence the dissemination of research results?
Researchers
  • How is digital practice different among different schools of thought, or “thought collectives?”
  • What are the competencies researchers have, or need to have,to apply specific digital methods?
  • What are the goals served by particular digital research activities?
  • What criteria do digital humanists hold on the quality of digitally-enabled research?
Place and time
  • Is your research activity confined to a particular place or could it be conducted remotely?
  • What is your physical research environment like?
Table 1. 
Facets and questions for the qualitative meta-research in the Digital Humanities
Another integral part of this early, conceptual stage is the choice of the unit of inquiry. DiMPO protocol distinguishes between three kinds of units, depending on the types of research questions: research community, research project, individual researchers. The focus on Research Community enables a view on a particular field, discipline or community of practice, and an assessment of the role the digital in this field. It could also assess the popularity of certain tools and approaches in the community as well as its value judgments concerning the evaluation of the digital work, i.e. what is considered a good practice in the field. The Research Project/Site approach allows for a comprehensive and formal description of a particular project, research site, or bounded activity of an organizational unit in its entirety, in order to provide a situated picture and understanding of a particular research endeavour, and the interaction between people, technology and its outputs. The analysis of Individual Researchers supplements these approaches with the biographical perspective on a researcher, her individual motivations and goals, particular methods, beliefs held about the digital, and career trajectory. The pilot studies presented earlier are good examples of types of differences in results between particular approaches.

Project framework establishment

The second step is to establish the framework of the project, i.e. translate research questions into a concrete study workflow. The questions which emerged in the previous step will be now operationalised and particular methods and techniques should be chosen. In the pilots we focused on personal interviews, group interviews and observations, described in more detail in the fieldwork section below. Once the choice is made, a researcher needs to generate the study form, i.e. questionnaires, or observation protocols, relevant to research questions. Additionally an ethics review should be performed and suitable policies applied, especially in cases where there is risk and potential for harm (professional, reputational, etc.) for study participants or investigators. Some funding agencies may require rigorous ethics review, yet, for most of the cases, it involves considering such aspects in clear terms as regards risks, conflicts of interest, preservation of anonymity of study participants, right for withdrawal, obligations for data retention, conditions of access to research data, and consultation with study participants before publication. For instance, one should know beforehand how raw data will be stored, or if it is going to be published in any form. All such cases will be outlined in a consent form prepared for the study, which should be signed by subjects prior to the interview.
Depending on the particular unit of analysis chosen for the study, investigators will need to establish criteria for selection of cases, i.e. which particular individuals, research projects or communities will be approached in this study. They should consider carefully what kind of sampling technique will be relevant to gather heterogeneous material and different approaches.
Finally, the internal work environment should be organised, which entails a selection of tools for data collection, storage and processing, that includes means of recording, transcribing and annotating the communication with the actors, as well as a clear data management plan. The entire framework should be put into a single case-study manual, which can serve as a point of reference throughout the study for the principal investigator and the research team.
Once all of the above items are in place, a short pilot study should be conducted to assess the research design and materials gathered in the case-study manual. Relevant amendments should be made to the manual.

Desk research on the units of inquiry

The first phase of the actual research consists of gathering data about the unit of inquiry (individual, project, community), which will inform the interaction with actors and allow for the better understanding of their work. Desk research should include a basic case overview, like research scope (what is the field), objectives (what are the aims of research) and contribution (what are the achievements). This includes gathering relevant materials, such as publications, project descriptions, earlier interviews, developed software, etc. The outcome should be presented in a form of short bios of the actors, including descriptions of their projects and contributions, together with a bibliography of publication. Such a resource could be later used by other team members. Materials created in this stage could be later reused in the final report.

Fieldwork

The next stage consists of the actual fieldwork. Its scope depends on the types of methods chosen and the size of the team conducting the study. One of the easiest ways of acquiring material, often used in our pilot studies, is an interview by videoconference, which is an easy and inexpensive way to reach out to a wider, international base of subjects. But fieldwork may also include participant observation of particular research activities, which could entail attending project meetings and presentations, testing tools developed by actors or observing them at work. This also can entail collection of documentary evidence, such as photos of resources and workplaces, as well as gathering outputs of DH research activity. Such an approach allows for gathering knowledge about the processes which could not be noticed by subjects and not reported in interviews.

Data preparation and transcription

Fieldwork produces a diversity of raw data, which needs to be put in shape in preparation for analysis. Field notes need to be organized and annotated with the time and place where they were taken; recordings, photos and videos need to be inventoried and may also need to be processed before transcription. Given the scope of DiMPO research, audio (and optionally video) recordings should be transcribed, so as to capture fully and faithfully the verbal content of participant speech, but also salient paralinguistic aspects of speech, as well as relevant non-verbal actions and events. Finally, it includes conventions for describing gestures, pointing at objects and screen events, as these can be important features of observed scholarly practice. CAQDAS packages include useful transcription functions as a matter of course, allowing for slower playback and rewind, pausing, and insertion of timestamps. Transcribers should allow for 6-7 hours of transcription time for each hour of audio recording, or roughly ten person days to transcribe a typical set of audio recordings for each case study. Given the multiple parameters and dimensions of DH research practices to be investigated, the DiMPO protocol stipulates full transcription of audio recordings, rather than open (live) transcription of only the segments that investigators find of interest at first hand.
If anonymization has been decided on as part of the ethics review, visual material may have to be redacted, and names of individuals in text documents and transcripts replaced either with participant codes or, preferably, with pseudonyms (nicknames). But, on the whole, the DiMPO protocol recommends retaining participant identity as an important part not just of ensuring interpretability of results, but also as an ethical commitment to giving due credit to individuals, organizations and projects participating in the study for their frequently innovative practices, methods and ideas.

Analysis and interpretation

The DiMPO protocol recommends that all data collected through fieldwork, as well as in earlier bibliographic and desk research, are imported for analysis into a CAQDAS package such as MAXQDA, Atlas Ti, nVivo or Dedoose, or a free alternative. Segments of transcripts, documents and publications, as well as of visual records collected for the study, can be then annotated systematically and consistently by the use of short jottings and longer analytic memos, i.e., free text annotations capturing the insights, questions and ideas of investigators, and codes. In the context of QDA, a code is defined as “a word or short phrase that symbolically assigns a summative, salient, essence capturing, and/or evocative attribute for a portion of language-based or visual data” [Saldaña 2013, 3]. As soon as research data from preliminary research and fieldwork are ready for analysis, the protocol recommends that investigators immerse themselves in the data, and produce memos which capture initial questions, ideas and insights, as well as associations with sensitizing concepts from the scholarly literature they consider relevant.
During the first cycle of analysis, the DiMPO protocol advocates a combination of deductive (provisional) and inductive (open) coding. A hierarchically-organised provisional code scheme, including ca. 50 initial codes representing the conceptual structure of scholarly practice, will be used to encode transcript and other data segments referring to the compositional structure of DH research activities, their input and output, the roles and involvement of researchers and other stakeholders, the epistemic context of their action, their motives and goals, as well as the digital methods, procedures and tools involved in each activity. In tandem, an inductive coding approach, based on open coding of relevant segments of the qualitative data, can be applied to capture unforeseen aspects of DH work, as well as issues, ideas and arguments introduced by study participants [Miles et al. 2014] [Saldaña 2013].
The second cycle of analysis is based on theoretical coding, and is performed simultaneously with sorting, filtering, summarization and visualization of data coded according to the inductive/deductive approach described in the previous paragraph. Theoretical codes, i.e., codes that may represent possible explanatory and causal mechanisms, underlying factors and characterizations underlying DH scholarly practice, will be defined through identifying associations, relationships, axial differentiations or shared categorizations of provisional and open codes developed so far, and of the aspects of evidence represented by data segments they are connected with. Intra-case study theoretical coding will be used firstly to make sense of individual cases. This will be followed by theoretical coding across cases, aimed to capture diversity and complementarity between the practices of different DH sites or projects, rather than to achieve confirmation through repetition. Additional fieldwork, e.g., in the form of follow-up interviews to clarify or enrich understanding about some phenomenon that became visible at the stage of theory building, may be planned at this point for the purpose of saturation.

Outputs

There are three main outputs of the study: report, data, and additional publications. The genre of a report gives more space for the inclusion of larger chunks of evidence (esp. quotes from interviews) but should also be supplemented by a bibliography of the publications (prepared through desk research) and a glossary of key project concepts, i.e. its theoretical, methodological and technological terms. Ideally, the report should contain the research protocol and codebook, material created through desk research, discussion of the data collected and the interpretation. A report in the field of meta-research should tackle the questions of the role of digital resources, methods, tools, as well as services and infrastructures in the analysed case. Reports could be stored in the DiMPO Community on Zenodo,[7] or, if applicable, in the DARIAH HAL collection[8] for better visibility. Additionally, data from the project (esp. annotated transcripts) should be made public whenever it is possible, as it could enable comparative perspectives or reexamination of the material by other investigators (Directory of Open Access Repositories[9] lists academic repositories around the world).

Evaluation of the research protocol

The last step of the study is the evaluation of the research protocol, i.e. assessment of the advantages and disadvantages of the chosen research framework and methodological choices. This knowledge is extremely important, as it informs further studies in the field, and — as such — should be shared in the report.

Next Steps

The main goal of this work is to enable comparable meta-research on digital humanities, covering various disciplines. Such a body of knowledge would help in developing a self conscious — if not self critical — approach to the research conducted in the field. Moreover, it could serve as a methodological guideline for non-DH researchers willing to adopt certain methods in their disciplinary work.
The DiMPO team is now elaborating a detailed online handbook to accompany the protocol.[10] Future plans include a series of workshops, presentations and tutorials for scholars wishing to conduct such research. Furthermore, DiMPO will be conducting a series of case-studies deploying this protocol. The authors are also considering whether to open the protocol to other topics and uses within the DH community, including such specific fields as scholarly writing practices in DH, or quality assessment. Another avenue for development would be to adapt parts of the protocol for evaluating DH projects and generating detailed reports about their activities, and/or for sharpening the specification of DH development projects.
While local conditions are always the most relevant, scholarly practice increasingly includes conceptual, rather than physical, connectivity because of norms and competencies that are shared across geographic and disciplinary boundaries. These collaborative practices are further enabled by online, mainstreamed and broadly available pervasive digital infrastructures and tools (cf. Dallas 2016), which in turn produce new regimes for inter- and cross-disciplinary scholarly work. The only approach that could conceivably handle this variability must take into account research norms, environments (both physical and digital), tools, social formations, and technological limitations which set the parameters — and shape the container — in which each instance of digital research finally takes place. Therefore, we see this paper as a step towards establishing a systematic reflection on the dynamic field of digital humanities, a set of practices which is more often “defined” and “debated” than studied, and which always flows to fit the shape of the container in which it is placed.

Acknowledgements

This work was conducted within the framework of Working Group 5 of the COST Action CA16213 New Exploratory Phase in Research on East European Cultures of Dissent (NEP4DISSENT), and the Digital Methods and Practices Working Group (DiMPO) of DARIAH-EU, the Digital Research Infrastructure in the Arts and Humanities in Europe. It was also supported by the project Digital Humanities work in focus: multiple case studies of research projects across Europe, funded by DARIAH under the funding scheme for Working Group Activities 2017/2018.
Maciej Maryl’s research at Stanford University was supported by Fulbright Senior Award granted by the Polish-American Fulbright Commission.
Costis Dallas gratefully acknowledges support by (a) the E-CURATORS Insight Grant of the Social Sciences and Humanities Research Council of Canada, (b) a senior research fellowship of the Polish Institute of Advanced Studies, Polish Academy of Sciences, and (c) a short-term visiting research fellowship at the Long Room Hub, Trinity College Dublin.

Works Cited

Allen et al. 2011 Allen, D., Karanasios, S., Slavova, M., 2011. “Working with activity theory: context, technology, and information behavior”. Journal of the American Society for Information Science and Technology 62, 776–788.
Antonijević 2015 Antonijević, S., 2015. Amongst digital humanists: an ethnographic study of digital knowledge production.
Benardou et al. 2010 Benardou, A., Constantopoulos, P., Dallas, C., Gavrilis, D. 2010. “A Conceptual Model for Scholarly Research Activity,” In: Unsworth, J., Rosenbaum, H., Fisher, K. E. (Eds.), IConference 2010: The Fifth Annual IConference, Urbana-Champaign, IL: University of Illinois, 26–32. http://nora.lis.uiuc.edu/images/iConferences/2010papers_Allen-Ortiz.pdf.
Benardou et al. 2013 Benardou, A., Constantopoulos, P., Dallas, C., 2013. “An approach to analyzing working practices of research communities in the humanities”. International Journal of Humanities and Arts Computing 7, 105–127. https://doi.org/10.3366/ijhac.2013.0084
Borek et al. 2016 Borek, L., Dombrowski, Q., Perkins, J., Schöch, C., 2016. “TaDiRAH: a case study in pragmatic classification”. Digital Humanities Quarterly 10.
Borgman 2007 Borgman, C.L., 2007. Scholarship in the digital age: information, infrastructure, and the Internet. MIT Press, Cambridge, MA; London.
Bourdieu and Wacquant 1992 Bourdieu, P., Wacquant, L.J.D., 1992. An invitation to reflexive sociology, Polity Press, New York.
Case and Given 2016 Case, D.O., Given, L.M., 2016. Looking for information: a survey of research on information seeking, needs, and behavior. Emerald, Bingley, UK.
Conklin and Begeman 1987 Conklin, J., Begeman, M.L., 1987. “gIBIS: a hypertext tool for team design deliberation”, in: Proceeding of the ACM Conference on Hypertext  - HYPERTEXT ’87. Presented at the Proceeding of the ACM conference, ACM Press, Chapel Hill, North Carolina, United States, pp. 247–251. https://doi.org/10.1145/317426.317444
Dallas 2007 Dallas, C., 2007. “An agency-oriented approach to digital curation theory and practice”, in: Trant, J., Bearman, D. (Eds.), The International Cultural Heritage Informatics Meeting Proceedings. Presented at the ICHIM07: International Cultural Heritage Informatics Meeting, Archives & Museum Informatics, Toronto.
Dallas 2015 Dallas, C., 2015. “Curating archaeological knowledge in the digital continuum: from practice to infrastructure”. Open Archaeology 1, 176–207. https://doi.org/10.1515/opar-2015-0011
Dallas 2016 Dallas, C., 2016. “Digital curation beyond the ‘wild frontier’: a pragmatic approach”. Arch Sci 16, 421–457. https://doi.org/10.1007/s10502-015-9252-6
Dallas et al. 2017 Dallas, C., Chatzidiakou, N., Benardou, A., Bender, M., Berra, A., Clivaz, C., Cunningham, J., Dabek, M., Garrido, P., Gonzalez-Blanco, E., Hadalin, J., Hughes, L., Immenhauser, B., Joly, A., Kelpšienė, I., Kozak, M., Kuzman, K., Lukin, M., Marinski, I., Maryl, M., Owain, R., Papaki, E., Schneider, G., Scholger, W., Schreibman, S., Schubert, Z., Tasovac, T., Thaller, M., Wciślik, P., Werla, M., Zebec, T., 2017. “European survey on scholarly practices and digital needs in the arts and humanities - Highlights Report”. Zenodo. https://doi.org/10.5281/zenodo.260101
Dalton and Charnigo 2004 Dalton, M.S., Charnigo, L., 2004. “Historians and their information sources”. College & Research Libraries 65, 400–425. https://doi.org/10.5860/crl.65.5.400
Delgadillo and Lynch 1999 Delgadillo, R., Lynch, B.P., 1999. “Future historians: their quest for information”. College and Research Libraries 60, 245–260.
Duff and Johnson 2002 Duff, W.M., Johnson, C.A., 2002. “Accidentally found on purpose: Information-seeking behavior of historians in archives”. The Library Quarterly 72, 472–496.
Edmond 2018 Edmond, J., 2018. How Scholars Read Now: When the Signal Is the Noise. Digital Humanities Quarterly 012.
Edmond et al. 2016 Edmond, J., Bagalkot, N., O ’Connor, A., 2016. Toward a Deeper Understanding of the Scientific Method of the Humanist.
Ellis 1993 Ellis, D., 1993. “Modeling the information-seeking patterns of academic researchers: A grounded theory approach”. The Library Quarterly 63, 469–486.
Engeström 1999 Engeström, Y. 1999. “Activity Theory and Individual and Social Transformation,” In: Engeström, Y., Miettinen, R., and Punamäki-Gitai. R.-L. (Eds.), Perspectives on Activity Theory, Cambridge; New York: Cambridge University Press, 19–37.
Engeström 2000 Engeström, Y., 2000. “Activity theory as a framework for analyzing and redesigning work”. Ergonomics 43, 960–974.
Enserink 2018 Enserink, M., 2018. “Research on research”. Science 361, 1178–1179. https://doi.org/10.1126/science.361.6408.1178
Farge 2013 Farge, A., 2013. The allure of the archives. Yale University Press, New Haven.
Filiciak et al. 2013 Filiciak, M., Mazurek, P., Growiec, K., 2013. Korzystanie z mediów a podziały społeczne. Kompetencje medialne Polakóww ujęciu relacyjnym. Centrum Cyfrowe Projekt: Polska.
Fleck 1979 Fleck, L., 1979. Genesis and development of a scientific fact, Repr. 11. Aufl. ed, Sociology of Science. University of Chicago Press, Chicago.
Flick 1997 Flick, U., 1997. The Episodic Interview: Small Scale Narratives as an Approach to Relevant Experiences. LSE Methodology Institute, London.
Garnett and Papaki 2019 Garnett, V., Papaki, E. 2019. Barriers and Pathways to Community Engagement (Version 1.0). Zenodo. http://doi.org/10.5281/zenodo.3956414
Grindley 2006a Grindley, N., 2006. “What’s in the Art Historian’s Toolkit? A Methods Network Working Paper.” London: AHRC ICT Methods Network. http://www.methodsnetwork.ac.uk/redist/pdf/wkp01.pdf.
Grindley 2006b Grindley, N., 2006. “Digital Tools and Methods for Historical Research: A Methods Network Working Paper.” London: AHRC ICT Methods Network. http://www.methodsnetwork.ac.uk/redist/pdf/wkp04.pdf.
Grindley 2007a Grindley, N., 2007. “Digital Tools for Archaeology: A Methods Network Working Paper.” London: AHRC ICT Methods Network. http://www.methodsnetwork.ac.uk/redist/pdf/wkp06.pdf.
Grindley 2007b Grindley, N., 2007. “Digital Tools for Museums and Cultural Heritage: A Methods Network Working Paper.” London: AHRC ICT Methods Network. http://www.methodsnetwork.ac.uk/redist/pdf/wkp09.pdf.
Hodder 1997 Hodder, I., 1997. “‘Always momentary, fluid and flexible’: towards a reflexive excavation methodology”. Antiquity 71, 691–700.
Hsieh and Shannon 2005 Hsieh, H.-F., Shannon, S.E., 2005. “Three Approaches to Qualitative Content Analysis”. Qualitative Health Research 15, 1277–1288. https://doi.org/10.1177/1049732305276687
Hughes 2008 Hughes, L.M., 2008. “Conclusion: virtual representations of the past - new research methods, tools and communities of practice”, in: Greengrass, M., Hughes, L.M. (Eds.), The Virtual Representation of the Past. Ashgate Publishing, Ltd., London.
Hughes et al. 2016 Hughes, L.M., Constantopoulos, P., Dallas, C., 2016. “Digital methods in the humanities: understanding and describing their use across the disciplines”, in: Schreibman, S., Siemens, R., Unsworth, J.M. (Eds.), A New Companion to Digital Humanities. Wiley-Blackwell, pp. 150–170.
Ioannidis et al. 2015 Ioannidis, J.P.A., Fanelli, D., Dunne, D.D., Goodman, S.N., 2015. “Meta-research: evaluation and improvement of research methods and practices”. PLOS Biology 13, e1002264. https://doi.org/10.1371/journal.pbio.1002264
Julien and Duggan 2000 Julien, H., Duggan, L.J., 2000. “A longitudinal analysis of the information needs and uses literature”. Library and Information Science Research 22, 291–309.
Kaptelinin and Nardi 2006 Kaptelinin, V., Nardi, B.A., 2006. “Acting with technology: activity theory and interaction design, 1”. MIT Press paperback. Acting with technology. MIT Press, Cambridge, Mass. London.
Knorr-Cetina 1999 Knorr-Cetina, K., 1999. Epistemic cultures: How the sciences make knowledge. Harvard University Press.
Knorr-Cetina 2001 Knorr-Cetina, K., 2001. “Objectual practice”, in: Schatzki, T.R., Knorr-Cetina, K., von Savigny, E. (Eds.), The Practice Turn in Contemporary Theory. Routledge, London, pp. 184–197.
Laužikas et al. 2018 Laužikas, R., Dallas, C., Thomas, S., Kelpšienė, I., Huvila, I., Luengo, P., Nobre, H., Toumpouri, M., Vaitkevičius, V., 2018. “Archaeological knowledge production and global communities: boundaries and structure of the field”. Open Archaeology 4, 350–364. https://doi.org/10.1515/opar-2018-0022
Le Boeuf 2015 Le Boeuf, P., Doerr, M., Ore, C.E., Stead, S., 2015. Definition of the CIDOC Conceptual Reference Model, Version 6.2. Produced by the ICOM/CIDOC Documentation Standards Group, Continued by the CIDOC CRM Special Interest Group.
Leont’ev 1978 Leont’ev, A.N., 1978. Activity, consciousness, and personality. Prentice-Hall, Englewood Cliffs, NJ.
Loesch 2013 Loesch, M.F., 2013. “Tech services on the Web: Bamboo DiRT http://dirt.projectbamboo.org”. Technical Services Quarterly 30, 228–229.
Lotman 2005 Lotman, J., 2005. “On the semiosphere”. Σημειωτική-Sign Systems Studies 205–229.
Meho and Tibbo 2003 Meho, L.I., Tibbo, H.R., 2003. “Modeling the information-seeking behavior of social scientists: Ellis’study revisited”. Journal of the American Society for Information Science and Technology 54, 570–587.
Miles et al. 2014 Miles, M.B., Huberman, A.M., Saldaña, J., 2014. Qualitative data analysis: a methods sourcebook, Third edition. SAGE Publications, Inc, Thousand Oaks, California.
Moretti 2016 Moretti, F., 2016. “Literature, Measured, Pamphlets of the Stanford Literary Lab”. Stanford Literary Lab.
Nicolini 2010 Nicolini, D., 2010. “Zooming In and Out: Studying Practices by Switching Theoretical Lenses and Trailing Connections”, Organization Studies. https://doi.org/10.1177/0170840609349875
Nicolini 2013 Nicolini, D., 2013. Practice theory, work, and organization: an introduction, First Edition. Oxford University Press, Oxford.
Palmer et al. 2009 Palmer, C.L., Teffeau, L.C., Pirmann, C.M., OCLC Research, 2009. “Scholarly information practices in the online environment: themes from the literature and implications for library service development”. OCLC Research, Dublin, Ohio.
Perkins et al. 2014 Perkins, J., Dombrowski, Q., Borek, L., Schöch, C., 2014. “Building bridges to the future of a distributed network: From DiRT categories to TaDiRAH, a methods taxonomy for digital humanities”, in: International Conference on Dublin Core and Metadata Applications. pp. 181–183.
Pertsas and Constantopoulos 2017 Pertsas, V., Constantopoulos, P., 2017. “Scholarly Ontology: modelling scholarly practices”. Int J Digit Libr 18, 173–190. https://doi.org/10.1007/s00799-016-0169-3
Reimer 2009 Reimer, T., 2009. “Classifying the (digital) arts and humanities”, in: 2009 5th IEEE International Conference on E-Science Workshops. Presented at the 2009 5th IEEE International Conference on E-Science Workshops, pp. 197–197. https://doi.org/10.1109/ESCIW.2009.5407979
Saldaña 2013 Saldaña, J., 2013. The coding manual for qualitative researchers, 2nd ed. SAGE, Los Angeles.
Schreibman et al. 2016 Schreibman, S., Siemens, R.G., Unsworth, J., 2016. A new companion to digital humanities.
Shum et al. 2006 Shum, S.J.B., Selvin, A.M., Sierhuis, M., Conklin, J., Haley, C.B., Nuseibeh, B., 2006. “Hypermedia support for argumentation-based rationale: 15 years on from gIBIS and QOC”, in: Dutoit, A.H., McCall, R., Mistrik, I., Paech, B. (Eds.), Rationale Management in Software Engineering. Springer-Verlag/Computer Science Editorial, Berlin, pp. 111–132.
Stake 2013 Stake, R.E., 2013. Multiple case study analysis. The Guilford Press, New York.
Stone 1982 Stone, S., 1982. “Humanities scholars: information needs and uses”. Journal of Documentation 38, 292–313.
Svensson 2012 Svensson, P., 2012. “Beyond the Big Tent”, in: Gold, M.K. (Ed.), Debates in the Digital Humanities. University of Minnesota Press.
Unsworth 2000 Unsworth, J., 2000. “Scholarly Primitives: What methods do humanities researchers have in common, and how might our tools reflect this?”, in: Humanities Computing: Formal Methods, Experimental Practice Symposium, King’s College, London. Presented at the Humanities Computing: formal methods, experimental practice symposium, King’s College, London.
Vukotic 2015 Vukotic, A., 2015. Neo4j in action. Manning Publications Co, Shelter Island, NY.
Wikipedia 2018 Wikipedia, 2018. “Meta-research”. Wikipedia.
Woolf and Silver 2017 Woolf, N.H., Silver, C., 2017. Qualitative Analysis Using MAXQDA: The Five-Level QDATM Method. Routledge.