Prospects of altmetrics in demonstrating research impact

In this interview, Kim Holmberg talks about the connection between open science and scientific communication and how altmetrics could be used. Kim Holmberg is Senior Research Fellow and Head of Unit at Economic Sociology at the University of Turku, Finland. Kim obtained his doctoral degree from information studies at Åbo Akademi University in 2009.

Kim has played a central role in several projects dealing with Open Science such as a project advising EU member states on advancing Open Science and renewing research evaluation, and a project investigating societal impact of Open Science in Finland. His research interests are various current phenomena on the web in general and in social media in particular. 

ORCID: https://orcid.org/0000-0002-4185-9298

 

Since you have been actively involved in projects that advance open science both nationally and internationally, could you tell us a little about your background relation to open science?

Basically, how I have become linked with open science is precisely through research. I have studied impact assessment of research and, therefore, the impact of open science and how it is distributed via networks.

 

How is science communication related to open science, generally speaking?

Science communication is very strongly related to open science. Considering that the most important goal of open science is precisely openness, the availability of science to anyone, so if a study is funded with the taxpayers’ money, the taxpayers should have access to it. In that regard, spreading and sharing information on the study is very important. Few taxpayers know of scientific journals and regularly check what gets published in them. So, it is very important that researchers and universities also share information on new findings, new studies, and publications. And when it comes to that, social media plays a very important role today, as many people receive their daily news through it. But another thing related to availability of research is that researchers should keep in mind that they should write their studies in a way that it is possible for anyone to understand at least the findings – at least they could write the abstract in a popularizing manner. If a research article is open and available online, but no one understands what it says, the openness kind of loses its point.

 

Have there been any changes in science communication in recent years?

I mean, the availability of scientific publications has improved thanks to open science. And that has maybe led to an increase in broader discussion where scientific research and findings are also included, although occasionally it also leads to negative outcomes. Science scepticism has increased, and, in some cases, some people maybe tend to take some findings out of their context and use them to support their own opinions, in a false way. So, for better or worse, discussion on science has increased – thanks to availability. If we look at things from the perspective of the past 15 to 20 years, social media has definitely revolutionized the way science is shared and how it is discussed.

 

Open science has become more common and there are now more demands related to it, so how has this changed the way researchers operate?

I think what is also strongly related to the matter is that the demand for larger-scale impact has kind of gone hand in hand with openness. Which, in turn, has led to the fact that researchers must think more about the people to whom they write, for whom they do their research, so whether they only do it so others can read it or whether they actually do something that will make a change in the society. If the starting point is the will to make a change in the society, the society must be aware of the research and have access to it. Things like impact pathways and impact plans have forced researchers to think a bit further: what comes after the research? Before, the thinking was that they were just researching something and, once they are done with it, they move on to research something else. So, before they did not think that much about the outcome to which their research would lead, what happens after you get your findings.

 

How should science communication be in the future then? What is the direction towards which it’s headed?

I am sure it will become somewhat more diverse. Social media platforms come and go. I do not know whether X will exist next year or whether something like Mastodon will have replaced it as a platform where all the researchers and science communication are. So perhaps it is not a good idea to focus on a single platform when you think about its future and how it will be used. But I hope the discourse will remain and increase and that it would lead to a growth in people’s trust in science and researchers. For the past few years have seen something of a blow in people’s trust in science.

 

You already touched on social media, but could you still describe the role social media has today in science communication, in a broad sense?

Well, social media plays an important role indeed because people get a major share of their news through it. But at the same time, science journalism suffers from it because anyone can act as a journalist and give their opinions or assessments on any given research. Meanwhile, there is an increase in science scepticism and a decrease in trust – perhaps precisely because of social media, at least partially. But social media still plays an important role, as does science communication in general. It is perhaps somewhat dependent on the branch of science and, of course, on the target audience at whom the research is aimed, to whom it is done and written. So, some research may be basic research that is meant for other researchers to use, whereas other research has clear societal impact, in which case it is important, perhaps more important, to get the information out to as many people as possible. And social media comes handy in that.

 

You have done a lot of research on applying altmetrics in research evaluation as well as a metric of visibility. Could you tell us what altmetrics is?

Well, firstly, the word altmetrics refers to a research area which measures mentions on the internet and how research is discussed on the internet. Secondly, it refers to the data, the metrics, you can collect from the internet. So in altmetrics, you map out and investigate, say, if a study has been shared on Twitter 10,000 times and on Facebook 1,000 times, whether these numbers say something about the study’s impact or some other factors. In altmetrics, you analyse these online mentions – and not only their quantity but also the context in which a given article has been mentioned: whether it is mentioned in a positive or negative sense, whether it is recommended forward as a good article that is worthy of reading, or whatever the context of sharing is. And most altmetric research is precisely about analysing whether these online mentions say something about the impact or something else. Then this is used to, let’s say, complement more traditional methods that have been used for measuring scientific impact – number of citations, that is. For before, the view was that the more citations an article gets, the bigger a deal it is, the more important the article is. Of course, there is much discussion on how bad a metric it actually is to merely look at the number of citations.

 

Does this also have something to do with how people used to think citations are an indicator of scientific impact, but now there is a need to verify this sort of societal impact, so has altmetrics emerged as a response to that?

Yes, precisely. You see, citations are only made by other researchers, which perhaps allows one to say something about or analyse scientific impact to a degree, how a given article has impacted other researchers’ work. But citations come nowhere close to evaluating societal impact. So, in that sense, alternative metrics – online mentions on different platforms, blogs, Wikipedia, X; in fact, anywhere on the internet – can shed light on impact outside of science – be it cultural impact, impact on teaching or the economy or perhaps the environment.

 

Is there something that researchers could do themselves to advance their altmetric visibility or success?

There is actually a lot they could do, but I guess the question is more whether they should. So almost anyone can, for example, write, well, copy code on the internet and create an automatic bot on X that that tweets all the articles I have written 10,000 times a day, in which case, of course, were someone to have a look at how often my articles are discussed online, they would be surprised at the high altmetric numbers I have got. Or if you really start making clickbait titles to get yourself more clicks, I do not think that would be a good thing, because that could be considered a deceitful activity, manipulation of these numbers.

 

Is the use of altmetrics  common these days or is it just becoming more common?

Well, I would not say it is common as of yet, but people are following the situation closely. So, I am sure research funders are following what altmetrics can bring into the evaluation of science because massive amounts of money is spent on, for example, evaluation of funding applications every year. Thousands of people around the world get compensated for reading an X amount of applications. So, if you could replace even a small portion of that with altmetrics, it would lead to huge savings. It is partly because of this that the research funders and many other stakeholders as well are following the situation closely, how this is developing. But already now I would say that you can probably use altmetrics to support narratives. A researcher can bring it up themselves and demonstrate that this research, which has not necessarily gotten that many references, has nevertheless sparked lots of discussion among doctors, for example, or in some other networks. So as a narrative support, but it is also possible already now to use it for pointing out the contexts and networks in which a certain study is discussed.

 

Apart from acting as support for the narratives, what other significance could altmetrics have in future evaluation?

Well, this is precisely what we are currently researching, so we will know more about this, too, in a few years. Especially about the things for which altmetrics is not applicable, for it is important to keep in mind, and also to bring up the observations and findings whether there is something for which it is not good. But like I mentioned, it is already known that altmetrics is workable at least when it comes to identifying and analysing the networks, which enables you to, if not measure, at least verify impact to some extent. So, you can map out that a certain network has lots of teachers in it and they are discussing a certain study or article. Then, one can conclude that the study has made some kind of an impact on teaching or at least picked the interest of teachers.

 

How altmetrics can shed light on the impact of research – are there some other ways, besides the ones mentioned, in which altmetrics could indicate this impact?

Well yeah, artificial intelligence and machine learning bring an extra dimension to altmetric research and altmetrics, because they allow us to analyse more closely whether an article is mentioned in a positive or negative sense, and perhaps allows us to analyse what kind of impact the article has made. Some articles are never mentioned, but there are articles that are mentioned thousands of times which makes it very laborious for a researcher to read through all the mentions on different platforms to see how the article is discussed. But artificial intelligence would read it all in a blink of an eye and write a little summary as to what the discussion is about.

 

We already touched on this, and you told us that you’re studying the matter, but what could be an obstacle for using altmetrics, what kinds of challenges are there?

There are two challenges I already mentioned. So, if the metric becomes a goal, it is a bad metric. That is when it loses its meaning. Another thing we touched on is that it is possible to manipulate altmetrics, so it is possible to buy followers on different platforms or create automatic bots that increase the visibility of a given article. So, I find these probably the most central challenges in altmetrics. Actually, I would also add that people are different: some use social media, others do not, so were altmetrics to become mainstream in research evaluation, it would create inequality between researchers. Those who like social media and actively take part in discussions would probably get more visibility through that, and if visibility is valued, they are a step ahead of other researchers who do not want to step out of their researcher circles and discuss their research online.

 

In altmetrics, how do you identify that something has been mentioned? What is needed?

Most altmetrics uses persistent identifiers, such as DOIs, PubMed IDs or arXiv IDs to identify that a certain article or study is being discussed. Links on and to science publishers’ websites are also used to identify articles. What is used less is the kind of text analysis where mentions are identified within texts. But maybe it will see a growth, once artificial intelligence becomes more common and easier to use and eventually does the searching.

 

How could researchers start tracking the impact of their own science communication?

Well, this is where altmetrics can introduce some new tools, in that you can track which research of yours is being discussed and how it is discussed. Commercial services exist that produce and collect this data and provide it to researchers, universities, university libraries. Many science publishers and journals have also started using these, disclosing information next to a published article as to how and how much it has been discussed. Following these links, one can access and read the blogposts or Wikipedia articles or tweets that mention the article in question.

Open practices through the lifecycle of research promote high-quality and impactful science

In this interview, Leo Lahti talks about open research practices and reveals what is meant by open workflows. Leo Lahti is professor in Data Science in University of Turku, Finland. Leo obtained his doctoral degree at Aalto University in 2010.  

Leo and his research team focus on computational analysis and modeling of complex natural and social systems. Leo works actively in numerous national and international working groups and networks in which the common denominator is open science. He is one of the founders of Open Knowledge Finland.

ORCID: https://orcid.org/0000-0001-5537-637X

 

You are a methodological researcher and professor in data science at the University of Turku. Please tell us about your own research.

I could summarize it by saying that we operate between computational science and applied sciences. What my research is focused on is that we try to combine these two worlds that are traditionally a bit separated. What is perhaps highlighted here are the ecosystems of data science and the way modern data processing is combined with algorithmic problem solving, and this genuine data from genuine applications lives on in computational ecosystems, and it is developed in a communal way, in that it’s not just a single person or group that develops a method that gets applied to certain kind of data. It’s rather about building more extensive computational systems for different fields. It is problems related to this that we solve and to which we try to contribute, both to bring relevance to the applied side and, regarding the methodology side, perhaps solve some important problems that people are lacking the inspiration to solve. This feeds method development, for when you work with applications, you get new, fresh ideas as to what is relevant and what kinds of solutions are needed. So that’s the kind of research I do. Our research group is especially focused on complex systems in general or complex systems in the nature and society.

In data sciences there is quite a long tradition of such communal development. In your view, has it still kept increasing or changed somehow from how things used to be done before?

Of course, the open-source movement has been strong for a long time now. In general, in computational science, there has been a lot of code sharing. But I guess what has changed especially during the last few decades is the way it has become, in a certain sense, better organized and somehow more professional, with regard to how it’s done. What has developed a great deal are the systems with which we can maintain complicated structures where a thousand different contributors can make their contributions. I feel like the way it is organized has taken a leap forward. Secondly, the required standards have been developing all the time, so now as peer reviewing of software, for example, has become more common, in many repositories, such as R/Bioconductor, there is a certain peer review process that you must pass. So, you cannot put just anything in there, as there are actually quite strict minimum criteria that you must meet. Some of it is automatic testing and then some of it is curated by humans. In addition, you have the best practices, that is some unit tests and documentation, repeatable workflows and repeatable examples, which help to ensure that the software works as it should. However, I don’t want to overemphasize the software, because the software is, in a way, just a certain kind of tool we use after all. It is ultimately the issues related to the field of application that are the interesting ones. Thirdly, these things have spread across different disciplines. More and more disciplines have their own developer community and their own traditions for this kind of work. If I think about how it was 20 years ago, when I started doing bioinformatics, it was new to almost everyone to do computational genomics and stuff, so there were no tools and little expertise. Of course, there was some, but it was modest in comparison to how it is today. It is now an established discipline, and this same process has occurred in many other fields as well, and it is currently on-going in many fields. I think computational human sciences is a good example of that; of course, there are social sciences and linguistics, where they have been conducting computational research for a longer time, but when it comes to digital humanism or some other fields, there was pretty much nothing ten years ago, while now the communities are getting bigger.

You are a long-time active advocate for open science. What is your view on the current state of open science, and what is your understanding of the openness of research?

The way I see openness is that it is an ordinary part of the quality and impact of research, and I wouldn’t want to view it as being something else. The open science movement was, in a way, born to solve problems that have created bottlenecks that hamper quality and impact. As the society and science has changed, data volumes have increased, algorithms have become more complex, and information dissemination has also undergone something of a revolution. This has led to new kinds of challenges and possibilities with regard to traditional good principles of science. In computational science, which I represent, many open science practices were already common before the current wave of open research. And that’s probably one reason why I have been working with these things for a long time, because these were normal research methods already before anyone viewed them as representing ‘open science’. Now that open practices are spreading and becoming more common in all disciplines, or at least in a whole lot of disciplines, there are many differences from one discipline to another. Research funding, traditions, research ethics and research culture set very different kinds of limitations in different disciplines in terms of advancing open practices, and the interesting thing in the current state of open science may indeed be that we have now been given opportunities to start resolving these concrete bottlenecks. Another thing is that we have to make open practices an even more integral part of researcher training, and there are a lot of good things going on in this regard in my opinion.

I’d also like to mention the FAIR principles. Since I talked about the concrete problems we encounter and how they are solved in different disciplines, one concrete challenge is, of course, that data often requires protection, so there is quite a lot of data that cannot be opened. But in many cases, it can be opened partially or opened to some extent, and FAIR principles are helpful in reconciling the needs of opening and protection in a way that gets us closer to the motto of open science: as open as possible, as closed as necessary. So FAIR principles are perhaps one means to make progress towards a concrete solution between these tensions.

Why do you think research should strive for open practices to an even larger extent?

I once again emphasize that openness is one part of the quality and impact of research, and because we want to conduct high-quality, impactful research, we also need to strive for open practices to a larger extent. But the openness is not a target per se, rather the reason we want to strive for it is that we think it strengthens quality and impact. So, in this regard, if research should strive for open practices to a larger extent, what does that mean in practice? It doesn’t only mean that, once the results are published, research is open in some way, rather open practices can be applied throughout the research lifecycle, and as early as possible in the research process. Related to this, there are such things as research transparency, repeatability, reusability, and other points, the prominence of which of course depends on the stage of the research.

Since you brought up openness throughout the lifecycle of research, if we were to look at things from the perspective of your own research, what kinds of practices that one might call open can you differentiate throughout its lifecycle?

Indeed, there can be lots of differences between disciplines as to what it means in practice. In our work, too, since I work with both bio and human sciences, there are some differences depending on which kind of research we are conducting. But if I tried to differentiate some things that are related to it and that are highlighted in it, I would mention that the findability and accessibility of research is one thing. This is related to data, methods, research data, publications, and how findable and accessible they are. Another thing is the aspect of interoperability and reusability: whether the research can be further utilized and reused in new ways, or whether it’s like a one-time thing only to appear in a specific publication, so that it would not lend itself as easily for further use. Then there are transparency and repeatability as research quality guarantees. All of these are important in different ways. Maybe in data sciences, in practice, the algorithmic openness and reusability are highlighted when it comes to these matters. Of course, the things that I talk about are related to all research in one way or another, but when it comes to data sciences, algorithms and source code and their openness and reusability quickly surface. Open licencing, for example, is something that can concretely and technically ensure that the results are open and reusable. I’d also like to bring up preprint publishing, because it’s only common in some disciplines. I feel like there are a lot of cultural differences, as some are not accustomed to the idea, and we are now hearing some of the same arguments that were used in computational sciences in the 1990s.

We have talked about the openness of the research process from the perspective of workflows. What is a workflow and what does it mean to open a workflow?

Well, firstly, what is a workflow? The way I see it, a workflow refers to the whole chain of reasoning. There is some sort of a chain of reasoning in all scientific work. We have an observation of some kind, we also have a systematic method or approach to examine this observation, and then we have a conclusion. And, it doesn’t matter if the research is qualitative or quantitative or whatever, we have some kind of a chain of reasoning. An open workflow, then, refers to a chain of reasoning that is transparent and reusable. So now if we talk about data science and algorithmic work and methodological work in computational science, we tend to talk about computing infrastructures, code libraries, and stuff like that. In my opinion, it is still lamentably common in research that only single methods or algorithms are made open, yet the way these pieces are combined into a whole is not made open. And this combining is, after all, pretty essential, if you want to assess the sources of error in your research, the robustness and details of your research. The reusability also takes a hit, if everyone has to piece these things together over and over again, and they still won’t manage to do it in the exact same way as in some other study. And I know from experience that it can take months or even longer to make even a simple description of how to piece things together for them to work. So, workflow is that whole thing. And when you don’t pay attention to whether it is open, the research may look very open to the outside world, everything is open-source, and so on, and you may talk about how open it is, while in practice, it’s not necessarily like that.

What do you consider the most central benefits of open workflows?

At its best, they bring extra visibility to the work and enable early feedback, suggestions for development. We have a lot of experience that they help us to mature the work already before it’s published. In addition, we might also make it easier for others to utilize, so that it’s much easier for other parties to reuse it after we have had some discussion already in advance as to what it could be. This can save time and effort and take the research forward. Everyone benefits from this: the researcher community benefits, and being an active contributor helps you to get a lot out of it, having the chance to impact the field’s development. Another benefit for the researcher is precisely that opening your work in general makes you pay more attention to the documentation of the work, so it helps to improve quality.   When it comes to the researcher community and research on a broader level, research in general does get more efficient, resources are saved, and the cycle of research becomes faster, as there is no more need to develop every solution on your own, for you can utilize those made by others. Also comparing and identifying best practices is probably easier in such a context, because many people are working on the similar problems.

What kinds of challenges have you encountered in your own research with regard to opening workflows?

It does take resources and investments after all, it’s indisputable. Proper documentation, for example, is the kind of infrastructure work for which it is difficult to get actual research resources in general, because research resources are not usually targeted at building systems but at achieving research results. Development tends to be a side product of that unless you’re a methodologist such as me. Another factor is, of course, that if you want to ensure the work’s quality, you have to build all sorts of automatic tests and follow the standards, which is surprisingly time-consuming, if you want to do it well. There are many levels to this, and I don’t think it’s always even realistic to demand that we perfectly succeed in whatever we do. I think what is essential is that you consider case-by-case what is reasonable in whichever study, and then you try to advance the building of good practices and at least avoid needless bottlenecks, so that at least the incentives would work in the right direction. I guess another practical challenge, which I already mentioned, is that workflows are often linked with a certain computing environment and built on a physical platform, on a machine or in the infrastructure. And this separation of the code and the computing environment is not always that simple. Nowadays there are technical solutions to this at least in data science, but this is a practical problem sometimes. I guess the slowness of cultural change is another big challenge that I encounter all the time when I talk to colleagues. Not everyone is as keen on this. So, what to do about that? Perhaps rethink incentives.

What do you think the role of research organizations could be in advancing and supporting open research practices? In your view, how could we provide better support in the future?

I would say the implementation and monitoring should be tighter. You have to really look into how these things are implemented and monitored, because the truth is these things won’t happen, if there is no monitoring. And there must be incentives and support for this. So, there are a lot of good things under way on the level of principles, but the cultural change is slow, and if you don’t monitor whether these things are advancing, it easily happens that you start repeating the same old practices. And of course, preferably the researchers should be encouraged to do that, rather than punished, so I think positive incentives would be a means of support. Another thing of importance is clearing misunderstandings, because I still see that there are concrete erroneous conceptions about things that are related to open practices that we should try to identify and correct. Finally, maybe there should be a greater emphasis on openness in research ethics, which is – indeed – mainly focused on protecting research subjects. But in addition to protection of individuals, also open science is a question of research ethics, and when it’s possible, research should be opened as an act of good scientific practice.

 

OPEN SCIENCE COMMUNITIES FOSTER CULTURAL SHIFT TOWARDS OPEN SCIENCE

In this interview, Lydia Laninga-Wijnen talks about her experience with Open Science Communities. Laninga-Wijnen is Senior Research Fellow in INVEST Flagship at the University of Turku.  Her research interests are adolescent peer relationships and bullying, and her current research project SOLID is funded by the Dutch Research Council (NWO) and the Academy of Finland. Laninga-Wijnen is an active promoter of Open Science and the founder and the coordinator of the Open Science Community Turku.

Website: https://lydialaningawijnen.nl/

ORCID-ID: 0000-0001-6158-8950

 

What is an Open Science community?

I think an Open Science community can be described as a network for researchers, students, or even societal stakeholders, who are interested in Open Science. It is both for newcomers and for experienced colleagues to interact and learn from each other with regard to Open Science. It’s a bottom-up learning community. It can involve members of all career stages and all disciplines. I have myself founded the Open Science community of Turku. It consists of the community coordinator, that’s me, the ambassadors, and the members. The ambassadors are also in a kind of a leadership position so they can also bring up ideas on directions that we should take, and take the lead some of our meetings. Of course, members are welcome to bring in ideas as well.

What are the Open Science communities aiming for? Why do they exist?

The main aim is to make Open Science the norm in research. There are a lot of researchers who acknowledge that Open Science is really good, important and valuable, but there are still a lot of them who have hesitance to endorse Open Science practices, or they just simply do not have the tools, or they do not know where to start. Open Science communities facilitate this. Mutual respect is very important: if there are researchers who are critical about OS practices, we are very willing to start a conversation with them, because then we can learn from them to further improve the directions that we want to take.

I think, overall, an Open Science community has four aims. First one is to reach and engage researchers so that they really can learn about Open Science. It’s basically a platform where we can learn from each other and where we can also discuss our failures or bottlenecks in applying Open Science practices. So, it is really a learning community.

The second goal is to inspire and enable the adoption of OS practices. We want others to learn how to adopt these practices. We just had a preregistration workshop where we did preregistration together. The participants got to experience it once and hopefully that’s a small step towards the next one that they will try to do it on their own. In this way, we hope to be inspiring researchers to take the first steps, but also to consolidate the current OS practices.

The next goal is to shape institutional policies and advise organizations on how they can promote Open Science. It is important that an Open Science community, in general, operates independently from an institutional policy. So, it is a self-steering, researcher-led organization, and we do not receive instructions or tasks from other parties. But it is, of course, also important to be connected with the organization in a way that we can inform policies from the bottom up, because the policies describe what is required or desired, and what is incentivized. We can also provide advice on infrastructure, that is, what we need for a smooth transition to a more open science.

And I think the fourth goal is that we aim to foster interactions between the academia and the society. An Open Science community is open not only for academics but also to people from societal stakeholders who may also become members and participate in our activities. We hope that such stakeholders can also enable us to engage more in citizen science and to have more contact with people for whom we are actually doing research.

We have already touched upon this topic, but how do you perceive the role of Open Science communities in this transition to Open Science?

The main role would be to bring about such a culture shift that Open Science becomes the norm, I think. You may say, yeah, Open Science is good, why do not all researchers do it. And I think that it is mostly because there is this “older” culture of doing research that people are adjusted to. This is how they are used to do their research and feel comfortable, so why change it.

There is an interesting blog post by Brian Nosek about this. He says that to generate a culture shift, it works like a pyramid. At the top of this pyramid is the policy, so you need a change in policies that promote Open Science to ensure that people also want to do it. So, it should be rewarding for researchers. And at the bottom of the pyramid, there is the infrastructure. You must have a user-friendly infrastructure so that you can put Open Science into practice.

And well, in the past years, a lot of attention has been paid to improving this upper and lower layer of the pyramid. With regard to policies, funding agencies increasingly require that we adopt Open Science practices (e.g., publish open access) and journals now provide us with badges if, within the research process of the paper, open science practices (e.g., pre-registration) were endorsed. We also have the infrastructure. We have Open Science Framework to pre-register our studies, we have WORCS to open up our workflow, and every day there are new user-friendly tools being developed to apply Open Science practices. But in the middle of the pyramid, there’s the challenge: the majority of researchers still sticks to old habits – it is not really the norm to adopt these open science practices yet. And that is where the OS communities come in, to really make this culture shift. So, you need people who actually do it and who diverge from the status quo basically. The transition to Open Science is mostly a social challenge.

 

Do you have critical voices in your community?

Yes, sometimes we indeed have some. And sometimes I myself am a critical voice. I think it’s very important that we are not staring blindly, like oh, Open Science is perfect as it is, and everybody should do it. It would also scare people off from doing at least something to make their work a bit more open.

Applying OS practices can also be a bumpy road sometimes, and that is also important to acknowledge so that people who are initially very enthusiastic and want to do everything with regard to Open Science won’t get disappointed. I think that Open Science communities are helpful here, because otherwise it might be that researchers mostly hear about success stories from ambitious colleagues like yeah, I did a registered report, and it went very well. Whereas in an OS community you can also share some of the things that did not go right.

 

How do you view the benefits of participation from a researcher’s perspective?

There are many benefits. I think one important thing is that Open Science will become the norm. So, it is not a movement that will blow over. The culture shift is already happening, and as a researcher you will eventually be asked to do these things. So, it is good to step on the train that is rolling now. And one way to do that is to be part of a community and to learn more about open research practices. There are still many researchers who believe that Open Science is mostly about publishing Open Access, but there’s so much more to it. Topics can include transparent methods, preregistration, sharing data, code, research tools, reproducibility, replication research, citizen science, open peer review, diversity and inclusion, research integrity. All these are covered by Open Science.

As a researcher, you do not have to do it all. As I said, you can first take steps that fit best to your own project. I also think that Open Science can be very liberating for researchers. It kind of frees researchers from this ‘publish or perish’ thinking. For a long time, researchers may have felt pressure to publish clean and interesting stories. Sometimes a researcher may have worked on a project for years and then they conduct their analysis and the analyses do not produce significant results.  The researcher may become afraid, like oh, maybe I can’t publish it. And all the work is seen as non-relevant. But it’s now recognized that this thinking is super problematic and can create a replication and publication bias.

I think, as researchers, we may increasingly experience freedom that, regardless of outcomes, if our study is well-designed, we can have more and more confidence that it can be published. Also, this way, we will have a fairer view of reality. Because if you only have significant and interesting stories, then to which extent does science really say something about practice? Open Science is also increasingly rewarded, by journals and also by funding agencies. For a researcher, it is still important that you sometimes publish in scientific journals or that you gain funding from agencies. Yet, by being a member of an Open Science community, you show like, okay, I am active in the field of Open Science. I think that is really a big thing, if you can say that.

 

The adoption and the relevance of OS practices vary across disciplines, and then again, the OS communities are interdisciplinary. How disciplinary differences manifest themselves in the activities of OS communities?

I think that, in general, we can learn a lot from other disciplines. A particular OS practice may be very common in a certain discipline, whereas in another it can be quite new. For instance, with regard to registered reports, some disciplines are underrepresented. If I look in the literature, I do not see many registered reports for longitudinal projects, and I think that’s partly because it’s quite hard to make a registered report for such projects. This is not desirable so there is a need to think about how we can fix this and make sure that it is possible also for longitudinal projects. Within an OS Community, we can share experiences and bundle them together with researchers from other disciplines who may have some keys on how to do it. And I think that in this way we can really learn from each other. At the same time, there may be practices that are only relevant for a certain discipline, and then they can be organized as a kind of a sub-activity just for that discipline. We researchers have a lot of things in common and sometimes more than we think, so that is really the value of interdisciplinary OS communities.

 

Could you share some practical tips or recommendations for a researcher who wants to get started with an Open Science community?

My first step would be to connect with the international network of Open Science communities as soon as possible, because they can really help you to make a solid plan. I was initially quite enthusiastic and did a lot of things in between my other tasks. If I had contacted earlier, it would have helped me a lot in that I wouldn’t have needed to reinvent the wheel myself sometimes. It is also good to be aware of what is already happening in your university regarding Open Science, so that you can really learn from people who are already involved in it and who know the right people to make a connection with. It is also good to be aware that endorsing Open Science can be a bumpy road and to be prepared for that. The initial enthusiasm of people can be tempered when they experience these bumps.

It is also important that the university has a favorable attitude towards Open Science and takes it into account also when assessing researchers, so that this enthusiasm for Open Science can remain, and that people do not take it as a failure, if for instance, they do not succeed with their registered report even if they try hard. They should be assessed based on their endorsement of Open Science practices rather than based on maybe not having everything published. So, I think these are important things for an Open Science community to flourish and also to be sustainable. Because you can activate people to do Open Science; but if it eventually doesn’t pay out for them because the university evaluates and rewards other aspects of research (e.g., publishing many papers), they may lose their enthusiasm and stop doing it.

Open data and open code advance scientific progress but must be rewarded in researchers’ careers

In this interview, Maria J. Ribeiro talks about her experiences on data sharing. Maria J. Ribeiro is a researcher in human neuroscience at the Coimbra Institute for Biomedical Imaging and Translational Research (CIBIT), ICNAS and Faculty of Medicine at the University of Coimbra, Portugal. Maria completed her BSc in physics in 1996 by the University of Porto, Portugal, and obtained her DPhil in neuroscience in 2003 by the University of Sussex, UK.

Her research contributes towards unveiling the neural mechanisms behind perception, decision making and actions in humans using non-invasive neuroimaging techniques and physiological recordings. Since 2015, Maria has been working in the field of ageing. She is interested in understanding the neural changes that occur in ageing, and how they are causal to cognitive decline and contribute to the onset of neurodegenerative diseases associated with the ageing process.

Website: http://neuroscience.pt/

ORCID-ID: 0000-0001-6422-3279

GoogleScholar

Twitter: @Ribeiro_neurosc

 

You do research in the field of human neuroscience. How do you view the value of open data in your field?

I think there are two important aspects to opening data. One is to verify the results that are published and try to tackle fraud in science. Once the data is published, at least we can verify that the data exists and that the findings were not fabricated. Then we can also reanalyze the data and make sure that there are no errors in the analysis. In fact, it is not just that you can publish the dataset, but you can also make all the analysis code publicly available. That is something that I have done for my latest publication. That means that if someone else wants to repeat my analysis they will have the analysis code to do it. Also, if they want to use some part of the analysis for their own study, that can be done too. I’ve done that myself. I’ve gone through other people’s papers and adapted their code for my own research. That is a big help, as then we don’t need to keep reinventing the wheel. But that also ensures that errors in the analysis may be caught by other people, and then, the study may be reformulated. It’s something that I think is still not done enough.

The other reason why opening datasets can be very important is to maximize the usefulness of the data itself. In neuroimaging, the data are very rich. There’s so much that we can study with one single dataset. And usually, in the original study, the researchers will only look at one aspect of the data. For example, if we have data that covers the whole of the brain, or data that covers EEG data from the cortex, we can look at the visual cortex, we can look at the frontal cortex, we can look at frequency data, we can look at connectivity, we can look at so much. There are also constantly evolving new methods that permit you to go back to old datasets and reanalyze them with these new methods.  So, it’s important to have the datasets available so we can go back and forth and have different people analyzing them from different angles.

As you already mentioned, it does take some effort to open data. Could you explain the process of opening the data from the perspective of your research?

I did, obviously, some research on the internet to try to find out what would be the most appropriate place to deposit my data. Then I followed the suggestions of the EEGLAB toolbox. They have a plugin that helps you to format the data into the BIDS format. I read whatever was available about how you should format the data. EEGLAB also has a YouTube channel where they have videos explaining how to organize your EEG data for sharing it. They also suggest publishing it in OpenNeuro. So, I just followed their advice, I used their MATLAB code, and adapted the code to do it. It took me one week of work to finish the process, to organize it and to make it work. The data has a little bit of a strange format, because it has two groups of participants, each group has three different tasks, and the tasks have different numbers of runs. I had to adapt some of the code that was available to make it work and organize the data as it should be organized. The reason why I could justify taking the time and effort to do this is because I wanted to publish my study in a journal that values open data, and I think this is an important incentive for researchers. If we want the data to be useful and be reused by other researchers, we want to make sure that all the relevant information is with the dataset. It’s obviously something that takes time, you must sit down, think about it, and have all the information there.

Was it your personal interest to share your data?

It was my personal interest. I think within my local research community, I was the first one to open data although we have been talking about it for a while. There are others who are now also interested in doing it. Obviously, it is easier with EEG data, because it’s easy to make it non-identifiable, to anonymize it. With MRI data, we have the problem that the person can be identified, because you can identify the face of the participant. So, with MRI data it is a bit more challenging to actually make it completely anonymous. You will need to deface the data, and there are some challenges there.

What were the reasons that made you think about it in the first place? How did you come up with the idea?

It was also because I don’t want this data to be lost. I know there’s so much potential to it, and I don’t have time to do everything. So, if someone else can get interested and pick it up I’m really happy to make it available for the community. For me, it is quite rewarding to know that people are interested in my work and the dataset that I produced. That is one reason. The other incentive to actually spend the time and do it was that I knew that with an open dataset, the study would be viewed as stronger for publication. So, if the journal values open data, that is obviously another incentive to do it.

Have you had any concerns over opening your data?

My dataset is published in OpenNeuro. We’ve published three papers already with that dataset, and we are now working on a fourth paper. But the data is now open, and it has been downloaded over 200 times. However, I don’t know what the other researchers are doing with it. I’m not worried that they are going to publish the same that I’m working on, but I’m curious. I know, in some cases, the datasets are made open, but the researchers require that you contact them and tell them how you are going to use the data. But in OpenNeuro, you don’t have that option. The idea is that it is open, and anyone can download it and do whatever they want. I was just asking them this week, that if someone publishes something with the dataset, I would like an email to be sent to me. I know that I can always look for it online but sometimes you might miss it. It would be good to have an easy link from the dataset to all the publications that resulted from someone using that dataset, and that is not yet implemented in the repository.

How do you view the role of data management and data management planning in relation to opening research data?

I think it’s important that the laboratories and the institutions have a clear policy on open data. It is a fairly new idea, and a lot of people will find it strange. You know, one may wonder, what if someone else publishes before me, or what are the other researchers going to do with the data? So, it is important that these kinds of issues are discussed with the researchers and that there is some sort of local policy of what should be done in terms of how to organize your data, how to anonymize the data and make sure that it’s not identifiable? I think that the researchers need to be supported here, because it’s not trivial to do it well the first time.

Do you find that open data and its benefits are discussed enough?

Not as much as I would like to. I think that one of the big advantages I already mentioned at the beginning would be to have the published studies double-checked. So, you publish a study, and someone else goes and checks the data, checks your analysis, and then finds any mistakes. This is something that I don’t think is done enough. I still don’t think we are there and see this being done in a systematic way to capture the mistakes and studies that are not replicable.

Your data has received quite a lot of downloads already. Did you take some specific actions to increase its visibility?

I haven’t done anything, actually. Now thinking about it, maybe I could have disseminated it a bit more. So, maybe it is just the dataset that catches people’s attention because it has a relatively large number of participants, EEG, ECG and pupil data. It also has a passive condition that people can use for resting state studies, and that is something that is also quite popular. I hope we captured people’s attention for the right reason, and they’re doing nice research with it.

What do you think are the biggest barriers for those not opening their data?

Researchers are focused on whatever output is valued for their career progression. So, if it’s publications, that’s what they’re focused on. And if publishing a dataset is not seen as important, then it will be something in the back of their minds. Then there may be concerns about privacy and the identifiability of the data that can be problematic. Another concern may be people finding errors in your studies. Not everyone would be happy about that they publish something, and someone else comes and says that it’s wrong. Finally, if you are still going to keep using the data, there may be concerns about someone else publishing the same results.

Could you share any tips with researchers who are considering taking the effort of opening their data for the first time?

It is really important to have the data well documented. Try to put yourself in someone else’s shoes and think what it is that they need to know in order to use the data. I also recommend using a standard format that has been validated by the international community. That makes it easy for others to use it, and also helps to structure all the information, also for ourselves. Also, don’t think that you are going to do it in your free time. You will need to assign time to do it. That is also why I think it is important that the researchers feel that this time that they are investing is actually valued for their careers. It’s very easy for us to say that it’s good for science. But then, if we don’t have a job or if we cannot progress in our careers, that’s also going to have a big impact. So, it is important that the decision-makers send a clear message if it is important to share data or if it is not. I don’t think that this message is very clear yet.

 

 

Open workflows enhance transparency in research and make your work accessible and usable for your future self

In this interview, Mrudula Arunkumar talks about transparent workflows in research. Mrudula Arunkumar is a doctoral candidate in the Department of General Psychology II at the Institute of Psychology at Friedrich-Schiller University Jena, Germany. She completed her B.Sc Psychology in 2013 at the Madras School of Social work in Chennai, India and finished her research masters in cognitive neuroscience at the Donders Institute in Nijmegen, Netherlands. She then briefly worked as a research assistant at the Max Planck Institute for Psycholinguistics exploring the influence of cultural inventions, such as literacy, on visual and auditory perception. Now in her PhD, her research focusses on investigating whether automatic response retrieval or conscious awareness underlie higher order forms of contingency-based learning by using techniques from pavlovian conditioning such as overshadowing and sensory preconditioning.

ORCID: https://orcid.org/0000-0002-6441-9623

 

You have been quite determined to pursue transparency in your own research from the outset. What do open workflows in research mean from your perspective?

Personally, I view open workflows in a holistic way as the whole research process being transparent in each of its steps. The overall idea is simply to make sure that every part of the research from experiment design to analysis code is made open. I’ve personally benefited from other researchers’ open workflows in my own research, for example, when I have needed programming help with my experiment design. Maybe someone else can benefit from my work.

What kinds of steps are you taking when you make your workflows open?

Firstly, I have an ORCID account – an open researcher identifier – and an OSF account. For me, having these accounts was just a good starting point. Specifically, the OSF account helps because they have a section where you can create your pre-registrations by using existing templates, and everything is easily stored in the project folder. As we all know, the first experiment rarely works or is perfect. Instead, you have multiple versions of the experiment, or maybe you have trials and errors. For me, personally, it has been helpful to have pre-registrations of all my experiments in one project folder on my OSF account. Secondly, when coming to the experiment design phase, I use PsychoPy that is an open source software for designing psychological experiments, for example behavioural experiments. Especially with the pandemic, the software has immensely improved to allow eye tracking and has features beyond simple behavioural experiment measurements. That’s a really good and easy-to-use open software that I’ve used throughout my PhD. You can easily download the software and just use the materials that someone’s put up on their OSF account and run the experiment on your own. When it comes to analysis, I use another open source statistical analysis software, RStudio to carry out my data analysis and data visualization so that it can easily be replicated by fellow researchers.These are basically the main things that I do to make my workflows open. Of course, finally, I make sure that I have a link to my accounts in my final research paper because this way everything is accessible to the public.

How do you find the popularity of these practices? Are they becoming the norm in your field?

Yeah, I think so. Psychology went through the replicability crisis some years ago, and I think that’s what kickstarted the seriousness of Open Science. I don’t know if it’s just been my journey that’s made me part of labs that already practice open science, but I’ve tried to maintain it wherever I go as well. So, if I collaborate with someone, then it just becomes automatic that, “Yeah, of course, I will pre-register.” The same thing goes for the next generation of researchers. Last year, I taught a course for bachelor’s programme in psychology. I think it’s really good for them to know of these things at that stage already, so I dedicated one lecture to Open Science practices just to make these open science practices more familiar to the next generation of psychology researchers. It’s pretty much the norm that they have to do these things while conducting reseach.. Still, I could see that open practices may not be as common everywhere as they are among my colleagues. I would definitely emphasize these if I collaborate with another university where that is the case.

What do you think are the main benefits of open workflows from a researcher’s perspective?

The main benefit, I would say, is that it saves time and progresses research much faster. There probably is someone who did something similar and then you can just access their OSF project, get their experiment, and easily solve any issues that you have within your own workflow. I think that it makes a big advantage timewise that you don’t need to create solutions on your own, that there are other open projects available from which you can get inspiration and ideas.

How do you find the effort that it takes? It does sound that there are a few things to think through.

It is definitely an added thing to your usual tasks that you make sure that the OSF repository is clean, that you have a README file that everybody can understand, and so on. These are all extra steps that you need to take, but, personally, I haven’t found them labour-intensive in a way that it prevents me from doing it. So, I don’t think it’s come to a point where it’s too much of a task for me to arrange all this. I also feel like it is a continuous journey. So, maybe two years from now, my code is better readable, or I get more efficient, so I don’t need to spend that much time on making the project clean. Probably it comes with constant practice. But it’s not always easy for sure.

You have already mentioned a few tools that you have been using. If someone’s new to open workflows, which would be some of the good ones to start with?

Another good thing about Open Science is that many of the resources are also openly available. There are many introductory YouTube videos on Open Science, open workflows, and pre-registration. I have used AsPredicted.org’s template for my own pre-registrations, and that has been very helpful because it is quite straightforward. The template has a list of questions, and all you need to do is to answer those questions. So, it is not like you must create the whole document on your own. The OSF also has a pre-registration template. Also, discourse.psychopy.org is another good website where you can find a lot of help and solutions to start programming in PsychoPy. For R, there are many communities, and people seek help for R even on Twitter. Then, there are RStudio’s communities and Stack Overflow that have been really helpful in finding all kinds of solutions.

Lastly, could you share any tips with newcomers to open workflows? Where to begin?

A good perspective is to think about making your data openly available and accessible for yourself and maybe even for your future self, as you may change universities or change jobs. Making your work open may be a less daunting experience if you just approach it from your own perspective. I also think it’s good to start small as, eventually, it will make a bigger impact. So, I think a good step would be to at least make your data openly available, then go on with pre-registration, and then with open software. Or maybe all in parallel; that would also be great.

Open Access publishing as a mode for scholarly publishing

In this interview, Nicolas Pinet talks about Open Access (OA) publishing. Nicolas Pinet is a library curator, deputy director of the Service Commun de Documentation (SCD) at the University of Poitiers and head of the SCD’s “Research Support” department which offers wide range of support and training on open science for researchers.

• IdRef : 110355091
• 0000-0001-5235-2364

How do you view the level of researchers’ awareness with regard to open-access publishing?

It is very discipline-based. We have a rather decentralized academic practice in France, in the sense that the institutions themselves cannot impose professional practices on their researchers. Any improvement in terms of practices open science is endogenous. And usually, change appears because there are some very motivated researchers that bring some change in the field. We as institutions, again, cannot force the researchers to change their practices, because there is a lot of self-evaluation within the community. And as long as those structures do not shift their position, it is difficult for us to get people involved. That being said, especially the younger generation of the researchers are interested in these matters, and they, for instance, do not approve the current publishing model. There is a strong interest among these younger researchers to make things available as easily as possible.

You mentioned that the organization cannot impose these practices on researchers, but does it somehow promote open-access publishing, and in what ways?

By training the new generations of researchers and establishing the idea that open science is now the norm. We also have an institutional stance that views open science as facilitating recognition of the scientific work made by our researchers, so we tell them that open publications will be read more easily by more people, by more diverse publics. By making their publications accessible, they are making a favour for their careers and the visibility of their own work. The more a publication is read, the more it is cited.

What do you view as the main barriers for researchers to publish in OA?

It is usually a bit difficult to get established researchers to shift over to better practices in terms of open science. That is why we work so hard on training doctoral students, because they are the most open to change in this regard, and they are usually very sensitive to the ethical issues of open science, so they are easier to get involved with these things. That is not to say that there is a rejection of the idea of open science or the fact that you have to register on HAL, for example, but it is much easier to get young researchers to do it than to get the more experienced ones to. It is also linked to the current reform of the evaluation and reward systems of research to include open science practices. To give a bit of a crude example, it’s difficult to convince an experienced researcher that, instead of publishing in Science with extraordinary costs, that he should publish in a less-known but open-access journal. There’s quite some resistance on this front, but again, we hope that by training the younger generation, there will be more sensitivity towards this aspect.