The Challenges of Measuring Scientific Research Productivity and Impact in Developing Economies

Scientific research is a difficult investment to measure. On the one hand, many developing countries around the world use scientific research as a mean to improve their economy and has found way to reduce their dependency on their natural resources. On the other, scientific research is an expensive endeavor. In this paper, we look at the different factors impacting scientific research and consider the relationship between governments and researchers, the impact of businesses on basic research funding, and the methods through which the performance scientific research is measured. This paper also considers the impact of scientific research funding measures on researchers’ careers and briefly discuss the common public research funding mechanisms around the world. We also discuss knowledge-based economy and Solow’s theory of economic growth and its later development made by Romer and some the elements that are not addressed by them. In this review, we have considered developing economies and the difficulties they face in moving away from their dependence on their natural resources towards become economies dependent on producing knowledge and implementing this knowledge into their business practices. This paper takes a closer look at the state of Qatar since it is a developing country trying to move away from its economy’s dependence on fossil fuel production to a knowledge-based economy.


Introduction
Human curiosity has always been a driver of our race's prosperity. However, as humanity's prosperity increased so have our needs. As a result, humanity often finds itself needing to find new ways to meet the increasing demands of rising human populations. During the past century, science has made great technological development and innovation has driven humanity to a new era of prosperity. Despite the admirable accomplishments of science today, political leaders find it difficult to justify the need for scientific research. This came as a result of the low success rate of small innovative companies in favor of larger companies with large scale production [1]. Furthermore, scientific research is a difficult investment to measure through traditional methods [2]. This can be attributed to the fact that scientific research encompasses different disciplines and, by definition, looks for unconventional solutions that often challenge conventional performance measurement tools. Nonetheless, many scholars have argued that the "leading edge of the economy in developed countries has become driven by technologies based on knowledge, information production, and dissemination" [3]. As a result, developing countries today are moving towards establishing an economy that is based on knowledge and its dissemination [3]. However, today's lack of methods for demonstrating the impact of publicly funded research on the economy has lead the public to question whether developed countries' economies are made stronger by scientific research or if scientific researcher is simply an expense they can afford.
To better understand the role of research in economy, it is important to discuss Solow's theory of economic growth. In this theory, the macroeconomic setting includes four key components: (a) resources constraint related to the GDP whereby "the output is allocated to different uses" such as consumption and investment, (b) a production function that describes how output is produced from capital and labor, (c) the equation describing the accumulation of capital and labor and finally (d) a specification of how much GDP (output) is used towards investment. His theory shows that endogenous economic growth is possible through providing new technologies that would improve the efficiency of labor. Additionally, studies building on Solow's work have demonstrated that the increase in skilled workforce available for conducting further research and development activities that would further improve economic performance. Solow's theory demonstrates that if the labor augmenting technology grows then the macroeconomic variables will also grow in the long run [4].
This theory was further developed by Paul Romer who had built upon Solow's work to demonstrate the impact of the creative destruction process. According to Solow's theory, developing countries would have a faster growth rate than developed countries until their economy reaches that of developed countries. However, Romer's work, using countries' growth figures available at that time, had demonstrated that this was not the case. Romer explains this phenomenon through demonstrating that other factors also affect endogenous economic growth including: (a) the accumulation of ideas as the source of long-run economic growth (b) Ideas are not viral (c) A larger stock of ideas makes it easier to find new ideas (d) Ideas are created in costly but purposeful activity (e) ownership of ideas and the owner's right to sell those ideas [4]. Despite the comprehensiveness of both Solow's and Romer's works, they had not taken into consideration differences between scientific disciplines and the entry barriers of different industries, which add another layer of complexity to the matter.
In first world countries like the United States, biomedical research investment returns $10-$16 for every invested dollar [5]. Many governments today seem to focus on funding research projects that promise short term results and big economic returns [6]. The reasoning behind such decisions is that funding such projects would improve innovation and therefore improve the governments' return on research investments [7]. Around the world, there is an increasing pressure to maximize the return on investment (ROI) in medical research products [8] which makes it difficult to justify expenditure on basic/foundational research that primarily aims to generate new knowledge rather than new products. Nonetheless, research in the basic sciences adds great value. For example, untargeted basic medical research also provides the data necessary for designing experiments that tests cures for the diseases which humanity suffers from today [5].

Knowledge-Based Economy
A Knowledge-Based Economy is defined as the "production and services based on knowledge-intensive activities that contribute to an accelerated pace of technological and scientific advance as well as equally rapid obsolescence" [3]. Unlike traditional business investments, investments in knowledge generation do not always directly lead to the generation of revenue [3]. However, Powell, et. al. points out that knowledge does not contribute to economy through new technologies only, but also through improvements to management services [3]. Some social scientists focus has been on "patent-based measures to quantify R&D activities and stocks of knowledge" such as human, organizational, and intellectual capital [3]. As a result, funding agencies around the world find themselves pressed to support projects that directly help in improving the economy and create jobs and focus on the common criteria such as publications or newly developed businesses to measure the performance of research projects [9].
Both Solow's theory and the suggested improvements made by Paul Romer demonstrate how research helps in improving economies and generate value to the customers. However, neither theories explain the importance of research that is focused on generating new knowledge that is essential for generating new technologies to serve the customers. Further, Solow and Romer did not take into consideration the challenges that may be faced by developing countries while trying to develop their knowledge-based economy. For example, the United States, have shifted to producing intangible goods and information to capitalize on new innovations since the 1970s [3] which had helped in developing institutions that the could incorporate new knowledge into their processes which indirectly improve their economic performance. However, during difficult economic times, the indirect benefits of academic research become difficult demonstrate during difficult economic times which often results in the reduction of scientific research funding. Additionally, funding agencies often struggle to demonstrate the value of publicly funded research through the tools used today to measure the success of research projects (such as patents and total number of publications…etc.). In our discussion here, we will focus on biomedical research since most of the studies regarding the issues mentioned above are very well demonstrated in that field. We will also be using Qatar as a sample of a developing nation trying to change its economy into a knowledge-based economy. We have selected Qatar since the country has embarked on a national vision to change from a hydrocarbon economy to a knowledge-based economy and the country's contribution to scientific research was very low prior to embarking on the Qatar National Vision of 2030.

Ethical Dilemma of Patents
In line with the theories described above, many medical innovations are aimed at cost reduction; however, biomedical research patents (which are often used to measure research's return on investment) seem to increase medical care costs in middle to low income countries and impede use of research data [16]. Such ethical dilemmas make it difficult for governments and sponsors to require researchers to patent all their research products. On the other hand, policy makers often find themselves resorting to measurements that demonstrate direct economic returns to measure the value of publicly funded research projects. Despite this dissonance, many publicly funded researchers perform well and produce outcomes that serve both the community and the economy. In order to fully understand the different factors impacting research performance, it is necessary to take into consideration the direct and indirect returns that research brings.

Measuring the Economic Impact of Scientific Research
Return on investment on biomedical research does not only manifest in drug companies and medical devises products profits, but also in medical expenses savings in countries where patents do not raise medical expenses [5]. Another venue where academic biomedical research's ROI can be considered is in the training it provides to junior researchers who support medical advancement not only through the production of new drugs and technologies, but also through bringing innovative solutions to logistical issues with patient care. Further, those trained in solving scientific problems brings a new perspective that policy makers, who may lack the background, do not necessarily have. In fact, some scholars argue that the need for performance measurements for research stems from the following fact: researchers are able to achieve tasks that policy makers/governments do not necessarily know much about [10]. Even though scientific research impacts economy in different ways, the methods used to evaluate the performance of publicly funded research projects tend to focus on the scientific impact or direct economic impact (new business, patents,..etc.).
Funding agencies around the world have tried to measure the performance of scientific research projects through different methods. For example, the Netherlands considers the number of degrees and the number of PhD defenses when measuring the performance of R&D funding [10]. Other measurements in Germany and Denmark use numbers of publications and citations as a measurement of research performance [10]. According the OECD, only the United Kingdom has unique ex-post allocation mechanism that has a direct financial impact [10]. The difference in the measurements used is a result of the goals of the funding agencies which may be influenced by political pressures In 2016, the European Research Council (ERC has conducted an audit of funded scientific research projects using opinions from reviewers in respective fields instead of bibliometrics [11]. Even though many of the reviewers had worked for the ERC in the past, they found that two thirds of the audited projects had achieved a scientific breakthrough [11]. This breadth of the impact and methods for measuring the performance of research and development activities had further compounded the difficulty of establishing knowledge-based economies in developing countries. As a result, funding agencies around the world find themselves pressed to only support projects that directly help in improving the economy and create jobs [9] and do not consider the value added by research through spreading awareness and improving existing practices and policies.
For businesses, the appropriate amount of research investment is determined by the potential value that a project can bring to the company. Government funded research projects that produce patents and promise "short-run payoffs" [13] are often adopted by such businesses or result in a business which use them to improve the business's value and contribute to the economy. As a result, academic research is often evaluated by its ability to generate profits through innovations, but not the value it brings to society. As a result, basic biomedical research (which usually take a long time to reach commercialization stage) is mostly dependent on the funding provided by governments, which allows researchers to consider the benefits that their research could bring to their communities, but puts them under the scrutiny of policy makers and nations who typically need to demonstrate the value brought through funded biomedical research projects during their terms in office [11]. It is worth noting here that the basic research necessary to create such patents is often supported by public research funds. Further, researchers (particularly medical researchers tend to avoid patenting their research to ensure it can benefit as many people as possible which prohibits medical researchers from utilizing all possible funding opportunities [12]. Finally, the behavior of drug companies may be justified by the fact that entry barriers for new drugs into the market are high and companies are often hesitant to adopt newly discovered treatments. Despite the fact that the patent system has been developed to encourage the private industry to contribute to research and development the majority of basic funding research (which is necessary for driving the research activities described in Solow's growth theory above) leading to drug discoveries is being done by universities and public research institutes using public research grants. In France, only 11% of the newly approved drugs between 2004 and 2019 were real advances [14]. Further, the patent system in Europe has been criticized for being too focused for allowing companies to set their prices of the newly discovered drugs [14]. The control of the drug companies on which drugs will go into the market has also led to focus on producing medications that yield profit while important health problems (such as drug resistance issues) remain poorly studied [14].
These issues are further compounded in developing countries that had recently started considering knowledge a source of income which need to develop their communities (which may suffer from health problems different from those found in developed countries) and demonstrate the added value of publicly funded research. Biomedical and healthcare research in developing countries lack of "research education and training for health professionals, lack of appreciation for the value of health care research as an important tool for progress, shortage of funding and research resources, special bioethical standards and concerns, limited access to health informatics and individualism and inability to work within groups" [15].
Even though Solow's and Romer's subsequent models are an important basis for demonstrating the different elements affecting economic growth, this theory is lacking when considering developing economies such as those in the MENA region which are famous for their political instability. Additionally, the first and last factors in Romer's model cannot be applied to different areas of research, and neither Solow's or Romer's work consider basic research and focus instead on applied research. Finally, both theories do not provide a method to determine the impact of basic scientific research on the economy or the need to maintain the integrity of scientific research which cannot be left to institutions that have vested interest in showing research data that would help improve their shareholders' (tax payers in most cases) returns.
Despite the geopolitical pressures in the Middle East and lack of research infrastructure, Qatar's resources and commitment provide them a great opportunity at building a research program that could serve as a model for developing countries. Additionally, the challenges Qatar faces in developing sustainable biomedical research programs present a great source for research about the best practices for administering biomedical research [9]. However, a big hurdle against achieving these goals is the fact that many of Qatar's academic and non-academic institutes did not have research as part of their original mission. As a result, many of these institutions find themselves in a position where they needed to change their strategies to contribute to the QNV 2030. In addition, many of the Qatari institutions (including the universities) are currently working to further develop their policies and processes so that they can utilize and generate new knowledge.

Research Intensive and Teaching Intensive Universities
There are different ways for classifying universities depending on their focus and activities. The methods used for determining the performance of universities varies depending on whether a university is teaching intensive, research intensive, or Entrepreneurial. Institutions that are focused mainly on teaching often use the number of graduate and undergraduate students as a measurement for their success. On the other hand, research intensive institutions use the number publications as their main indicator for their performance. With respect to innovation, Bonaccorsi, et al. state that many schools use the funds raised by non-governmental entities to measure their performance [17]. Depending on the primary focus of a university, the school's policies and processes become optimized to improve its performance in the indicator relevant to the university's mission. Prior to 2008, Qatar's higher education institutes have not been focused on research or entrepreneurship. This was also true for many of the governmental entities and hospitals in Qatar which were focused on providing services. However, adjusting the existing policies to encourage employees and students to conduct research requires a lot of time. Additionally, Qatar unique environment makes it difficult to implement procedures that had been successful in the west. Additionally, even developed countries like the United States or some countries in Europe face challenges when it comes to managing scientific research and matching their research funding mechanisms to address the challenges faced by the industry and society. Further, the vastness of differences between industries and disciplines introduces challenges when considering universal performance indicators for academic research.

Scientists' Careers
In 2014, many promising researchers were not confident about their future careers [18]. According to Albertsa et al, the funding mechanism in the United States has assumed that research is going to continue growing indefinitely [18]. However, the funding for medical and environmental research in the United States has been significantly reduced [19]. As a result, many researchers today find themselves stuck in post-doctoral positions and competing for an extremely limited number of faculty positions. In fact, in 2011 it was estimated that less than 15% of American citizens graduating with a PhD will be able to land a faculty position in research institute where the U.S.'s more significant research is being done [20]. Additionally, the number of tenured faculty tracks in the United States has reduced and did not increase worldwide [21]. Despite the limited number of available positions for their graduates, most PhD programs today primarily focus on preparing graduates for an academic career in research and ignore any other potential career paths, which creates an imbalance in the market [21]. This problem is recognized by the NIH funded consortia BEST which is "aimed at identifying best practices in graduate and postdoctoral training for diverse career options" [21]. However, these efforts need to be supported by changes to the way funding agencies define the success of research projects so that it helps in redirecting researchers where their skills are best utilized.
Lack of training in careers outside of academia has led graduate students to spend more time as post-doctoral associates and for leading investigators to use them as well trained "cheap" labor. The leading investigators' behavior was encouraged by both universities that focus on hiring faculty who have the best publications profile and funding agencies who focus on bibliometrics. Weis et. al has stated in a recently published commentary about multidisciplinary research in the United States that the "infrastructure, systems of rewards, claims to authority, are rooted in the pursuit of deep, highly specialized knowledge" [22]. While universities' goals are primarily to generate and teach new knowledge, many PhD holders find themselves in a "hypercompetitive environment in which scientific productivity is reduced and promising careers are threatened" [18]. In the United States, the number of biomedical researchers kept increasing while funding for biomedical research remained relatively the same or even reduced over the past decade.

Funding Mechanisms Around the World
While research presents a necessary tool for humanity's development, it introduces a dilemma to those trying to regulate research funding. Funding agencies face great difficulties in setting universal performance measurements for funded research projects. This fact can be easily demonstrated through considering the difference in the objectives between that of a scientist and a government.
In the case of biomedical research, researchers tend to focus more about producing new knowledge rather than how this knowledge could be utilized in the future. Scientists' career development is tied to becoming more recognized and learning new techniques that would permit them to move towards their selected career path [23]. On the other hand, governments expect research to help create new knowledge, induce the utilization of this new knowledge in the society, and help the community to learn about how research is conducted [10]. Murdok and Kospell have tried to analyze this problem using the principal-agent problem [24]. This problem usually arises when the owner of a business (funding agency in this case) and the agent managing the business (researchers or research institutes) have different objectives [10]. To address this issue, funding agencies often need to create funding contracts that ensure that the scientists are using research funds to work on the objectives of the agencies. Research funding contracts may differ between funding agencies depending on many factors including both the objectives of the funding agency and the relationship of the funding agency with the proposal submitting entity.
According to the Organization for Economic Co-operation and Development (OECD), funding contracts can be grouped into three major types: 1. Ex-post funding contracts are contracts where the funding agencies "outsource control on who does the research and what research is done to the research performing organizations but decides to monitor the efforts and research output closely." Funding agencies in these scenarios do not directly control the size of each entity's budget but decides how to reward them based on the outcomes of their efforts. 2. Ex-ant funding contracts are often used with competitive grants contracts. In ex-ante funding contracts, the funding agency selects what research projects are conducted and the researchers who conduct them. Funding in these contracts is not directly dependent on the output. Such contracts are often associated with inefficiencies due to "suboptimal allocations". 3. Fixed funding contracts are contracts where the funding agency decides the budget of a research institute regardless of how it performs on measurable research outcomes. The funding agency in such cases does not monitor performance of the research institute and leaves control of the what research projects are performed, their budgets, and the identity of the research performing these projects to the research institute [10]. The problem with using simple measurements to monitor the performance of researchers is that the contracts need to state the selected measurements. Funding agencies around the world use different ways to review researchers and research institutes performance. However, as discussed earlier, scientific research breakthroughs do not always lead to revenues. This had led funding agencies to introduce new processes to help researchers focus their efforts on the funding agencies' goals. The discussion above had put a lot of focus on developed countries efforts to evaluate scientific research performance. In the upcoming sections, we will consider Qatar as a developing country establishing a knowledge-based economy. In order to properly grasp the research culture in Qatar we will briefly look at the status of healthcare in Qatar.

Healthcare in Qatar
To properly design a research funding mechanism in Qatar, a country that is dependent on its oil and gas exports, we first need to understand the common problems that biomedical research funding faces, the relationship between knowledge-based economy and research, and finally determine what indicators should funding agencies in Qatar may look for in research projects so that the biomedical research funding program in Qatar would become sustainable.
Investment in research is necessary for the development of knowledge economy that can attract foreign investment since research and development are an important source for finding or creating competitive advantages in industries. To attract high quality minds that can support such an economy, it is imperative to have a good educational and healthcare systems. In addition to good educational systems which is reflected in the private schools in Qatar and Education City branch campuses, the healthcare system in Qatar needs to provide quality that is on par of that offered in the West. This issue is reflected by the QNRS and the National Health Strategy (NHS) [25].
The healthcare system in Qatar has undergone great development over the past few years. Qatar spends 2.2% of its GDP on health, and the populations' private spending on health has exceeded most countries in the GCC including UAE, Kuwait [32]. In 2015, the life expectancy in Qatar was 77 years for males and 80 for females, and according to the WHO the incidence of infectious diseases such as HIV or tuberculosis is extremely low. The most fatal of these diseases is cardiovascular diseases (18%) followed by diabetes which accounts for 9% of mortalities in Qatar. However, it still faces some challenges. According to the WHO, in 2014 42.3% of the Qatari population were obese. Additionally, the affluent lifestyle of Qataris has contributed to the propagation of non-communicable diseases which cause 69% of the of mortalities in Qatar. Additionally, there remains a skill gap in recruitment of medical personnel necessary for managing such facilities. Despite these challenges, Qatar has a great advantage when it comes to biomedical research.
Biomedical research is necessary for finding ways to efficiently reduce the burden of healthcare either through reducing spending on healthcare management or through finding cures for disease that represent a burden on the community. Further, Qatar's location and diverse population creates fertile ground for medical research since Qatar hosts many nationalities and will become even more diverse as the country continues to expand and attract talents from all over the world. Finally, a good quality healthcare system can support projects supported by the Qatar Foundation that aim to improve the quality of live in impoverished countries.  [27]. In 2012, the QNRF launched the Qatar National Research Strategy (QNRS) [28]. In line with the QNV 2030, QNRS's vision was for Qatar to become "a leading center for research and development excellence and innovation" [28]. The QNRS discusses five main pillars:

Strategies for Creating KBE
1. Enterprise-Wide Pillar 2. Energy and Environment Pillar 3. Computer Sciences and Information Technology Pillar 4. Health Pillar 5. Social Sciences, Arts, and Humanities Pillar For research in Qatar, it is important that biomedical research funding addresses the challenges faced when new drugs enter their target markets. It can be inferred from governmental regulations managing the entry of drugs and new medical devices in the United States (which often require a lot of research in basic research including animal research) that "direct" value of funded research projects is often distributed over many products and services. When the value of biomedical research is questioned, it is not unusual to turn to the direct profits made through producing new drugs that result from a research project. However, the economic value of biomedical research can be further enhanced by considering the indirect benefits it makes. We will focus on four major outputs to biomedical research: 1. Development and improvements to existing policies, laws and procedures; 2. Development and improvements to existing medical procedures and medications; 3. Experts in biomedical research that can provide technical expertise; 4. Innovation

Measuring Performance of Biomedical Research Projects
As discussed above, policy makers and researchers do not often interact [29]. As a result, funding agencies often use measurements that could be understood by the average individual to demonstrate the impact of cutting-edge research. The lack of communication between researchers and policy makers makes it even more difficult for policy makers to fully utilize up to date research in constructing policies and healthcare systems that are most efficient. Additionally, researchers do not often agree on how research outcomes, which are usually communicated towards other scientists, are best interpreted [29]. As a result, policy makers and the taxpayers do not see the ROI on scientific research and only have the word of the scientists on this matter, who have depended on public research funds for their livelihood.
Funding agency efforts are currently moving towards improving biomedical research efficiency and have been focused on increasing the number of discoveries made in targeted areas. Evidence of this can be seen in the increase of the number of grants offered to research specific diseases or translational medicine, which has been the case with countries like the United States [30]. However, the relationship between research and innovation is not simple [2] and it is not unusual for research projects to pave the way for innovations in areas that are different from the original objectives. In addition to social scientists, those working within the biomedical field have also attempted to find measurements for the effect of biomedical research [2]. Nonetheless, these efforts have mainly focused on improving the productivity of research considering scientists as the only stakeholders. Comparing the outcomes of science to the communities served has not yet been formally studied.

Conclusion
The impact of scientific research on economies can be demonstrated by looking at the performance of knowledge-based economies in the west. However, even in those countries, determining the best way to measure this impact remains difficult. This problem is further demonstrated in developing countries that lack the necessary administrative infrastructure to utilize the outcomes of scientific research. Further, transforming economies that are dependent on natural resources requires significant changes to most of the institutions in these countries. Finally, political instability in the MENA region adds to the difficulties of conducting research. To address these issues, developing countries such as Qatar would need to consider a different approach to demonstrate and measure the impact of their research investment so that it encourages the local population to take up new scientific findings and help in bridging the gap between their countries and those of developed countries. Such strategies definitely need to consider the experience of developed countries; however, they must take into consideration the different challenges faced by developing economies including political instability, lack of administrative and regulatory infrastructure, lack of local talent expertise necessary to ensure the sustainability of such an economy, and finally processes that would encourage both researchers and policy makers to the incorporate of new research findings into day to day business practices.