Using IT for Non-Hierarchical Performance Assessment: New Ways to Develop Professionals in Health Care
Manzo Fabio1, *, Agrifoglio Rocco2, Salvatore Domenico1, 2
1IRCCS S.D.N., Naples, Italy
2Department of Management, Accounting and Economics, University of Naples "Parthenope", Naples, Italy
To cite this article:
Manzo Fabio, Agrifoglio Rocco, Salvatore Domenico. Using IT for Non-Hierarchical Performance Assessment: New Ways to Develop Professionals in Health Care. Journal of Human Resource Management. Special Issue: Challenges and Opportunities in the Performance Measurement and Control Systems of Human Resources Management for the Services Industry. Vol. 3, No. 2-1, 2015, pp. 11-19. doi: 10.11648/j.jhrm.s.2015030201.12
Abstract: The aim of this article is to describe how technological innovations are being used to address the specific challenges of assessing health care professionals’ performance and to discuss the theoretical implications of these innovations for human resources management. Performance management is specifically challenging for organizations employing skilled professionals, such as physicians, due to the complexity in defining performance and to the autonomy that professionals enjoy. Professionalism of employees implies that supervisors cannot evaluate adequately their subordinates’ performance and, thus, alternative assessment systems are necessary. This explains why in health care schemes such as 360-degree feedback and peer review are increasingly used. We define these alternative assessment systems as "non-hierarchical performance assessment" and report exploratory data from an Internet search of performance assessment Information Technology (IT) solutions offered by IT providers. We use a classification of these IT solutions in order to identify and discuss new ways for assessing and for developing professionals in health care. We conclude by discussing differences and similarities between 360-degree feedback and peer review and by identifying current trends and possible future developments about how IT enables new assessment systems in this industry.
Keywords: Performance Management, Appraisal, IT
Performance management is a fundamental human resource management practice. Using this practice, organizations can identify their employees’ strengths and weaknesses and motivate them to develop their competencies. Performance assessment, a central part of the performance management process, is challenging in service industries and, in particular, in health care because performances are more complex and less measurable than in most other industries. In health care, the use of performance assessment practices is currently rapidly increasing, in our view thanks to three parallel developments:
1) human resources scholars and practitioners have experience with 360-degree feedback which is a form of performance management involving less hierarchy than traditional supervisor assessment;
2) knowledge specialization in health care has reached levels such that the functioning of traditional routines of professional control is challenged;
3) IT can now enable frequent, locally dispersed and privacy-savvy performance appraisal.
In the first part of this article, we discuss these three trends. Then we analyze the features offered by a number of software developed by IT vendors to support 360-degree feedback and peer review; the scope of this limited empirical analysis is exploratory and we only use these data to generate a classification employed in the discussion section of the article. Therefore, in the discussion we analyze the implications that the combined effects of the three above-mentioned trends can have for performance management in health care and, in general, in knowledge-intensive industries.
2. Non-Hierarchical Performance Assessment: 360-Degree Feedback and Peer Review
In most organizations using performance management practices, each employee is assessed by his or her direct supervisor, according to the organizational chart; in this way, each worker receives a feedback from an individual that can customarily observe his job and has often the necessary expertise to evaluate the other’s performance. In health care organizations, because of physicians’ high professionalism and autonomy, supervisors often cannot monitor and evaluate adequately their performance, so traditional hierarchical assessment fails.
We suggest the distinction between "hierarchical" and "non-hierarchical performance assessment" and categorize both 360-degree feedback and peer review as non-hierarchical performance assessment practices. Viewing 360-degree feedback and peer review from a unified perspective allows, in our view, a better understanding of performance management in cases where the traditional hierarchical performance assessment is unfeasible or unwanted.
2.1. 360-Degree Feedback
In a 360-degree feedback scheme (also known as multi-rater feedback, multi source feedback, or multi source assessment), appraisal comes from different assessors that are combined to obtain a more complete and more reliable feedback; we could say that feedback is given by members of an employee's immediate work circle. In fact, most often, 360-degree feedback will include feedback from employee's subordinates, colleagues, supervisor, customers, suppliers as well as self-evaluation. Since the feedback is given also by people that have a different job compared to the employee being assessed, 360-degree is used in particular to judge behavior and "soft" competencies (such as teamwork or interpersonal relationships and communication skills), and not to evaluate job performance or technical skills . Individuals tend to better accept a feedback, even if negative, and to work to improve their performance when appraisal comes from different sources, because she or he considers this more reliable [2,3]. However, this behavior also depends on the assessed personality and on the evaluation purpose. In fact, accepting a negative feedback is easier when it is formative and so it is used to personal and professional development. Instead, fewer individuals accept a negative feedback if it is linked with payment or awards. In the last case, the process is defined as summative assessment . To facilitate improvement, an organization could implement some post-feedback interventions, such as coaching [2,5], by which it could drive the individuals to achieve the expected levels of performance.
2.2. Peer Review Programs
A Peer Review Program (PRP) is an evaluation scheme, often used in medicine and in academia, in which each individual is assessed by others with the same professional specialization. It can be used for jobs with high level of expertise, such as physicians. Through this tool, appraisal is performed by an individual with a similar background of competence and in a similar position. Peer review is used to evaluate the results of a work and it is the most commonly used method for assessing performance by physician’s peers in order to identify opportunities for additional education, error reduction, and self-improvement . An adequate PRP should be fair and unbiased, so participants choice and tools arrangement are critical. Assessors and assessed anonymity could improve results quality . In fact, if an individual does not know the evaluation "subject", he/she has to inevitably focus on evaluation "object", so feedback is not affected by personal issues between the concerned individuals. However, other authors highlighted that it is more likely that a professional accepts an evaluation from a known individual. Therefore the choice between blinded or open PRP depends on tool purposes; anonymous feedback could be useful when peer review is performed to improve provision quality and for individuals’ professional development (formative PRP), while nominal one is suitable if assessment is linked with workers’ rewards and penalties (summative PRP), since in this case each individual would accept negative judgments only if they come from well-regarded professionals. A PRP, just as any other performance management system, can be judgmental and inspection oriented or development-oriented . Another categorization is between proactive and reactive PRP : proactive PRP are programs in which evaluation objects and subjects are chosen routinely and randomly from the daily workflow and there are often blinded; reactive PRP, instead, is performed ex post when a discrepant report bobs up, to analyze the quality of the achieved output.
2.3. 360-Degree Feedback and PRPs: Main Similarities and Differences
Different assessors can be selected to perform non-hierarchical assessment, with different advantages. Co-workers are the closest to the assessed and so they can monitor the other’s on the job behavior. Instead, the word "peer" as used in peer review process does not identify co-workers but individuals with the same professionalism who may also be working in other organizations; peers know in depth the other’s job and so they can judge adequately the achieved performance, while co-workers may not have the right knowledge to assess them. Peers could evaluate their peers output, even if they are far, on condition that this output can be identified easily and univocally. PRPs, as a result, focus on the evaluation of results whereas 360-degree feedback is best suited to judge behaviors. Therefore, to some extent, 360-degree feedback and peer review are complementary perspectives.
Several studies showed that one of the most recurring cause of both failure of 360-degree feedback and peer review systems is that they are very time-consuming [6,9]. IT offers a way to address this issue and, for this reason, the supply of new performance assessment software is currently booming.
3. Professional Control, Professional Development and Non-Hierarchical Assessment
Professionals are defined as workers who, among other things, adhere to a set of professional norms  and enjoy autonomy from supervision . Non-hierarchical assessment is highly relevant in professional contexts, such as medicine, because of the nature of knowledge applied by professionals and of the implications of assessment for professional control and professional development. Moreover, current trends in the organizations of health care systems make increasingly important IT systems supporting non-hierarchical assessment.
Most functionalist theories of professional control are based on the "argument from knowledge" which emphasizes the arbitrary nature of applied professional expertise . Freidson  considers professionalism as a form of social organization positing that workers with specialized and complex knowledge should organize and control their own work, without directives from management or the influence of free markets. Only someone who knows about the activity to be performed can organize and control work; therefore professionals, for instance medical doctors, cannot be controlled by clients or managers: only peers with the same specialization master the complex and specialized knowledge necessary to understand and evaluate colleagues’ behavior and results. Peer-control among doctors is at first induced through medical education, a training process that also involves sending new recruits through an adult socialization process . During this stage of academic training, hierarchical evaluation systems are employed much more than peer review . After becoming a doctor, attending rounds, departments meetings and clinical case discussions represent a structured learning and peer review tools often used for development and control of hospitals staff. The broader local, national and international medical communities are also functional to coordination and control, for example, by organizing meetings and by developing guidelines. In most countries, a local college of physicians has some levels of legal jurisdiction on medical practice and doctors’ behavior. Compared to control by management or to market incentives peer control is probably weaker and allows more deviance. For example, medical literature has always documented large and irrational variations in medical practice across regions, hospitals and physician practices for almost every condition and procedure . Self-regulation by professional communities is sometimes considered an inefficient result of power dynamics among professional groups and the wider society. Insurance companies, governments, managers and patients often try to reduce this doctors’ autonomy and sociologists often classify these attempts as "de-professionalization", or "corporatization" or "marketization" of medicine . Most of the time, doctors strategically adapt to these attempts of non-peer control and their autonomy is rarely significantly reduced .
The centrality of knowledge to professional practice also has implications for professional development. Most countries have implemented continuing medical education (CME) programs to help doctors to maintain competence and learn about new and developing areas of their field. In most cases, training is the only human resources development practice used for doctors after the initial certification. Performance feedback, instead, can have an important developmental purpose. Feedback can be used to coach employees and improve performance on an ongoing basis. This feedback allows to identify strengths and weaknesses as well as the causes for performance deficiencies (which could be due to individual, group, or contextual factors) .
Moreover, current trends in the organizations of health care systems make increasingly important IT systems supporting non-hierarchical evaluation. In many countries, the number of hospital beds is shrinking because in-hospital stays are becoming shorter and focusing on acute needs. Primary and community care tend to grow; more health services are been provided at patients’ home, in community medical centers or in non-hospital rehabilitation facilities. This trend is induced by the advancement of medical knowledge and the increasing prevalence of chronic conditions. Less doctors working in the same physical building (the hospital) reduce the opportunities for interactions and the strength of unstructured peer-control. Another important current trend in the organizations of care is the ever-increasing importance of interdisciplinary collaboration. Finally, health care organizations are also investing in IT systems that are able to store and process huge quantities of data about patients and the provided service. Therefore, the geographical dispersion of doctors deriving from the growing importance of primary and community care, the evolution of medical science and the interdisciplinary collaboration required to apply it, and the wealth of data that is becoming available, open new possibilities to structure professional control and professional development in new ways.
4. Using IT for Assessing Non-Hierarchical Performance in Health Care Industry
Non-hierarchical performance assessment systems that have been adopted over time in health care for assessing behavior and output of physicians can be traditional, such as paper-based evaluation systems, or electronic, such as web-based evaluation system and mobile-based evaluation system.
Paper-based evaluation systems (also known as pen-and-paper evaluation systems) are those traditional systems that use a survey or a questionnaire to gather information in several areas. Electronic evaluation systems allow to assess physicians performance using IT. As suggested by Rosenberg and colleagues , electronic evaluation systems have many advantages when compared to more traditional paper-based systems. For instance, data collected using an electronic evaluation system can be automatically entered into a database to available to the assessed worker, positively impact the development program in many ways. Moreover, electronic evaluation systems allow to reduce costs of assessment and to evaluate data making easier the time-consuming activities related to assessment. In this regard, Archer and colleagues’ research  showed a successful system in which assessments were centrally scanned into a computer for summary and analysis. Moreover, apart these recognized advantages, a previous research has compared traditional with electronic evaluation systems showing that the method did not influence the consistency of raters . web-based and mobile-based evaluation systems multiply the advantages that the first electronic systems have brought. A web-based evaluation system is an online electronic tool for measuring physician’s work outcomes. Like paper-based systems, this tool also allows to assess physicians’ performance by using a survey or a questionnaire, even if data gathering and analysis are different. Some software houses have developed ad-hoc solutions for evaluating physicians’ performance by using online surveys or questionnaires within health care. Other health care organizations, instead, have designed in-house their software for evaluating performance rather than using commercial (out-of-the-box) packages . With reference to evaluation techniques, these web-based systems allow to perform both 360-degree evaluation and peer review [1,6]. Although many health care organizations use web-based evaluation systems for assessing physicians’ skills, behaviors and work outcomes, the health care literature has paid little attention on this topic. In particular, a "web-based 360-degree evaluation tool is a feasible way to obtain reliable ratings from rehabilitation staff about resident behaviors" [23, p. 845]. On the contrary, as Mahgerefteh and colleagues [6, p. 1225] suggested, "online peer review systems include secure, personalized Web-based log-in, electronic data entry, and automated routing of performance data to the appropriate databases and quality assurance officials".
Mobile-based evaluation systems are the last frontier of evaluation systems used, also in health care, for measuring and assessing the worker’s performances. These systems are useful tools for performance appraisal, real-time data sharing, and data analysis within health care organizations and may also allow inter-organizational assessment processes. Compared to other evaluation systems, mobile-based systems enable health care organizations and managers to perform performance appraisal in a more effective and efficient way, so reducing costs and time, and to obtain a higher quality and more complete data availability.
5. Empirical Data about IT for Non-Hierarchical Assessment in Health Care Industry
In order to explore how IT-based non-hierarchical assessment systems are used in health care, we conducted an Internet search during November and December 2014 about the features currently offered by IT providers that advertise their software as systems for 360 feedback or peer review. In this article we do not use the collected data to test hypotheses, but we use data to generate a categorization. Categorization is increasingly being mentioned as a theoretical development tool that could help management scholars to take into account context, such as the type of industry, in their theory development [24,25].
Using the search engine "Google" we designed a broad search process by using various keyword combinations such as "360 feedback", "peer review", "system", and "health care". In particular, searching the keyword combinations "360 feedback"+"system"+"health care" and "peer review"+"system"+"health care", we found respectively 22,200 and 11,000,000 results. Analyzing the results, we noted that the use of keyword "system" lead to a large volume of irrelevant issues. Thus, we chose to change the term "system" with "tool" or "software" for enhancing the relevance of results. However, this choice led to the exclusion of paper-based evaluation systems from results. Although paper-based evaluation systems may still be utilized in health care organizations for performing non-hierarchical assessment, we decided to exclude them from search for two reasons. First, traditional evaluation systems are increasingly replaced by electronic evaluation systems. Moreover, including traditional evaluation systems in search could lead to a large volume of irrelevant results.
Then, we performed again the search using the keyword combinations "360 feedback"+"software"+"health care" and "peer review"+"software"+"health care", founding respectively 40,000 (of which 283 most relevant) and 436,000 (of which 400 most relevant) results.
Finally, we have proceeded with a screening process of electronic evaluation systems search returned. Duplicates systems and systems that were not related to our requirements were deleted. After the screening process, the final dataset for the subsequent analysis consisted of 77 non-hierarchical evaluation systems. The first author analyzed all the information available to users on each vendor’s website and classified each software according to criteria commonly agreed among the authors on a sample subset. The information on the website usually consists in videos, screenshots, features lists, and case studies descriptions.
6. Results and Discussion
As shown in Table 1 and 2, we found 77 tools for non-hierarchical assessment in health care, of which 62 are 360-degree feedback software, 14 are peer review software and just one can be used for both purposes. Results of Internet search were reported in Appendix 1.
Dataset results were examined for each instrument identified in terms of pervasiveness, specificity and width of technology used. Based on the Information Systems literature, we chose these three criteria aimed to understanding "what" and "how" IT-based non-hierarchical assessment systems are nowadays used in health care. In particular, about pervasiveness, we could distinguish between web-based and mobile-based evaluation systems, according to the possibility of perform assessment through different devices. Moreover, about specificity, we classified software as "general", for software that can be used for any industry, "customized", when there is already available a version for health care, and "ad-hoc systems", if the solutions has been built specifically for health care industry. About width, instead, we distinguished software used only within the organization’s boundaries from software that could be used also to assess individuals between different firms, as well as software that could be used both within and between organizations.
With reference to 360-degree feedback, only six of 62 tools can be considered to have a high level of pervasiveness, including mobile devices, but each of them are low in specificity. Moreover, 54 of 62 results are general tools that can be used for different industries, through the high allowed customization. Then, some of them are simple platforms by which users can build an ad-hoc survey, without any further features. Just three software are developed specifically for health care, whereas six include a default scheme for this industry, because of they are already bought by health care clients or by producers sales choice. All 62 360-degree feedback software can be used only within the organization.
With reference to peer review, instead, all 15 tools are web-based, but no one is mobile based. Among them, two are low in specificity, whereas the other 13 are all ad-hoc solutions for health care industry. In particular, then, five of 13 results have an extremely high level of specificity, as they can be used just by Radiologists, often taking advantage of connection with radiology-specific IT, such as RIS and PACS systems. About width, 10 of 15 peer review software can be used within the organization, two of them only between different firms, while last three can be used both within and between companies. Besides, four out of five tools with a broad width are for radiologists, whereas only one can be used for any health care specialization. Radiology, because of the type of activities that doctors perform, is a specialty highly dependent on IT. Almost all the activities performed by professionals in this specialty involve producing data that are stored on very advanced IT systems and professionals tend to spend in front of a PC a very high percentage of their time when compared to doctors in other specialties.
All research results and related analysis are shown in following tables.
Many software houses have designed ad-hoc solutions for physicians’ assessment in health care industry in order to make more reliable and suitable data processing. Combining pervasiveness and specificity of technology, dataset results have shown that vendors have not designed ad-hoc and customized solutions for non-hierarchical assessment in health care via mobile. Although mobile applications in health care can be recognized as emerging and enabling technologies because they transfer vital information about patient to physicians and/or to hospital, they also play a crucial role in non-hierarchical assessment of health care personnel. Mobile tools, instead, enable physicians to evaluate performance of colleagues, as well as to provide feedback and suggestions, from anywhere, at any time in their daily ward rounds. Like other industries, mobile ad-hoc systems designed for assessing performance of physicians in health care industry, as well as customized solutions, can enable managers and organizations to a more complete data availability and a more efficient and effective management of services provided.
Moreover, combining specificity and width of usage of technology, findings have shown that ad-hoc solutions allowing non-hierarchical assessing between colleagues from different health care organizations were developed just for peer review. Vendors have mainly designed software for 360-degree assessment within health care organization, without paying much attention to different needs of physicians and human resources managers that are changing over time. Since medical and economic needs in the health care industry are leading more and more towards the specialization of physicians, organizations often have only one or a few physicians with similar clinical knowledge and skills. Within these health care organizations, above all if are small sized, non-hierarchical assessment between colleagues is not always possible due to a few number of physicians and health care workers with similar skills. These organizations, even more than those of large size, should be more willing to overcome organizational boundaries for performing non-hierarchical assessment, so allowing physicians to obtain useful feedback and suggestions for improving their work and for augmenting their knowledge.
Finally, combining pervasiveness and width of usage of technology, findings have shown that ad-hoc and customized solutions for both 360° and peer review assessment from within and between health care organizations via mobile were not designed by vendors. In particular, unlike web-based systems, as findings have shown, these mobile applications were just designed for assessing physicians within the same organization, while the performance assessment via mobile among physicians working in different health care organizations was not provided. Mobile performance assessment systems can enable physicians and health care personnel to read real-time data about performance of far away colleagues and to give feedback and suggestions from anywhere and at anytime. Often workers of a health care organization do not have enough skills for assessing performance of colleagues. In many knowledge-based industries, and health care in particular, the comparison between colleagues allows to overcome the self-referentialism boundaries and, thus, it has recognized as crucial for working practice improvement and knowledge building. Using health care mobile applications for workers’ performance assessment from outside, physicians and managers can get various benefits such as decreasing clinical errors, knowledge and skills improvement, and real-time access to database.
This study contributes to literature by framing peer review and 360° feedback as two tools of the same process, which we named "non-hierarchical assessment" which, in turn should be framed in a broader performance management process. Personnel development is a key human resource practice in any industry and is even more important in professional contexts such as health care. PRPs, which are central to the everyday practice of medicine and academia, are rarely considered by the human resource management literature. By suggesting that a PRP is a non-hierarchical assessment practice we widen the available tools to human resource management. The industry-specific and theoretical experience gained in health care with PRP can be generalized and applied to most other industries.
We highlighted that IT is opening new practical ways of implementing "non-hierarchical assessment" and big and small companies are experimenting with these possibilities showing theoretically-relevant opportunities. Thanks to web-based and mobile technology, IT helps a lot in addressing two big problems of performance assessment: the resources necessary to administer the system and the delay between the actual performance and the related feedback.
The IT solutions we found through our Internet search vary in their specificity. We believe that health care-specific solutions make life easier for health care human resources departments, many of whom may have little experience with physicians’ assessment. Each organizations should be able to customize the solution according to its needs and vendors should advice not only on technological but also on the managerial issues.
The width of the IT solutions we found is mainly focused within the boundaries of a single organization. Our integrated perspective on 360° feedback and PRP and our classification shows that non-hierarchical assessment can be between organizational borders. Inter-organizational non-hierarchical assessment can be extremely useful in this context, because of the increasing importance of primary and community care and the exponential production of new medical knowledge. Peers able to assess a very specialized physician can be more easily found in other locations and in other organizations.
The main limitation of this paper is in its empirical data. We make clear that this is mainly a theoretical paper, we used a survey of the services on the market only to generate new insights specific to the professional services context, not to test hypotheses. Future studies may first qualitatively describe the actual use of non-hierarchical assessment in health care organizations and then design a quantitative study.
Moreover, health care is a big industry and the nature of the activities performed by professionals is heterogeneous. As shown by the presence of radiology-specific solutions in our Internet search, there may be interesting differences related to organizational factors such as specialization area. Another interesting organizational factor that we have mentioned is the size of the organization: IT makes available to small and medium-sized organizations human resources practices that previously could only be implemented in big organizations with many employees and specialized human resources staff. The differences among public and private health care organization would also be interesting to analyze. Future empirical studies may investigate how these organizational factors intervene in the use of IT for non-hierarchical assessment.
Professional contexts, and health care which is the most stereotypical industry known for employing professionals, are an "extreme" case where non-hierarchical assessment is useful, but to some degrees non-hierarchical assessment can be useful in many other contexts where employees enjoy autonomy from supervision and the developmental function of performance management is relevant.
|20/20 Insight GOLD||Resource Unlimited Group (USA)||X||X||X||X|
|360 Degree Feedback||3D Group (USA)||X||X||X||X|
|360 Degree Feedback||Cognology (Australia)||X||X||X||X|
|360 Degree Feedback||Full Circle Feedback (Australia)||X||X||X||X|
|360 Degree Feedback||PeopleStreme Pty. Ltd.(Australia)||X||X||X||X|
|360 Degree Feedback||TalentGuard, Inc. (USA)||X||X||X||X|
|360 degree feedback assessment tool||TENCompetence Foundation (UK)||X||X||X||X|
|360 Degree Feedback Assessments||INSIGHT Mirror 360 (USA)||X||X||X||X|
|360 Degree Feedback Pro Pack||Adaptive Business Management Systems Ltd. (UK)||X||X||X||X|
|360 Degree Feedback tool||SpidergapSupport (UK)||X||X||X||X|
|360 Feedback||Progreso HR Software (Belgium)||X||X||X||X|
|360 Feedback / Multi-rater tool||PeopleFluent (USA)||X||X||X||X|
|360 Feedback Eval||Atomic Design, LLC. (USA)||X||X||X||X|
|360 Feedback Survey||Quantum's HR Tech Services (USA)||X||X||X||X|
|360 Leader||CDR Assessment Group Inc. (USA)||X||X||X||X|
|360 Revalidation Software||Cyber Interactive (UK)||X||X||X||X|
|360 Software||Lumus360 (UK)||X||X||X||X|
|360-feedback||TribeHR Corp. (USA)||X||X||X||X|
|360° Degree Feedback||Grapevine Evaluation (Canada)||X||X||X||X|
|360° Feedback Assessment||The Booth Company (USA)||X||X||X||X|
|360° Feedback Alveria||Alveria. Consulting,Training and Solutions (Italy)||X||X||X||X|
|Acesis Peer Review||Acesis Inc. (USA)||X||X||X||X|
|Allibo Perf||Alliance Software s.r.l. (Italy)||X||X||X||X|
|Appraisal 360||Third Eye Resolutions Ltd. (UK)||X||X||X||X|
|Ascendus||Ascendus Technologies Inc. (USA/India)||X||X||X||X|
|Blue/360||eXplorance Inc. (USA)||X||X||X||X|
|Carbon 360||Element 78 Solutions Ltd. (UK)||X||X||X||X|
|CCI Surveys||CCI Surveys Internationals (USA/Canada)||X||X||X||X|
|Cezanne||Cezanne HR s.r.l. (Italy/UK)||X||X||X||X|
|CR 360||CR Systems Limited (UK)||X||X||X||X|
|Custom View 360||Envisia Learning (USA)||X||X||X||X|
|Data Star||Data Star Inc. (USA)||X||X||X||X|
|DiaShare Quality||Real Time Medical (Canada)||X||X||X||X|
|e360||CEO Infotech Pvt. Ltd. (India)||X||X||X||X|
|EPR||Medkinetics LLC (USA)||X||X||X||X|
|eRADPEER||The American College of Radiology (USA)||X||X||X||X|
|ExecuSurv’s online survey||ExecuSurv (USA)||X||X||X||X|
|Feedback Online360||Hang Lung House (China)||X||X||X||X|
|Focal 360||Custom Insight (USA)||X||X||X||X|
|Gestionale HR Inaz||Inaz srl (Italy)||X||X||X||X|
|H1 Hrms||EBC Consulting (Italy)||X||X||X||X|
|Halogen 360" Multirater||Halogen Software Inc. (Canada)||X||X||X||X|
|Hr Infinity||Zucchetti s.p.a. (Italy)||X||X||X||X|
|Intelerad Peer Review||Intelerad Medical Systems Incorporated (USA)||X||X||X||X|
|LF 360 Degree Feedback||NHS Leadership Academy (UK)||X||X||X||X|
|MSOW Peer Review Manager||Morrisey Associates (USA)||X||X||X||X|
|Online 360 Degree Feedback||Bowland Solutions (USA)||X||X||X||X|
|Online 360 Survey & Appraisal||Arc Software Consultancy (UK)||X||X||X||X|
|Online 360-Degree Feedback||Echospan Inc. (USA)||X||X||X||X|
|Online 360Degree.it System||Netready Ltd. (UK)||X||X||X||X|
|OpenConf||Zakon Group LLC. (USA)||X||X||X||X|
|Orange HRM Professional||Orange HRM Inc (USA)||X||X||X||X|
|PeerPoint||AllMed Healthcare Management, Inc. (USA)||X||X||X||X|
|PeopleSoft Enterprise ePerformance||Oracle (USA)||X||X||X||X|
|Perform Software||USR Infotech Solutions Pvt Ltd. (Africa)||X||X||X||X|
|PowerScribe 360||Nuance Communications, Inc.(USA)||X||X||X||X|
|PREP-MS Software||QA to QI Llc (USA)||X||X||X||X|
|Primalogik 360||Primalogik Software Solutions Inc. (Canada)||X||X||X||X|
|Primordial||Primordial Inc, (USA)||X||X||X||X|
|PsyTech 360||PsyJob di Fabio Biancalani (Italy)||X||X||X||X|
|Pulse 360 Surveys||Physicians Development Program (USA)||X||X||X||X|
|Qualtrics 360||Qualtrics LLC (USA)||X||X||X||X|
|Quick360||Reactive 360 (UK)||X||X||X||X|
|RCA - Root Cause Analysis||CCD Health System (Canada)||X||X||X||X|
|RL6: Peer Review||RL Solutions (Canada)||X||X||X||X|
|SelfStir 360 Degree Feedback||SelfStir Limited (UK)||X||X||X||X|
|Small Improvements||Small Improvements Software GmbH (Germany)||X||X||X||X|
|Smart360||Appraisal Smart Pty Ltd (Australia)||X||X||X||X|
|SuccessFactors 360 Reviews||SuccessFactors (USA) –SAP Group||X||X||X||X|
|SumTotal Talent-360 Degree Feedback||SumTotal Systems, LLC. (USA)||X||X||X||X|
|SurveyTracker||Training Technologies, Inc. (USA)||X||X||X||X|
|Talent360||Head Light Ltd (USA)||X||X||X||X|
|Trak Star Professional||Promantek (USA)||X||X||X||X|
|vm360 Feedback||Vision Metrics (Holland/USA)||X||X||X||X|