Mobile Phone Automated Messaging Service’s and the Fulfilment of Planned Clinical Appointments: Lessons from an Ebola Clinical Study in Kambia District, Sierra Leone

Introduction: The Ebola vaccine Deployment, Acceptance, and Compliance (EBODAC) project was established to develop a communication and community engagement strategy, including the development of appropriate technology, to maximize Ebola vaccination impact in the targeted population both in support of clinical trials and in the broader vaccination program beyond the clinical trial. EBODAC supported the EBOVAC-Salone trial in Kambia district in Sierra Leone to ensure that the novel prime-boost vaccine regimen is well accepted and successfully used in the context of clinical trials; while also preparing to maximize the impact of the potential deployment of a future Ebola vaccination program. The Mobile Technology for Community Health (MOTECH) platform was utilized and customized to help ensure compliance with the prime-boost vaccination regimen. Its main role was to support the communications and reminders surrounding the Ebola vaccine trial through an automated message service to participants’ mobile phones as Interactive Voice Responses (IVR) and SMS (Short Message Service) messages. Methods: The study was cross-sectional and adopted qualitative and quantitative research methods. The design was constructed to provide a representation of different user segments. The sample size was not constructed to provide statistical differences or detect differences in impact among groups but was simply designed to provide insights into how study participants interacted with the technology. Eligibility criteria involved randomly selecting participants from those that: a) received IVR and SMS and didn’t miss an appointment (booster compliant), b) received IVR and SMS and missed an appointment (booster non-compliant), c) received no IVR and SMS and didn’t miss or missed an appointment (MOTECH non-user). Quantitative information was entered in well-designed data entry templates. All participants’ names were anonymized for data analysis. Results: All the respondents, (n=81) revealed liking both the IVRS and SMS messages and majority of these, 80.2% (n=65) reported that IVR was very easy to understand. Respondents 47.5% (n=45) were comfortable receiving IVR, as opposed to 35.8% (n=29) preferred both text and voice messages. It was also found that 74.1% (n=60) of the respondents who were sent visit reminder messages never missed clinic appointment and 81.5% of those that received information preferred to know how to protect themselves from the disease. Conclusion: The EBOVAC-Salone automated voice messages contributed to valuable learning about areas of improvement for the MOTECH that will be of great importance for future outbreaks and medical interventions.


Introduction
The Ebola Virus Disease outbreak in West Africa exposed and weakened already vulnerable health, education and socio-economic structures and systems in Sierra Leone [1]. The World Health Organization (WHO) figures show that the country was the worst hit in the West African region [2], which attracted support from many global partners. After an initial declaration in November 2015, Sierra Leone announced a new case of Ebola Virus Disease (EVD) in January 2016 and declared it was Ebola-free on March 17, 2016 [3]. The Ebola vaccine Deployment, Acceptance, and Compliance (EBODAC) project was established to develop a communication and community engagement strategy, including the development of appropriate technology, to maximize Ebola vaccination impact in the targeted population both in support of clinical trials and in the broader vaccination program beyond the clinical trial [4].
EBODAC supported the EBOVAC-Salone trial in Kambia district in Sierra Leone to ensure that the novel prime-boost vaccine regimen is well accepted and successfully used in the context of clinical trials; while also preparing to maximize the impact of the potential deployment of a future Ebola vaccination program. EBODAC supported the EBOVAC-Salone clinical trial to evaluate the effectiveness, immunogenicity, and safety of a heterologous prime-boost regimen utilizing Ad26. ZEBOV and MVA-BN-Filo given at a 56-day interval [5]. The project started enrolment in October 2015 and ended in November 2019. The EBODAC project is a consortium of four partners: London School of Hygiene and Tropical Medicine (LSHTM), Janssen, World Vision Ireland and Grameen Foundation.
As part of the technology tools, the project utilized and customized the Mobile Technology for Community Health (MOTECH) platform, an open source enterprise software package designed to help ensure compliance with the primeboost vaccination regimen. MOTECH has an interface that allows sending reminders to the clinical trial participants whenever they are scheduled for a vaccination and scheduled visit as required by the trial. The main role of MOTECH is to support the communications and reminders surrounding the Ebola vaccine trial. MOTECH also facilitates generating daily clinic visit schedule reports for Community Mobilizers to do follow-ups. Reports on the number of people who have been vaccinated (Primer, Booster) and the number of participants who have missed their clinic appointments are also generated by MOTECH amongst others.
The MOTECH platform used an automated message service to communicate with participants. Once registered in the system, the trial participant received reminders to attend their clinic appointments and additional reminders if appointments are missed. These reminders are sent to participants' mobile phones as Interactive Voice Responses (IVR) and SMS (Short Message Service) messages. The IVR messages are the principal means of message delivery. If the IVR fails after three attempts, an SMS is triggered. To ensure a seamless operation of the electronic data capture (eDC), Biometric and MOTECH systems, it was expedient to technically integrate them to synchronize data for the benefit of all stakeholders.
The purpose of this analysis was to understand user experience with EBOVAC-Salone automated messages. The survey contributed to valuable learning about possible areas for improvement for the MOTECH platform that will be of crucial importance for future outbreaks and medical interventions. The main objectives included to explore user satisfaction with EBOVAC-Salone automated message service, understand user attitudes towards, and experiences of the features of EBOVAC-Salone automated message service (length, languages, and female/male voice) and explore challenges using EBOVAC-Salone automated message service, including technical issues. The MOTECH study was conducted in Sierra Leone in the Kambia District which is the hub district for the EBOVAC-Salone project. Kambia is located in the Northern part of Sierra Leone with a population of approximately 343,686 [6]. The District borders the Republic of Guinea to the north, Port Loko District to the south and Bombali District to the east. The ethnic diversity of Kambia is Temne, Susu, Limba, Fula, and Madingo. The geographical area of Kambia is approximately 3108 sq km (1200 sq. miles) and has the highest death rate (infant and maternal mortality) compared to other districts in the country [7].

Study Design
The study was cross-sectional in nature which was crucial for ascertaining the prevalence of the outcome of interest, for the population or subgroups within the population at a given time. Both qualitative and quantitative research methods were adopted. The integration of these two methods provided richer results. The quantitative method was helpful in quantifying and generalizing results in a chosen sample while qualitative was helpful in revealing the underlying reasons, motivations, and insights into addressing the problem.
The study design was constructed to provide a representation of different user segments. The sample size, while relatively small compared to other research designs, was not constructed to provide statistical differences or detect differences in impact among groups but was simply designed to provide insights into how study participants interacted with the technology and any challenges or benefits they experienced.
The random sampling procedure was adapted to select the survey participants from each of the categories above. This was done by assigning auto-generated random numbers to each of the group members using their list in MS Excel. The random numbers were sorted in ascending order taking care to ensure that the selection was expanded to all information pertaining to the individual. The first individuals with the smallest random numbers to make up the required sample from the group were considered for the study. If for any reason the selected individual was unable to participate in the survey, then the next person in line with the smallest random number was considered for the study. The purposive technique was adapted for qualitative study to select informants with advance knowledge and experience in MOTECH.

Target Population
The survey targeted the following Trial participants who received voice messages and SMS for appointment reminders and general information on Ebola Trial participants who did not receive voice messages and SMS for appointment reminders Field Workers who rely on trial participants to receive the voice messages and SMS to attend appointments Clinic management; Support staff and Field Coordinators are dependent on the scheduling and reporting components of the service Participants and eligibility criteria to participate in the study The trial participants were randomly selected from those that: Received IVR and SMS and didn't miss an appointment (booster compliant).
Received IVR and SMS and missed an appointment (booster non-compliant).
Received no IVR and SMS and didn't miss or missed an appointment (MOTECH non-user).
Ethical approval issues around study participants All participants were asked for their consent for participation in the survey or in-depth interviews. The surveys were paper-based and all answers were kept confidential. The study received approval from Sierra Leone Ethics and Scientific Review Committee (SLESRC) after studying the protocol and determined that it presented a minimal risk to subjects and also a confirmation of favourable ethical opinion from the Observational / Interventions Research Ethics Committee of the London School of Hygiene and Tropical Medicine, United Kingdom.

Data management and Interpretation
Data collection for the study was done by the research assistants assigned to a particular group in the study. Quantitative information was entered in well-designed data entry templates. Before data was entered, questionnaires were coded with unique numbers to ease identification during the data cleaning process and analysis. Research assistants and the data entry team were trained on the final survey tools and tool administration. All participants' names were anonymized for data analysis. A simple quantitative analysis was done in Excel showing the percentage of participants choosing particular responses and a simple report written to interpret the data. The KIIs conducted were thematically analyzed and the analysis also incorporated data from EBOVAC-Salone automated message monthly reports.

Measures and the interpretation of the indicators included in the study
Measures for social demographic Data: This focused mainly on the sex of the respondents in the different categories, where male or female. Where the participants received IVRs or SMS and defined as users. Residence of the participants where they live in Urban, peri urban and rural as defined by the Sierra Leonean demographics standards 8 .
Measures for effectiveness of IVRs or SMS: Since the messages are sent through a mobile phone, the study focused on who controls the mobile phone at the household level. Whether the respondents like or dislike the message, understand it, needed assistance in listening or reading the message. What reasons the participants gave for understanding the messages? Appointments: Lessons from an Ebola Clinical Study in Kambia District, Sierra Leone Measures for the utilisation of the IVRs or SMS: Did the respondents have any problem on the gender of the voice used, whether they respondents had a specific time they preferred to listen or receive the message, whether the participants had challenges in receiving the messages and what challenges these were and what type of messages IVR or SMS they preferred and trusted.
Linkages to adherence to clinical Visits: Did the participants meet their appointments due to the message, is the messaging responsible for the timely clinical visits, how many received the messages but still missed appointments and why. Would the IVR and SMS be a good tool for mobilization for visits in future vaccination programs.

Social Demographic Results
The study involved 120 respondents; 51.7% (n=62) being actual trial participants while 48.3% (n=58) being guardians for the trial participants below 18 years. Over half of the respondents, 58.3% (n=70) were males and 41.7% (n=50) were females. Respondents were between the ages of one and 73 years with only 2.5% (n=3) living in the rural areas, 25.8% (n=31) living in the peri-urban and the majority, 71.7% (n=86) residing in the urban setting.
In order to enrich the results from the MOTECH user experience assessment, the survey further focused on other stakeholders who were part of the trial. Therefore, four field workers (3M, 1F) and six field supervisors (5M, 1F) were interviewed. And Five Key Informant (3M, 2F) Interviews conducted focused on the same issues.

Controlling the Respondent's Mobile Phone
For this category, 86.4% (n=70) of the total respondents personally owned or controlled the phone that was used to receive the IVR and the SMS messages from EBOVAC Salone. The phone for the rest of the respondents, 13.6% (n=11) was controlled by either the parent or spouse or family member or another person (could be relative or neighbor) outside the family.
Likes and Dislikes of receiving Reminder messages Respondents in this section were asked to state if they liked or disliked the messages that they received and if they did, what they like or dislike about receiving the messages on their phones. All the respondents, 100% (n=81) revealed liking both the IVRS and SMS messages. Below are the likes that were articulated by the respondents

Understanding the IVR Message
Majority of the respondents, 80.2% (n=65) reported that the IVR was very easy to understand whereas 12.3% (n=10) found the IVR somehow easy to understand. A few of the respondents, 4.9% (n=4) found the IVR very difficult to understand. However, 2.5% of the participants claimed that they have never received any IVR since the phones on which the IVR was sent belonged to a relative living outside the family. The assessment further examined the reasons as to why the IVR was complicated to understand and the table below has a summary of the reasons. Further examination of the respondents focused on whether the respondents required any assistance to listen to the IVR at any time and if yes, what was the reason why they needed support.
The results from the assessment showed that 78.5% (n=62) didn't require any assistance to listen to the IVR, 12.7% (n=10) sometimes required assistance to listen to the IVR and 8.9% (n=7) always required assistance to listen to the IVR. The reasons cited by most of the participants for the need for assistance to listen to the IVR was the fact that they didn't control the phone on which the IVR was received among others as shown in the figure below.

Understanding the SMS Message
Of the respondents surveyed, 72.2% (n=57) acknowledged receiving an SMS while 30.4% (n=24) have never received any SMS. Over half of those respondents found the SMS very easy to understand, 5.3% (n=3) found the SMS somehow easy to understand and 36.8% (n=21) of the respondents stated that the SMS was very difficult to understand. Although only half of the respondents who received an SMS could easily understand it, about 71.9% (n=41) of them didn't require assistance to read the SMS. It's only 22.8% (n=13) and 5.3% (n=3) that always and sometimes needed assistance to read the SMS respectively. The graph below consists of reasons why the SMS was not easy to understand.

Figure 2. Reasons why respondents needed assistance to listen to the IVR.
Investigation on whether other people other than the targeted respondent read the EBOVAC Salone automated messages was affirmatively reported by 19.8% (n=16).

Effect of Gender Voice of the IVR
Majority of the respondents said that the voice of the IVR didn't matter at all but on the same note 17.9% (n=14) reported that the voice mattered a lot to them and 3.8% (n=3) think that the voice of the IVR somehow mattered. Results from further probing on how the gender of the voice of the IVR mattered indicated respondents disliked the male voice because of how deep it was. They asserted that the female voice is better because it sounds polite. Best time of the day to receive the automated messages The survey results indicated 38.5% (n=30) being affected by the time at which the IVR was sent. It is important to note that this was also one of the reasons stated by 16% of the respondents who disliked receiving the IVRs. Reasons as to why the time at which the message was sent mattered are summarized in the table below. Interviews from the respondent's regarding the best time to send the messages indicated morning as the best time reported by the majority of the interviewees, 40% (n=12) followed by 26.7% (n=8) and 23.3% (n=7) who preferred evening and in the afternoon hours respectively. The main difficulty faced by the respondents was the toil in charging the phone followed by using someone's else's phone. Appointments: Lessons from an Ebola Clinical Study in Kambia District, Sierra Leone Others included; the message being sent at a wrong time when the respondent has switched off the phone, when respondents are busy working or when the respondent was resting

Type of Message Preferred (IVR or SMS)
Respondents 47.5% (n=45) were comfortable receiving IVR, 35.8% (n=29) preferred both text and voice messages and 17.3% (n=14) said that they preferred text messages alone.

Meeting Clinic Visit Appointments Comparison
From the survey results, 74.1% (n=60) of the respondents who were sent visit reminder messages reported that they never missed an appointment while 25.9% (n=21) have missed an appointment at some time. Only three respondents (7.7%, n=3) in the category of those who did not receive visit reminder messages admitted to missing an appointment at some time-the majority 92.3% (n=36) have never missed an appointment. Details are in the table below: When respondents were further asked what contributed to their never missing any visits, for those who received visit reminder messages, 66.6% attributed this to the visit reminder messages while, 97.2% of those that did not receive visit reminder messages, that did not miss an appointment attributed compliance to Community Health Workers reminders. The figure below has additional results. Respondents who missed an appointment, 85.7% (n=18) were able to make their appointments with the clinic while 14.3% (n=3) were not able to make an appointment. Of those who missed an appointment, the majority, 38.9% (n=7) made their appointments a day after missing their original date, 22.2% (n=4) and 22.8% (n=5) made their appointments two and three days after the original dates respectively. Two respondents made their appointments five and seven days after their original appointment dates. Respondents who missed the appointment date and failed to make another one attributed failure to technology barriers. Reasons for missing the appointments included travel out of town (47.6%), working (33.3%) and the others can be found in the figure below:

Future Preparedness to Outbreak Response Comparison
In the event of an outbreak, 48.1% of respondents who received clinical visit reminder messages preferred to be informed through IVRs or SMS, 27.2% preferred Person to person and 24.7% preferred Radio. On the other hand, 41.0% of respondents who did not receive Visit reminder messages preferred Radio, followed by IVR and Person to person both with 33.3%. Details are in the Figure 5 below: In the event of an outbreak, IVRs and SMS stood out as the most trusted source of information for respondents who received Visit reminder messages-55.6%, followed by radio-28.4% and then 18.5% person to person. For respondents who did not receive Visit reminder messages, 41% trust radio would provide information to them, 38.5% trust person to person and 33.3% trust IVR. Details are shown in Figure  below: For information valued most if sent through the IVR during an outbreak for those who received visit reminder messages, 81.5% respondents wanted information on How to protect myself or others, 72.8% on How to treat myself or others, 71.6% on the location of the outbreak, 60.5% Symptoms of the disease/illness and 48.1% on Location of treatment centers. For those who did not received visit reminder messages, 74.4% on How to protect myself or others, 59.0% on How to treat myself or others, 53.8% on Symptoms of the disease/illness, 46.2% on Location of treatment centers, 43.6% on the location of the outbreak. Details are in the table 5 below:

Discussion
With a wide urban-rural connectivity gap as mobile phone access and ownership continue to become more common worldwide [8] and the exponential growth of mobile communications even in the developing world, there is improved communication across vast geographical distances that were inaccessible [9]. The various publications in the literature on mobile technology in the recent years have shown evidence that SMS reminders significantly improve both health-seeking behaviors and outcomes in sub-Sahara Africa [10][11][12][13]. Lessons from this clinical study as shown in Table 4 indicate that the innovation played a huge role in vaccine awareness and adherence to visits schedule evidenced by 74.1% of the respondents that were sent visit reminder messages not missing an appointment. Even those that missed appointments at some point for some reasons endeavored to make it to the clinic a few days after the appointment date and no one missed because of sickness. This finding is in agreement with a study on evidence of the role of mobile messaging in maternal and child health that found a statistically significant increase of over 10% in the proportion of women receiving at least four antenatal care visits, 50% less likeliness to fail to attend postnatal visits and statistically significant increase (from 34.5% to 44.2%) in the proportion of children receiving on-time vaccination after implementation of a mobile application for automated text message appointment reminders [14].
Many of the respondents (81.5%) in the study valued most information on how to protect themselves from the outbreak followed by how to treat themselves and the location of the outbreak. This mean that mobile health messaging could play a vital role in Ebola prevention and control as majority trusted and preferred it as a mode of getting information in a likelihood of an infectious disease outbreak like Ebola. This coupled with other studies which showed that even in a country like Sierra Leone that is so challenged, somehow technology makes its way even to the most remote locations [15] thus likelihood for increased adherence to Ebola prevention and control measures received through mobile messaging. This therefore confirms that mobile phone access and ownership promotes clinic visits fulfilment providing opportunities to leverage on mobile-health (mHealth) technologies to target demand-side barriers, such as forgetting appointments, not knowing the vaccine schedule [8]. This provides an added advantage in mobilizing and or alerting individuals and communities for appointments, prevention and control information that is not doubted. Notably, the study found phone ownership being very key in fulfilling clinical appointments with majority of participants (86.4%) owning personal phones or controlling the phone used to receive the IVR and the SMS messages from the EVOVAC Salone which also correlates with findings of the study where phone ownership among women was associated with increased uptake of some RMNCH interventions [16].
Another lesson learnt was that Mobile messaging contributed positively to the vaccination drive for primeboost vaccine for Ebola as evidenced by the high percentage of patients (74.1%) that never missed an appointment to the clinic. 66.6% of the participants who received visit reminder messages, and never missed an appointment attributed this to the innovation for the success of the vaccine trial. This finding further affirms the review [17] assessing the effects of mobile phone messaging reminders for attendance at healthcare appointments.
The study presented an opportunity for recommendations for example: Mobile messaging as an innovation in mHealth has potential to improve vaccination coverage for infectious diseases such as Ebola. Thus, for future vaccination programmers, there is the need to ensure improved synchronization process so that messages match visiting dates in order to minimize repetitions for participants who have already fulfilled their visits. There is also a need to align very well the clinic records with those of MOTECH to eliminate chances of giving participants wrong information in errors. Working with mobile telephone service providers to improve network in rural areas not to compromise messaging and making the system to be two-way so that participants can be given an opportunity to respond to messages would make the innovation more appreciated. The Ministry of Health teams might need to borrow visit reminder messaging from this project for current vaccination drives that have more than one vaccine regimen.
Like any other study, this one presented these limitations: The study focused on clinical trial participants and this called for careful interaction in order not to violate any agreed personal privacy issues and also to avoid any stigma at the community level. This therefore made it challenging to get sufficient information from the participants. Furthermore, due to clinical study protocol guidance for participants' privacy, it was difficult to include as many participants as possible and therefore fewer participants included that could affect generalization of results due to a small sample size.

Conclusion
EBODAC implemented MOTECH efficiently, effectively and appropriately and being the first time the innovation is used in an Ebola Emergency/Outbreak, it provided an opportunity for continuous improvement and learning of how MOTECH works well in a health emergency.
The top three Promising Practices are: The use of the local languages in IVR messaging enabled participants to understand the purpose of the MOTECH messages and acted accordingly. Over 70% of the participants did not require assistance to listen to the messages.
Trial participants who received IVR messages actively listened to the messages. Over 90% of the recipients/participants listened as extracted from the MOTECH monitoring system.
The IVR messages contributed to the high rate of participant retention as they were fully aware of the importance of staying in the exercise through actively listening to the messages.
The top three lessons learnt are: If IVR messages are not available in the preferred and available local languages in the area of deployment, vaccine participants will not understand them, and therefore recipients will not be able to act on their content (e.g. attending an appointment or receive a vaccine) which will have a resultant effect on the entire trial. It is worth noting that compliance to clinic visits will be significantly affected if IVR messages are not available in the local languages of the participants.
Participant retention could be negatively affected if IVR messages are not used to communicate with participants in low literacy and fragile contexts.
If MOTECH relies on external data entry in order to send messages at the correct time, the external data entry partner might not be able to meet MOTECH's requirements and this could result in incorrect messages being sent and at the wrong time.

Ethical Statement
The authors are accountable for all aspects of the work in Appointments: Lessons from an Ebola Clinical Study in Kambia District, Sierra Leone ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Conflicts of Interest
The authors have no conflicts of interest to declare.