*Center for Communication Programs, Johns Hopkins Bloomberg School of Public Health
**Mae Tao Clinic

The Mae Tao Clinic provides services to illegal migrant workers in an area near the Thailand-Burma border (on the Thai side). In 2001 the clinic launched a project with two aims: first, to improve reproductive health service delivery, and second, to build the local capacity for monitoring and evaluation (M & E). This paper presents the methods and results of the evaluation component of this project, and discusses the lessons learned with regards to improving the clinic staff's M & E competency.

Background

Established in 1989, the Mae Tao Clinic currently serves approximately 100,000 migrant workers and 50,000 IDP who cross the border from Burma to access the health centre. The clinic has undergone exponential growth in the amount and range of services it provides over the course of its existence and the clinic decide to embark on a quality improvement project beginning in early 2001. The basis for this project was the Quality of Care Conceptual Framework, which has three components, (1) programme readiness, (2) quality of care, and (3) impact. The clinic team had made a number of changes to improve on the first and second components by attending training courses, partially refurbishing the physical premises, improving its stock of supplies and streamlining its management systems. In addition the team sought to enhance the M & E capacity of staff members, and an evaluation design was developed using a participatory approach that was non-threatening and encouraging of full staff participation. A core M & E team of senior providers from different clinic departments implemented the project. The author approaches the problem from the point of view that the delivery of high quality health services is a human right, and that there is a substantial unmet need for reproductive health services for refugee and IDP women around the world.

Methodology

The evaluation team used three different instruments in the collection of baseline and follow-up data:

• A facility audit, which was to assess the clinic's readiness to provide quality services, indicators included service availability, infrastructure, supplies, drugs and commodities.
• Observation of client-provider interaction, which used observation checklists, one for antenatal care (ANC) sessions and a second to assess family-planning (FP) sessions. Checklists covered interpersonal relations, routine procedures, counselling, and health promotion.
• Client exit interviews were also carried out for information on client satisfaction levels and to assess the knowledge, attitudes and behavioural practices with regards to reproductive health.

A model known as the Bruce/Jain framework provided the basis for the six elements of quality that the research team sought to evaluate, these include: interpersonal relations, information given to clients, technical competence, mechanisms to encourage continuity, choice of contraceptive methods, and an, "appropriate constellation or services." Each of these elements contained a set of indicators that were present in at least two of the three evaluation instruments. The results are presented below by element, as they are in the article.

Results

Interpersonal relations are aspects of, "client-provider interactions that involve communication, privacy, confidentiality and respect."

• Modifications were made to the facility (the installation of screens) to improve privacy.
• Client-provider observations reported an increase from 53.8% to 92.6% of clients seen in privacy for FP meetings, and 83.3% to 100% for ANC sessions. Exit interviews, however, recorded far less improvement in privacy, with a shift from 39% to 41% of clients reporting a private meeting.
• Observation checklists reported a decline of 13% (FP) and 25% (ANC) in providers assuring clients of session confidentiality. Conversely, exit respondents showed increased confidence in the belief that their confidentiality would be respected.
• Exit interviews also recorded a substantial decline over the period from 48.4% to 6.3% of clients who believed that providers treated them "very well".

Information given to clients refers to the, "satisfactory coverage of topics that will enable clients to promote and protect their own health."

• One of the project improvements was the revision of all the client record cards to serve as better ‘job-aids’ for providers.
• In FP sessions, observers recorded an increase in discussion of contraceptive effectiveness (from 66.7% to 100%) and side effects (61.5% to 92.3%), but client knowledge of side effects dropped from 38% to 33%.
• In ANC sessions, observers recorded a modest improvement (from 50% to 64%) in provider emphasis on at least two prenatal visits, yet client knowledge of its importance improved substantially (from 37% to 71%).

Technical competence is the, "extent to which providers comply with clinical guidelines and infection-control procedures."

• Observations showed an increase from baseline to follow-up in the proportion of providers who washed their hands prior to client contact, from 17% to 70% (FP) and 9% to 63% (ANC). Hand-washing after client contact also improved.

Mechanisms to improve the continuity of care, through managed referral are important in assisting clients to continue accessing health services.

• The clinic developed new client record and referral cards for hospitals and sought to improve scheduling of follow-up visits.
• Both observations checklists noted that providers consistently discussed return visits, and 96% (at baseline and follow-up) of patients knew when their next visit was.

Choice of methods refers to, "the variety of contraceptives offered to clients and the programme response to their varied needs."

• Improvements were made to the supply system at the clinic to ensure a constant stock of commodities, and there were no instances of lack of contraceptives throughout the study.
• The observed proportion of providers who discussed the range of available options remained high (91% to 88%), though the proportion of FP exit interviews that reported discussion of three or more methods only increased from 53 % to 59%.
• The proportion of providers who gave their clients their preferred method remained high, 92% at baseline and 96% at follow-up.

Appropriate constellation of services refers to the, "degree to which services offered are convenient, acceptable and responsive to clients health needs." This includes discussion of HIV/AIDS.

• Observations showed that the percentage of FP providers who discussed HIV/AIDS prevention rose from 61% to 85%, while clients reported in their exit interviews an increase in discussion of these issues from 43% to 58%.
• Observations and exit interviews also showed high-levels of offering of HIV/STI pre-test counselling.

Conclusions

The results demonstrate that improved programme readiness can result in the provision of good-quality services, even though some of the outcomes were mixed. The improvements in programme readiness, such as checklists on client records and installation of hand-washing stations resulted in substantial improvements. The interpersonal relations results are inconsistent and the authors suggest that they are "puzzling." While the new clinic layout seeks to provide greater client privacy, clients did not perceive a similar increase, and observers and clients had contradictory perceptions on improvements in the level of respect shown to clients. The authors present some discussion as to differences in the two instruments that might have generated this discrepancy.



The authors also reported that the secondary aim of this project, improving local monitoring and evaluation capacity was achieved. The team identified a number of benefits to the exercise including the learning of new methods for determining effectiveness, acquiring skills to improve their understanding of the dynamics of health promotion, a greater appreciation of health provision as a human right, and the team learned how to use data as a tool for advocacy and coordination.