Overview

Data collection processes and instruments

Glimpses of the qualitative data

Approaching the quantitative analysis

Overview of Research Design

Longitudinal Panel Survey

Distinctive features:

• 4 rounds of data collection
• same respondents throughout
• confined to two areas

Timing of Survey Rounds

Sampling of panel respondents

Criteria:

• Representative of the two sentinel sites
• Adult respondents only (16-65 yrs)
• N=400 + 100 for control group per round

FINAL SAMPLE

Survey round	Panel group	Control group
1:Baseline	500	0
2: 1st Intermediate	500	100
3: 2nd Intermediate	500	0
4: Evaluation	500	100

The Sentinel Sites

Selection criteria:

• Urban / Rural (different provinces)
• NNVAW activity

MAMELODI

large, sprawling township in Gauteng

high density, mixed population

formal and informal areas

inadequate health & welfare facilities

EMAFAKATINI AREA

four small rural villages in KZN

very homogenous population

traditional authority

little infrastructure or facilities

MEASURES OF CHANGE

Knowledge and awareness

• content themes (eg awareness of HBP)
• specific health knowledge (eg. treatment / prevention)
• perception of risk (eg. HIV)
• rights and legal position (eg. DVA / Protection Orders)
• resources and recourse (eg. clinics / police / Helpline)

Attitudes

• acceptance (eg. no excuse or justification for DV / sexual harrassment)
• privacy (eg. DV is not a private matter)
• discrimination (eg. reduction of stigma around people with HIV)

Perceptions of social norms

• same range of items as for “attitudes”
• measure personal divergence from perceived social norm

Locus of Control

• efficacy / - helplessness (eg. Prevention of HIV / action against perpetrator)

Intermediate practice

(increasing preparedness to take action)

- thought about / talked about / considered taking action

• intervening in DV / sexual harassment
• having an HIV / BP test, or asking someone else to
• practising safe sex / a healthier lifestyle
• giving / seeking help

Practice

• specific health practice (eg. BP / HIV testing; condom usage)
• use of resources (eg. telephone Helpline)
• support-giving behaviour (eg. to abused women / HIV+ people)
• participation in community action (general & site-specific)

Interpersonal and environmental support

• support-giving and –seeking behaviour (eg. Where to go for advice / helping someone with HIV)

* categories not mutually exclusive - some overlap (eg. Pot banging)

* Questionnaires developed over intermediate rounds

Capturing Soul City Exposure

Electronic Media

• Reported frequency of access at evaluation round
• Access during intermediate rounds
• Specific TV access by episode
• Theme-specific reception (open-ended - post coded for message + plot)
• Sources of info on each theme (post-coded for Soul City)

Print Media

• Frequency of seeing booklets, newspaper inserts, lifeskills packs - all rounds
• Specific booklets (thematic, including relevant booklets from previous series)
• Sources of info on each theme (post-coded for Soul City)
• Recognition of specific newspaper coverage (Soul City & thematic) - intermediate

NNVAW

• awareness of Network & local organisations (all rounds)
• knowledge of / participation in activities (all rounds)
• quantitative hooks into community mobilisation: knowledge of / participation in specific Network activities in the respective sites (intermediate rounds)

QUALITY OF THE DATA

Good sample: Demographic information concurs with that of the 1996 census

Correct panel: Thorough supervision and in-field check-backs

For the most part, responses are consistent and shifts are logical

Interviewing: Experienced interviewers, additional training, knowledge of SC

QUALITY OF DATA MAY BE AFFECTED BY:

• Inability to match gender of interviewers and respondents
• Spontaneous translation of English questionnaires in the field
• Privacy of the interview was not always possible (but controlled for)
• Respondent exhaustion in long interviews
• Forewarning - initial contact with KZN site
• Design effect - sensitization of respondents to the research (some control resp. too)
• Rapid development of baseline questionnaire

ASSESSMENT: VALID & RELIABLE

SOME LIMITATIONS CAN BE CONTROLLED FOR IN ANALYSIS

ANOMALIES IN THE DATA SET NEED TO BE EXAMINED

COMMUNITY MOBILISATION

SoulCity:

• Move beyond targeting individuals
• Create environments supportive of change
• Mobilise community action
• Reorient Services

Research:

• Key service providers
• Opinion leaders
• Local structures

Community Mobilisation Interviews

Standard interviews:

• clinic nurses
• religious leaders
• teachers
• councelors
• police (officers & management)
• community police forums
• community based organisations

Respondent selection informed by rigorous fact-finding

• investigative research prior to interviews
• ‘surprise factor' built into interviews
• participants not selected on the basis that they know SC

Area-specific interviews: Mamelodi:

• misc. service providers
• orgs - women & child issues
• “Soul City” - residents & leaders eMafakhatini:
• traditional leadership
• orgs - women & child issues

Community Mobilisation Interviews

Semi-structured interviews

• Opinions on thematic issues - what is the message they put out there
• Implementation of DVA(police) - capacity / training
• Awareness of Soul City, and response to message portrayal
• Impact of Soul City / Partnership
  
  • individuals (professional capacity)
  • organisation / institution / structure
• Reported impact of Soul City on constituencies & broader community (perceived) - specific activities / anecdotes

COLLECTION OF EXISTING DATA

Reveal trends in support-seeking behaviour over evaluation period

• Health service providers: blood pressure tests, HIV tests
• Police Services: local reporting of domestic violence
• SMME service providers: loan applications, advice services
• Other services: counselling for abused women

Pragmatics:

• Bureaucratic barriers slowed the process further
• Poor or non-existent record-keeping systems (verbal information)
• Inadequate services can't reflect increases (esp. clinics)

Data collection still unfinished

QUALITATIVE INTERVIEWS

• sub-section of Qualitative Study
• panel respondents in Sentinel Sites

BLITZ ANALYSIS

MEDIA MONITORING

Impact of Soul City on Media

Impact of Media on communities

Impact of community mobilisation on Media

Multiple data sources:

Quantitative surveys - frequency of exposure to coverage (thematic)

- recognition of clippings (intermediate rounds) Community Mobilisation - exposure to media coverage (thematic)

- recognition of clippings

Monitoring of local media

- SC specific & thematic coverage

Interviews with local media (not yet done)

National Network on Violence Against Women

Partnership established to:

• reinforce intervention on the ground
• promote / facilitate community mobilisation
• assist with reorienting services

Multiple data sources:

Quantitative data

- awareness of Network / local structures

- awareness / participation in activities

Community mobilisation

- awareness of Network / local structures

- impact of Soul City partnership

Monitoring of Network activities

Interviews with local organisations (not yet done)

QUALITY OF THE DATA

Varied interviewer skill and experience

Consistent use of facilitators - familiar with Soul City and research

Gradual improvement of facilitation over time

Careful selection of respondents

Some technical problems:

• occasional poor quality of recording - gaps in transcripts
• late starts - impatient respondents and early departures
• one group reconvened (same participants)
• language difficulties in deep rural area - two male groups used female translator

ASSESSMENT: GOOD QUALITY - MEASURED AGAINST QUALITY STANDARDS INFORMED BY INTERPRETIVE-HERMENEUTIC PARADIGM, TAKE COGNISANCE OF PROCESS & INTERVIEWING QUALITY IN ANALYSIS

APPROACHING THE ANALYSIS

PRE-ANALYSIS CHALLENGES AND DECISIONS

1. Some questions were not consistent over all rounds

Analysis limited to identical items

2. Some response categories were differently formulated

Continuous data collapsed for comparability with baseline data

3. Number of missing cases increased as respondents were lost from panel

“Drop-outs” excluded from baseline for purposes of comparison

4. Massive data set - analysis and correlation of individual items unmanageable

Achieve data reduction by grouping variables into scales

APPROACHING THE ANALYSIS

PROCESS

Analysis Options

Frequency/Percentage Shifts

Correlations/CFA

Scale/Variable Development

T-Test

Anova

Regression

Sample Sub-Groups

Location: Gauteng Vs. KZ-Natal

Gender: Male Vs. Female

Demographics: Age, Race, SES

Frequency/Percentage Shifts

Valuable Descriptive Tool

Assess Significant Shifts (Chi Squares)

Correct For Significance Test

Correlations/CFA

Item-By-Item Correlation Matrix For Constructs

Calculate CFA For Constructs

Scale/Variable Creation

Develop New Scale for Constructs to Compare Across Rounds

Change/Difference Variable Creation for Construct

• Evaluation Score - Baseline = Difference
• Round 3 Score - Baseline = Difference
• Round 2 Score - Baseline = Difference

T-Test/ANOVA

Calculate Means and Standard Deviations for Items/Constructs

Compare Means Across Rounds

T-Test To Compare 2 Rounds

One-Way ANOVA TO Compare 3+ Rounds

Two-Way ANOVA

T-Test (K/A Practice)

T-Test (Baseline and Evaluation)

6pt scale (6=best, 12=worst)

KZN Males: Baseline (10.6) vs. Eval (8.71)

T-Test = 11.83, p<.05

(Males in KZN report an increase in knowledge/awareness related to domestic violence issues.)

ANOVA(Intermediate Practice)

2-Way ANOVA (IV: time, sex; DV: IP)

Differences Between B and Round 3, B and Eval

Time: significant main effect

F = 56.3, p<.005

Sex: borderline significance main effect

F = 3.67, p<.055

Interaction : no significant effect

F = .323, p<.808

Regression

Multiple Linear Regression

• Evaluation Round Data
• Baseline, Round 1/2, Evaluation Data
• Measure of Association: B weights/R^

Logistic Regression

• Evaluation Round Data
• Measure of Association: Odds Ration/R^

Regression (IP)

Multiple Linear Regression (Media Exposure and IP)

Media: TV and Print

B weights

• TV: .19
• Print: -.39

Measure of Association

• R^: .165

QUESTIONS TO THE ADVISORY PANEL

Since panel designs are relatively new in the field, what are the best mathematical techniques to assess/attribute change?

Are any of the recommended techniques not useful for panel designs?

How best to integrate the sentinel site components - hooks and methods?

How to merge sentinel site data with data from the broader evaluation?

How do we define “mobilisation”?

Qualitative Impact Assessment

(As “Stand alone”, National Study, and as an integrated sub-component within Sentinel Sites Studies)

Objectives of the study

to understand “change” in relation to Soul City

to understand the mechanisms of change

...

Methodologically embedded within an Interpretive- Hermeneutic paradigm

“What does Soul City mean to you in your life?”

Sampling - linking the qualitative interviews to the longitudinal panel survey - Sentinel Sites

Rapid analysis

• Baseline and Intermediate(s)
• Identify shifts
• Establish exposure to thematic coverage in Soul City TV
• These people became our respondents for qualitative interviews

First Round Interviews

	Thematic focus	Gender	Type
GAUTENG
1	VAW	Male	Individual
2	VAW	Young women	Group
3	VAW	Young men	Group
4	VAW	Older women	Group
5	HBP	Mixed	Group
6	VAW	Women	Group
KZ NATAL
1	VAW	Male	Individual
2	VAW	Female	Individual
3	HBP	Older women	Group
4	VAW	Younger men	Group
5	VAW	Female	Group
6	VAW	Female	Group

Second Round Interviews

	Thematic focus	Gender	Type
GAUTENG
1	SMME	Mixed	Groupl
2	HIV/AIDS	Men	Group
3	Youth Sexuality	Young women	Group
4	HIV/AIDS	Women	Group
5	HIV + Y.Sex + SMME	Male	Individual
6	HIV/AIDS	Male	Individual
KZ NATAL
1	SMME	Mixed	Group
2	HIV/AIDS	Female	Group
3	Youth Sexuality	Younger women	Group

The following additional group was conducted a few months later:

	Thematic focus	Gender	Type
4	VAW	Male	Group

Reflection after round 1 and 2

• Fieldwork assessment after the 1st and 2nd round of qualitative data collection
• Internal validity
• Strategic Intervention - design

Components of the study outside of the SS

• Theoretical Sampling
• Soul City needs
• Pragmatic considerations

Interviews outside of the SS

Area	Profile
Eastern Cape Lusikisiki (Rural)	Women
Eastern Cape* Idutywa (Rural)	Mixed Gender
Western Cape Cape Flats (urban)	Coloured Female Youth
Northern Province Solomon's Dale (peri-urban)	Male Youth
Northern Province Solomon's Dale (peri-urban)	Men
North West Lichtenberg (rural)	Men
North West* Mareetsane (rural)	Mixed Gender
KwaZulu Natal Umlazi (urban)	Women
KwaZulu Natal* Indwedwe (rural)	Mixed Gender

*Radio Only

Analysis

• Thematic Analysis
• Atlas / ti

Soul City 4 Evaluation [6]

Soul City 4 [7]