January 2000

Project Cycle Management and the art of Log-frame Analysis

Strategy and Tactics is presently running workshops across South Africa on behalf of the Department of Labour to assist Labour Centres and provincial offices in preparing implementation plans for the Labour Market Skills Development Programme. These plans are developed using the Project Cycle Management (PCM) approach.

We are also helping to design a monitoring system for the LMSDP based on the log-frames.

PCM and log-frames may appear to be new buzzwords in strategic planning, but are neither new nor just another planning fad. Introduced by the European Union a decade ago, PCM was developed to ensure the quality of project design and to strengthen the quality of management on the project.

Anyone who has been involved in PCM will know that its value relies on its simplicity. As the name suggests, PCM is a cycle which follows a logical path. Each step relies on the success of the previous step.

Most projects are conceived as part of a programme, often at a national level. As part of the programming phase, national problems are identified and the objectives of the programme specified. Next comes the identification phase, where projects (components of the programme) and other forms of intervention are identified. This is followed by the formulation phase, in which ideas for projects are turned into operational plans, and consultation takes place.

Once the proposals have been drawn up, financing is sought and/or apportioned to the projects. At this point the project can be implemented. Embedded in any coherent project design is evaluation, which can begin as soon as the cycle begins.

Each stage of PCM is a mini-cycle that should be monitored and evaluated. After implementation begins, evaluation becomes critical.

The results of the evaluation will feed back into the cycle at the identification stage. This allows the management of the programme and the project to assess whether the particular project is achieving its objectives. If not, it may be decided that another project be introduced, or the existing project be modified. S&T's preferred design sees evaluation as an ongoing tool, utilised to judge and improve performance in all phases. It also ensures that all players are open to evaluation, not just implementing agencies.

At the centre of the PCM cycle - in the version S&T utilises - is ongoing monitoring, evaluation and reporting (M,E & R). All steps along the PCM path are cycles in themselves, and each is open to measurement, assessment and dissemination of the results (the M,E & R sequence). The danger of the standard PCM design is that evaluation is left as a summative-only activity, and reporting is frequently forgotten.

Log-frame

The core tool for PCM is the logical framework (often simply referred to as log-frame), the construction of which is typically done in two phases. In the first phase, the analysis phase, key problems and opportunities are identified, from which objectives that will address these problems are clarified. The underlying principle is that projects are designed to address the problems faced by beneficiaries. In the second phase, the construction of the log-frame is completed. Once the log-frame has been designed, activities can be scheduled and resources can be allocated.

The log-frame matrix provides no magical solution, as the official EU PCM handbook warns: 'if you put garbage in you will get garbage out'. However, advocates of PCM argue that the log-frame is extremely useful in that it assists with making logical connections between different levels of the project. Moreover, it provides a useful aid to thinking strategically about the project.The key is to remember that all plans, proposals, and log-frames should be seen as dynamic tools that should be revisited and revised regularly.

The actual log-frame is a 4x4 matrix. Running down the matrix, in order, is the overall objective of the programme, the purpose of the project, the results of the project and the activities that the project will deliver. The other three columns provide the basis for measuring the project's performance.

OVIs are indicators that provide a means for measuring the objectives of the project and also lay the foundations for monitoring. The SOVs are the sources of information and the methodology to be used to measure the indicators. Assumptions are those factors external to the project which could affect the implementation and sustainability of the project. Be warned about the "killer assumption" as it kills the project dead.

The completed log-frame thus provides a concise summary of the objectives of the project, the indicators by which the project will be assessed and the key risks and assumptions which may affect the achievement of the objectives of the project. From this log-frame a detailed workplan can be drawn up.

S&T: a year on

In numerous evaluation projects we have been actively involved in shifting the mindsets of clients - including government and NGOs - to avoid situations where we are called in only at the end of a programme to provide final judgement on its worth and impact, too late to make key decisions based on sound research findings. We believe that evaluation, in all its forms, is a fundamental component of any social programme, interacting with regular monitoring to provide important information concerning the direction and impact of such pro- grammes.

S&T keeps in touch with theoretical developments in the academic arena through conferences and workshops where we have presented papers for critique and review, and submitting articles to various refereed journals. We will also be hosting an international summer school on youth, aimed at bringing together academics, policy-makers and youth workers, to bridge the gap between theory and practice.

The focus of our work remains the ability to translate theory into practice, and all our work - from involvement in government task teams to evaluation of poverty relief initiatives - aims to provide the tools and techniques of research and facilitation for decision-makers in order to better inform their decisions.

Committed to the transformation of South Africa into a powerful democracy and the development of Africa as a continent, we look forward to the new millennium as we continue to contribute in our small way to this transformation and development.

In Cape Town

The Cape Town office has played a significant role in several major national education and health initiatives, including participating in the research team that is measuring transformation in institutions of higher education in South Africa, and the National Business Initiative's study of further education and training in the Western Cape.

S&T worked on the Labour Market Skills Development Programme, facilitating Project Cycle Management and Log Frame analysis workshops with labour centres throughout the country. These will continue running for the early part of the year, and this workshop has also been run for a sub-division of the National Productivity Institute.

The Cape Town office had a busy year producing research reports and Sihaam also produced a baby girl.

The hidden side of research: high quality fieldwork

Having been involved in fieldwork training for research projects for the past three years, one begins to understand that each project has its own dynamics and challenges.

In planning a project, concepts and methodology are what specialists normally worry about. A proposal puts forward the methodology for the study, and once the client has accepted the project a brainstorm is arranged to design the questionnaire or instruments to use in field.

Designing a questionnaire could take weeks, especially if there are many interested parties involved. S&T normally goes through five to six drafts before anything is shown to the client. We then pilot the questionnaire to evaluate what works and what doesn't, once a final instrument is agreed, the research team is ready to go out to field.

Unemployed local matriculants are used to administer questionnaires, having been trained and sent out as teams to the various areas. Because of the unemployment 'epidemic', more fieldworkers have tertiary qualifications, but even so, a synergy of theory and practice must be achieved.

When planning for fieldwork teams, one has a set number of days within which to complete the sample. A fieldwork manager will always add days for 'unforeseen circumstances' which could affect the smooth running of data collection and which will vary from one project to the other. It could be violence, natural causes such as rain, areas which appear on the maps but are not necessarily known on the ground and (visa versa), and so on. In some instances, access may prove to be a problem where communities feel threatened by strangers asking for household information.

Using local people known in the area may avoid some problems, and should also lead to improved quality of data since there are no barriers of language or dialect or the sense of an 'outsider' asking questions. Locals know the area and are familiar with the culture.

Having chosen fieldworkers and supervisors (team-leaders) one is now set to train teams on sampling procedures (i.e. the selection of households and respondents) and administering the questionnaire.

During training it is imperative that the researcher who designed the questionnaire is part of the training team. Some clients also want to be present during training in order to introduce themselves and the project. We encourage this, and fieldworkers appreciate the respect it shows for their work.

In every training session, fieldworkers raise issues which may not be obvious to researchers. These may relate to the phrasing of questions, or perhaps to double-barrelled questions which may prove problematic during coding. Language is an issue, it is very rare that questionnaires are translated into vernacular languages because this is time-consuming and expensive. It is therefore important to get consensus during training on how certain questions will be asked. This is particularly true in metropolitan townships where people grow up speaking up to four and five languages.

The first week of fieldwork is crucial. It is important for the training team to be available for the issues and logistical problems that may come up, because of the unpredictable nature of issues on the ground.

If fieldworkers are experiencing access problems, the researcher may need to negotiate with local structures, amakhosi (traditional leaders) or others. During data collection a co-ordinator may be stationed in an area to act as a trouble-shooter. The co-ordinator is also responsible for check-backs (S&T normally checks 25% - 30% of all interviews) and meeting with teams during the survey.

Questionnaires don't always come back from field correctly completed, questionnaires must be checked before coding takes place. Again, coding poses a challenge because it is not only transferring numbers - coders must understand the questionnaire and pick up minor mistakes. Once coding is completed, questionnaires are sent for punching and then researchers can plan for write-up.

It is only at this point that top-line results can be shared with clients. Their first response is normally, "What took you so long?" If only they knew!

Development programmes and the challenges of partnership

Entering into partnerships

In most tendering processes, companies are encouraged to enter into partnerships. The argument is that the client can access a broader range of expertise and services.

However, people from different sectors have different perspectives. Private sector companies judge performance by time and money spent in delivery. People from the development sector use these measures but alongside, local participation, sustainability and so on. These are not merely questions of style or surface: they are fundamental.

Some companies that enter into partnership are either an emerging empowerment or new company, or well-established mostly white-owned firms. In the development arena one encounters a scenario where engineering firms want to collaborate with an emerging black company to maximise PDI scores. (PDI scores are the points given to 'previously disadvantaged individual' status by government departments and tender boards.)

Unfortunately, however, in many cases companies such as these have difficulty in accepting a slower, uneven and more developmental approach. These issues may be discussed while tender documents are being prepared, and all sides may claim to be fully committed to empowerment. Once implementation begins, however, many partnerships begin to struggle.

Faced with what amount to ideological differences - not merely questions of style, as often suggested - partners suddenly find themselves in a legalistic relationship, governed by Terms of Reference which become guiding principles. Budgetary concerns, if not absolutely clear beforehand, can become the central issue.

One side of the partnership expects the key concern to be delivery within deadlines and budget, regardless of the extent to which communities are on board. They voice concerns about foot-dragging and an over-concentration on consultation. The other side points out that consultation is critical to sustainability, and that delivering within time and budget is irrelevant if the assets are left to disintegrate because the community does not feel part of the project. Both are right, and the challenge is to find a balance of both views within a single partnership.

Evaluating the Gauteng Youth Development Initiatives

Each business is operated by a collective of young people aged 18 to 35. The group's number varies from site to site with a minimum of ten participants and a maximum of fifteen, who all receive business and life-skills training from CBE&T, FEBDEV and JEP.

The business training should enable them to design Business Plans and then run their own businesses. Life skills training offered by JEP provides assistance with interpersonal problems and working together as a collective.

The YDIs aim to generate income and create job opportunities for other youth, or to help support local community initiatives. There is a real challenge in linking entrepreneurship with collectivism.

Our methodology will combine qualitative and quantitative measures. A focus group with participants from all six sites will be used as a basis for designing a questionnaire, to be completed by participants from all the sites. This will be complemented by in-depth interviews with training providers and others. Analysis and reporting take place in early 2000.

Quantitative training for ANC officials

The training workshop looked at technical issues - sampling, weighting, questionnaire design, question 'stacking' and so on. The samples used for much of the public polling during the 1999 election were analysed, and severe skews and biases became evident to all participants - and helped explain why the public pollsters got the results so badly wrong.

On the second day, participants (who came from all nine provinces as well as some from HQ) used existing data to assess their strengths and weaknesses as the local elections approach.

Community profiles in the three poorest provinces

The Department of Public Works, Monitoring and Evaluation Section, commissioned S&T to conduct community profile surveys in three provinces: Eastern Cape, KwaZulu-Natal and the Northern Province. These will assist the Department to plan better for future projects and measure the success of past projects.

We used project lists provided by the Department, with project number, District Council, project category, name of the project, project type and budget size. We then stratified our sampling using the project number, District Council, project category and budget size.

To establish starting points we used maps provided by Davies, Lynn & Partners, with project numbers and areas, selected the identified projects and drew a 10 km radius to identify the sample area on the maps. In each of 25 starting points, fieldworkers had to visit four households.

Our training, done by S&T and the Department of Public Works, focused on sampling procedures and administration of the questionnaire. Our fieldwork team in KwaZulu-Natal comprised social facilitators used by the Department to work with communities, chosen because of their long-standing relationships with communities and with the amakhosi. A co-ordinator was based in the province for the duration of the survey.

However, using social facilitators did not prevent the problems that normally accompany field research. Access was a problem in some areas where the amakhosi felt that there was insufficient consultation and they would not allow strangers to ask 'their people' questions. Extensive negotiation was necessary, with District Councils in some instances intervening to meet with all stakeholders. Team leaders were called in to explain at meetings what the study was about and who would benefit. This meant that fieldwork stopped for days whilst access was negotiated.

Fieldwork has been completed in KwaZulu-Natal, work in the Eastern Cape is still in progress and Northern Province will be surveyed in 2000. The results will provide a fascinating look at the benefits of small- and large-scale anti-poverty programmes.

Gauteng CBPWP Directorate workshop

Under discussion were programme issues within the Directorate that relate to implementation, the need to devise a targeting strategy for the province, project identification and selection criteria, and the extent to which the strategies used by the province are in line with the national strategy of the CBPWP. The following immediate key areas were identified:

• The province should undertake its own targeting strategy and spend its budget where it is most needed, focusing on poverty pockets and the best strategies to tackle these.
• It is important to realign the programme within the province, and it was agreed that the province should undertake a brief review in the form of a desktop study.
• A financial audit of projects implemented in the last 3 years must be linked to the design of a new implementation strategy for the directorate.
• Finally, where the need for capacity development is greatest, this must be implemented as a matter of urgency.

National Enterprise Survey

S&T has been commissioned to undertake the survey among 1000 smaller firms (those with 50 or fewer employees). Face-to-face interviews are used with these smaller firms, and the results will be synthesised with those from the larger firm survey to provide a comprehensive picture of the current state of affairs.

The M&E system for the LMSDP

David has been involved in the second phase of our work in the Department of Labour, specifically the Labour Market Skills Development Programme (LMSDP). Parts of the inception workshop were facilitated by David and Matthew who then assisted two of the six EU-funded projects through Programme Cycle Management and log-frame exercises; the next step was to design a monitoring and evaluation (M&E) system for the LMSDP as a whole.

David, working with British consultant Mike Felton of ITAD, completed the log-frames for the EU-funded projects (which comprise about half of the LMSDP). This was a necessary precursor to designing a monitoring system that will effectively utilise the OVIs and MOVs in a systematic and useful monitoring system. One area of difficulty is the great detail required in each log-frame, resulting in a great number of OVIs (also called Key Performance Indicators or KPIs) and a potentially massive quarterly monitoring report. The initial design was completed in December and work on refining the project continues.