Approach

In the consulting practice, implementing evaluation and research, IMACON experts use traditional and generally accepted methodologies and the principles described below. For each of the methodologies there are experts in our team with experience of 5, and in some cases – 10 years. It is important to note that experts used these methodologies while working with organizations in Western Europe and in the East European region, as well as in countries of the CIS. Following this experience some of them published articles and tutorials. IMACON experts are also the authors of translations and adaptations of several methodologies into Russian language.

DAC Principles for Monitoring & Evaluation

These principles are the minimum standard for the evaluation of development projects and imply focusing on assessing:

  • the concept and design of the project,
  • its implementation in terms of quality and timeliness of inputs, financial planning, and monitoring and evaluation,
  • the efficiency, effectiveness and relevance of the activities that were carried out,
  • whether the desired (and other undesirable but not intended) outcomes and objectives were achieved,
  • the likelihood of sustainability of the results of the project, and
  • the involvement of stakeholders in the project’s processes and activities.
  • the sustainability from various perspectives: financial, social and institutional.

The Logical Framework Approach (LFA) is a management tool mainly used in the design, monitoring and evaluation of international development projects. It is also widely known as Goal Oriented Project Planning (GOPP) or Objectives Oriented Project Planning (OOPP).  The LFA as a design methodology is a rigorous process, which if used as intended by the creators will impose a logical discipline on the project design team. If the process is used with integrity the result will be a high quality project design.

The Logical Framework takes the form of a four x four project table. The four rows are used to describe four different types of events that take place as a project is implemented: the project Activities, Outputs, Purpose and Goal. The four columns provide different types of information about the events in each row. The first column is used to provide a Narrative description of the event. The second column lists one or more Objectively Verifiable Indicators (OVIs) of these events taking place. The third column describes the Means of Verification (MoV) where information will be available on the OVIs, and the fourth column lists the Assumptions. Assumptions are external factors that it is believed could influence (positively or negatively) the events described in the narrative column. The list of assumptions should include those factors that potentially impact on the success of the project, but which cannot be directly controlled by the project or program managers. In some cases these may include what could be killer assumptions, which if proved wrong will have major negative consequences for the project. A good project design should be able to substantiate its assumptions, especially those with a high potential to have a negative impact.

The core of the Logical Framework is the “temporal logic model” that runs through the matrix. This takes the form of a series of connected propositions:

  • If these Activities are implemented, and these Assumptions hold, then these Outputs will be delivered
  • If these Outputs are delivered, and these Assumptions hold, then this Purpose will be achieved.
  • If this Purpose is achieved, and these Assumptions hold, then this Goal will be achieved.

These are viewed as a hierarchy of hypotheses, with the project/program manager sharing responsibility with higher management for the validity of hypotheses beyond the output level.

Project Cycle Management (PCM)

Project Management (PM) methodologies in use around the World are definitions of project management processes aiming at standardizing and improving the quality of the project management lifecycle. Quality of the projects can be defined in terms of the relevance, feasibility and effectiveness of the impacts of the investment, including how well they are managed.

A project management methodology consists of process groups and control systems. The PM methodology aims at organising the project cycle structure and defining not only the content of each phase but also how it can be best accomplished.

Typically all project management methodologies imply a flow similar to the “Plan-Do-Check-Act” cycle with phases which are linked together by results – the result of one phase should become the input of another. Although, the way of defining the phases of a project can be subjective and often based on organizational procedures, each project management methodology approach should include well defined phases and the transition from one work phase to another should naturally involve the transfer of some sort of deliverable (a document, piece of software, invoice, report, an approval by committee, etc.) The term ‘project’ is primarily used for convenience and simply means the collection of related activities for which a contribution is provided to meet a specified objective.

Different project management methodologies have been adopted by several agencies such as the Asian Development Bank (ADB), the World Bank (WB), the EC, and national governments. In particular, the European Commission (EC) has produced a “Project Cycle Management” (PCM) manual, which has been subsequently adopted by other development partners as one of the systems for project development, funding and evaluation (EC, 2004).

As mentioned, the way of defining the phases of a project can be subjective and often based on organizational procedures. In the case of the EC PCM, the project cycle presents five phases: Programming, Identification, Formulation, Implementation and Evaluation.

Institutional Development \ Organisational Strengthening (ID\OS)

ID\OS and IOM are instruments developed by our partner – MDF.

The ID\OS approach is based on the idea of ​​broad-based support necessary for successful interventions to development. This kind of leadership development interventions that will be supported by all stakeholders. ID\OS are effective when management and evaluation and those who are responsible for implementation and the planned beneficiaries are involved in their planning.

Development projects can be called sustainable if they are able to maintain an appropriate level of service for a long time after the cessation of major financial, managerial and technical assistance from external donors. And ID\OS tools help evaluate and formulate sustainable development activities in terms of institutional capacities of organizations.

ID\OS  is a process that can not be successfully completed using single short-term projects or programs. Embedding organizations and their objectives in the activity area, structuring the network of links between organizations – is by definition a process that requires considerable patience and time. Moreover, every process is inherently dynamic. The situation and conditions are constantly changing. Therefore, in addition to the long-term obligations, the ID\OS process also requires a flexible work environment, in which the parties will be able from time to time reconsider their relationship.

IOM

Integrated organizational model (IOM) – a model that is used to describe, analyze and diagnose the condition and functioning of the organization. Like any other model, IOM is a simplified version of a complex system comprising a plurality of mutually influencing factors. The model should help answer important questions concerning the operation of the organization. In addition, the effect of the application of the model is highly dependent on the specific situation of the questions, as well as skills and knowledge of those who apply.

IOM – is a universal model that pays special attention to the relationship between the various elements of the organization. Despite the fact that these elements are to a certain extent to be considered individually, are interconnected, and, ideally, should be balanced. If, however, between the different elements of the model of the organization or its structural unit there is an imbalance, or whether this balance is not expressed clearly, the organization will not be able to work as efficiently as possible, and the need for organizational change will be evident.

IOM gives the working tool that will place the various elements of the organization in the most appropriate manner, regardless of whether the object of the analysis is a department of the government, NGOs, local authorities, public organization or entity in any country of the world. Applying for the analysis of the organization of the model, you can not miss out on any essential element. However, IOM is a comprehensive model and should be considered not as a tool, but as an organizational concept. For in-depth analysis of the organization you may need more specific tools, the choice of which will depend on the specific purpose of analysis.

Outcome Mapping (OM)

Outcome Mapping (developed by International Development Research Centre in Canada) focuses on “outcome” in the sense of particularly changes in behavior of people, groups and organizations with whom a program or project is directly working. Through “outcome monitoring” an organization can claim contributions to achievements of outcomes rather than claiming the achievement of outputs. This method:

  • Defines the program’s/project’s outcomes as changes in the behavior of direct participants;
  • Changes in behavior, relationships and activities that are possible to observe;
  • Focuses on expected significant changes;
  • Focuses on how programs/projects facilitate change rather than how they control or cause change;
  • Recognizes the complexity of development processes together with the contexts in which they occur;
  • Looks at the logical links between interventions and outcomes, rather than trying to attribute results to any particular intervention;
  • Requires the involvement of program/project staff and partners throughout all PME stages.

 Most significant change (MSC)

MSC (RJ Davies and J Dart, 2004) is participatory method of monitoring and evaluation. Project stakeholders are involved in deciding the sorts of changes to be recorded and in analyzing the data. Essentially the process involves the collection of significant change stories from the `field level’ and the systematic collection of the most important of these by panels of designated stakeholders or staff. MSC advantages:

  • Participants have a choice about what sort of information to collect;
  • Uses diverse rather than standard data;
  • Information is analyzed by all participants, not simply by a central unit;
  • Subjectivity is used rather than avoided;
  • Stories are a rich way of expressing the changes which are happening in people’s lives;
  • Gets away from pre-set indicators;
  • Non-linear, open to the changes/outcomes which are seen by participants themselves in their lives;
  • A way of ordering a lot of information, to give an overview of some of the most significant changes which are happening as a result of interventions.

Organisational Analysis (OA)

This methodology of organisational analysis (OA) within a not-for-profit organisation has been compiled by Swedish organisation Forum Syd and is aimed at those who in different ways work for not-for-profit organisations.  A large part of the contents originates from Sida material assembled by the consultant Peter Winai in collaboration with representatives of internationally active not-for-profit organisations.  The material has been revised and supplemented by Greger Hjelm, management consultant at Rörelse & Utveckling, Stockholm.

Various perspectives regarding organisation development and the process of change are brought to the fore, not least the various roles that an external consultant may adopt in assisting with organisational analysis and organisation development.

OA methodology includes a number of useful working models that can be used when giving shape to an organisational analysis, to indicate what is most important when the practical stage of observing level of development and capacity in an organisation is reached.  Various tools are introduced for use in describing and assessing the organisation, for example: Staircase model, INTRAC model etc.

The main practical tool of OA is a series of practical questions for use when implementing an organisational analysis.  The analysis or assessment is then ready for use once the organisation embarks on development.

The chief aim of the methodology is the introduction of the way for working with analysis and development of organisations – it does not, however, lay claim to providing successful formulas or solutions.

Octagon

In 1999, Sida invited a number of Swedish non-governmental organisations (NGOs) to a discussion on possible ways of measuring results of the support provided for capacity and organisation development at local partner organisations. In May 2000, a model for the evaluation and follow-up of organisations – Octagon – was presented to this group by a consultant, Peter Winai, on behalf of Sida.

The Octagon is a tool for the assessment of strengths and weaknesses in NGOs and can be used by both the Swedish organisations and their partners in cooperation. The Octagon can function as an instrument to structure the dialogue with a partner organization when the aim is to obtain an overall picture of the organisation and to get to know it well. It can also serve as an aid for the selection of partners; for grouping partner organisations in relation to their needs of internal organisation development; or for identifying the point in time when the organisation, as the financier, should phase out its support for organisation development.

The Octagon is based on the idea that it is possible to obtain a comprehensive picture of an organisation’s capacity and development profile by making systematic reviews and assessments of four basic aspects:

  • The organisation’s objectives and management/administrative structures, the so-called organisational base.
  • The organisation’s activities with or for selected target groups, i.e. output.
  • The organisation’s capacity to succeed in its work. This refers both to its professional skills and the funds at its disposal, as well as its administrative systems.
  • The organisation’s capacity to create and maintain relations with its target groups and other actors in civil society.

Apart from the fact that the Octagon is a tool for rapid and simple analyses of an organisation’s strengths and weaknesses, the model also identifies necessary measures to improve the organisation’s capacity to perform effectively. If the same type of analysis is made systematically on several occasions over several years, it is also possible to follow changes in the organisation in question. In this way, the Octagon can be used both for “base-line studies” and for measuring changes and results of internal organization development over a certain period of time.

SenseMaker

SenseMaker links micro-narratives with human sense-making to create advanced decision support, research and monitoring capability in both large and small organisations.

SenseMaker lies at the heart of a range of applications. It has been extensively used in research, providing an quantitative approach to what has traditionally been a qualitative domain. Its origins lie in weak signal detection and understanding the impact of culture on decision making, work that continues to this day. Employee satisfaction, citizen journaling, attitudinal auditing (such as ethics and safety) have all been created using SenseMaker as the core.

Increasingly SenseMaker provides decision support capability permitting whole of workforce engagement and the creation of human sensor networks (wisdom of crowds in the popular literature) to enable a whole new approach to evidence based policy under conditions of uncertainty as well as real time decision support. This combines with knowledge management capability ranging from field operations to issues around the ageing workforce.