Dina Solovyova
The information obtained during the expert survey is not a ready-made examination; it must be processed, and only after that it can be considered as a solution to the problem.
Marketing phenomena and processes for the most part relate to weakly structured systems that cannot be unambiguously described and formally studied. When analyzing and forecasting such systems, one cannot do without the use of experts. It is here, more than anywhere else, that their deep knowledge in a given subject area and, at the same time, intuition based on experience are needed. Great importance also has erudition, i.e. broad interdisciplinary knowledge of experts.
Expert methods of analysis and forecasting can be both informal and creative, since the analysis procedures do not have clear algorithms and often the expert himself cannot “sort them out,” or “softly formalized”, based on flexible algorithms.
Of course, expertise can be carried out by the marketing analyst himself if he has sufficient knowledge, but often the responsibility of the task or its specifics require the use of an independent opinion, and sometimes several, which can only be achieved by involving third-party experts. In the second case, it becomes necessary to solve the following problems:
Selection of qualified experts;
Choice effective way interaction of the research group with experts, and during group examination - experts among themselves;
Determining the method of processing and interpreting information received from experts.
Experts' Choice
To select experts, it is necessary, firstly, to formulate selection criteria (answer the question “Who can act as an expert on this problem?”), secondly, determine selection procedures (answer the question “How to determine the suitability of an expert necessary requirements?"). The main selection criteria include the following:
1. The level of competence of an expert in a given subject area, the indicators of which in total are:
· level and profile of education,
· work profile (connection with a given subject area),
· work experience in the profile (total work experience in the profile and work experience directly in this subject area),
· level of problems to be solved (correspondence of the position held to the nature and level of the problem encountered),
· quantity and quality of previously performed examinations, for example, forecasts that have come true.
2. The degree of objectivity and impartiality of the expert when analyzing and assessing phenomena in a given subject area (the expert’s disinterest in making a certain decision).
3. The ability to work in a team (this is especially important for a group expert survey), i.e.:
· communication skills,
· ability to co-create,
· flexibility of mind and open-mindedness of view,
· nonconformism.
When assessing experts according to the above criteria and selecting experts, the following procedures can be used:
1. Self-assessment of experts based on objective parameters.
2. Mutual assessment of experts.
3. Expert assessment by independent specialists.
4. Assessment of the level of competence of experts, taking into account the quality of previously conducted examinations (this assessment is carried out by the research team itself based on the analysis of retrospective data on the expert’s work).
The most preferable is the selection of experts on the basis of an independent qualified assessment, adjusted for the quality of previously performed examinations.
Organization of expert interaction
The organization of interaction between experts and researchers depends on the chosen method of collecting expert information. Examination methods in this context can be classified as follows:
Individual examination methods,
Group methods of examination.
Expert methods belonging to the first group involve individual work researchers with each of the involved experts. In this case, one expert can be involved if his qualifications are sufficient to remove information uncertainty on the problem, but usually several experts are involved to increase the reliability of the examination.
Individuality lies in the fact that experts do not gather together, do not get acquainted with the assessments of other experts, different experts can be interviewed regarding different aspects of the same problem, and the procedures for interviewing different experts may also be different. Most often, during an individual expert survey, the following methods are used:
1. Standardized expert survey. This method requires the research team to first clearly structure the problem and define a list of all questions to which unambiguous answers must be obtained. To implement the survey, a standardized questionnaire with questions is being developed closed type(with suggestions for answer options). Questioning can be carried out both during a personal conversation between the interviewer and the expert, and by “self-completion”. In this case, the presence of an interviewer is not necessary; the questionnaire can be sent by regular mail or e-mail, but a preliminary agreement with an expert on the survey is required. The method requires highly qualified research specialists at the stage of setting the problem and planning the study, but it is very simple in terms of organizing and conducting a survey, as well as in terms of processing the information received. The requirements for questionnaires (structure, wording of questions and answer options) are quite standard and similar to the requirements for non-expert level surveys. One of the main requirements is the use of generally accepted professional language and unambiguous interpretation of the terms used.
2. Non-standardized expert survey. The method is a personal interview with an expert on a specific issue. The degree of formalization of the interview may vary. Low level formalization of the survey - an informal conversation for which only the topic is determined, and then the expert himself decides how to cover it (the interviewer asks clarifying or leading questions). A high level of formalization involves the development of a clearly structured questionnaire with questions open type. This method, compared to the previous one, is more complex both at the stage of conducting the survey (requires highly qualified interviewer) and at the stage of interpreting the information received and requires highly qualified researchers.
3. “Individual notebook” method. The method consists of correspondence work by an expert without direct communication with researchers. The expert receives a notebook, on the first page of which the problem is described, and then, for a specified period of time (determined by the complexity of the problem and the urgency of its solution), he writes down all his thoughts, ideas, and comments regarding the task in this notebook, after which he hands the notebook over to the researchers. Subsequent processing of information and its interpretation poses significant difficulties. The method requires significant expert involvement and therefore involves high level payment for his work.
Unlike individual methods, group methods involve teamwork experts (full-time or correspondence), they require coordination of the opinions of all experts and the development of a common expert conclusion based on consensus. Group methods are preferable from the point of view of increasing the reliability of the examination, but they are very complex to prepare and conduct. Highly qualified specialists are required to develop a group interaction procedure. It is not always possible to collect at one time and in one place required amount experts who meet the required requirements.
Group methods for developing expertise, depending on the nature and focus of the discussion, are divided into analytical and creative. Analytical methods are aimed primarily at studying the characteristics of the object being studied. Creative people have as their goal the collective generation of ideas or the development of a solution to a problem. Expert groups are classified accordingly:
Discussion groups (the main purpose of the work is analytical),
Creative groups (the main goal is creative).
Group methods for forming expertise are very diverse; we will describe the main ones:
1. Nominal group method. The method is a kind of transitional variation from an individual survey to a group one. When implementing this method, first an individual interview of some experts is carried out, and then the results of the interview data are also discussed autonomously and independently of each other by other experts. Experts can express agreement or disagreement with previously expressed opinions; it is necessary that criticism or expressions of solidarity be clearly reasoned.
2. Brainstorming. The method is a joint face-to-face discussion of a problem by a group of experts. The method is implemented in two stages. The first stage is called the “conference of ideas”, its duration is approximately 1-1.5 hours. During this stage, experts put forward various ideas regarding the interpretation of the analyzed situation and or forecast of the development of the phenomenon. Ideas are recorded, but not discussed or criticized. At the same time, ideas can be very different, including “delusional”. The guiding principle is: what more ideas, all the better. After a break, at the second stage, ideas are discussed, evaluated, and those that are considered the most correct are selected. The final verdict on an issue can be reached by explicit or implicit voting. The procedures for generating and discussing ideas can be more or less formalized.
3. Method "635". The method is a fairly formalized variation of the brainstorming method. This method implies the following regulation of the work of the expert team: the group includes 6 people, each of whom, within 5 minutes, must put forward three proposals or express three hypotheses regarding some aspect of the problem being solved or the situation being analyzed. The ideas of each expert are recorded in special forms that are passed around. After all aspects of the task have been considered and all experts have had the opportunity to speak, the solutions are discussed and evaluated and the most correct one is selected.
4. Critical attack ("split" attack). The method is also a variation of the brainstorming method, fundamental difference- in the critical focus of the discussion. Implementation of the method includes several stages. At the first stage, each member of the expert group offers his own solution to the problem (his own interpretation when analyzing the situation) or his own version of the development of events (when making a forecast). The solution must be proposed with detailed arguments. Next, each expert must familiarize himself with the opinions of his colleagues and find and argue for the maximum possible number of weaknesses in the proposed solutions. On next stage The experts get together and take turns discussing all the proposed solutions. The task of each author is to defend his version of the solution, the task of opponents is to “tear it to smithereens.” Based on the results of the discussion, the experts choose the solution that caused the least criticism and was the most justified.
5. Expert Focus. The method is one of the forms of joint face-to-face discussion of the problem. Experts comprehensively consider the situation under study and “focus” on it. The main goal is to identify the structure of this problem, to determine, if possible, all the factors that determine this situation, establish relationships between them. The discussion is more business-like in nature than in the classic version of brainstorming, that is, it takes place without unnecessary “nonsense.”
6. Commission method. The method also involves discussing the problem together. The main difference from focusing is the desire to find out what the contradiction is between different options for proposed solutions, to find the maximum number of “points of agreement” and come to a consensus.
7. Method of solution integration. The method is basically similar to the commission method, but is more formalized. The method consists of developing a joint solution to a problem by identifying the strengths of individual solutions and combining them. The method is implemented in several stages. At the first stage, experts are presented with a problem, and they consider and solve it independently of each other. Then the experts enter their individual decisions into a pre-prepared form, i.e. interpretation of the analyzed situation or forecast of the development of events. At the next stage, experts jointly discuss the problem and all proposed solutions in order to identify strengths each individual decision, which are also recorded in the form. Upon presentation individual solutions variations are possible - either each decision is presented by the author and argued in detail, or the anonymity of decisions is maintained in order to avoid pressure from authorities. Once all solutions have been discussed and the strengths of each have been identified, a synthesized solution is developed by combining the advantages of the individual solutions.
8. Business game. The method can be implemented in different forms. The most common form is modeling of the analyzed processes and/or future development of the predicted phenomenon in different options and review of the data obtained. Developing a procedure for conducting a business game is enough difficult task, and should be given serious attention. The following elements of the game must be clearly defined and formally described: goals and objectives, roles of participants, plot and regulations. An important stage Any business game is reflection - analysis of the course of the game and summing up. In this case, reflection consists not only in the analysis of the game process itself, but also in the analysis of the results of modeling the phenomenon under study.
9. The "trial" method. The method is one of the varieties of business games. The discussion of the task at hand is implemented in the form of a trial: a “problem trial” is simulated. The “lawyer”, “prosecutor”, “court”, “jury” and other participants in the “trial” are selected. Everyone defends their point of view regarding the analyzed or predicted phenomenon, giving reasons for their statements. The final verdict on the problem under study is determined in two stages: voting by the “jury” and concretization of the decision by the “judges”.
10. "Consilium". Experts examine the problem in the same way that doctors examine a patient: the “symptoms” of the problem are determined, the causes of the problem are revealed, an analysis is performed, a “diagnosis” is made, and a forecast for the development of the situation is given.
11. "Collective notebook." The method is basically similar to the “individual notebook”, but in this case the notebooks are received by several experts, each of whom knows that he is a member of the expert group. An option is possible when, at the beginning of the work, all the experts gather together and are told about the essence of the problem that has arisen and formulate the task. Next, each expert works with his notebook for a certain time (it is also possible that different experts focus on different sides Problems). The second stage of the implementation of the examination is that notebooks are collected, the information is systematized (by the research team or the head of the expert group) and then, in a face-to-face joint discussion of the accumulated and systematized material, the experts come to a solution to the problem.
12. Delphi method. The method is an absentee and anonymous survey of an expert group in several rounds with the coordination of expert opinions. Experts are offered questionnaires on the problem under study. The degree of standardization of questions can be different (they can be either closed or open, imply both quantitative and qualitative answers). Variations are also possible in terms of argumentation and substantiation of expert assessments (which may or may not be mandatory). As a rule, the Delphi method is implemented in 2-3 rounds, and during repeated surveys, experts are asked to familiarize themselves with either the opinions and arguments of each expert, or with the average rating. In repeated rounds, experts may change their assessment, taking into account the arguments of their colleagues, or they may remain with their previous opinion and express reasonable criticism of other assessments. Exist various techniques coordination of expert assessments (taking into account (or without) the qualifications of experts (as weighting coefficients), with discarding (or without) extreme assessments, and others). The Delphi method has very significant advantages that sometimes make it indispensable. Firstly, absenteeism and anonymity avoid conformity or orientation towards authorities, which could arise if experts were brought together and they had to make their opinions public. Secondly, experts have the opportunity to change their opinion without the risk of “losing face.”
Possible options
When developing expertise, it is possible to use either one of the named methods or various combinations of them, which makes sense if the problem under discussion is too complex. For example, the following combination of expert procedures is possible:
1. Collective notebook - familiarization of experts with the problem, individual reflection on the problem and preparation of experts for group discussion.
2. Group discussion (focusing, brainstorm etc.) - joint study of the problem, putting forward and considering hypotheses, choosing the most acceptable hypothesis.
3. Survey using the Delphi method - assessing the phenomenon within the framework of the selected hypothesis.
Let's summarize the results
The information obtained during the expert survey is not a ready-made examination; it must be processed, systematized, assessed in terms of quality, subjected to analysis and targeted interpretation, and only after that it can be considered as a solution to the problem. The quality of the collected expert information is assessed based on the traditional requirements for marketing information - relevance, completeness and reliability. The quality of expert information, in principle, can be improved if you use group expert surveys in several rounds (especially with substantiation of opinions), which allow you to gradually bring together expert assessments, clarify and supplement opinions. It is believed that the smaller the spread of opinions, the higher the reliability of expert assessments, which can be determined by calculating the standard deviation. It is possible to determine the reliability of estimates indirectly, based on the stability of expert opinions. If expert assessments change sharply from one round to another, then the reliability of such assessments is low.
One of the most critical stages in processing the collected information is the coordination of expert opinions, which can be done based on one of the following rules:
Majority rule - the assessment of the phenomenon or the solution to the problem that the majority of experts adhere to is chosen (however, it should be noted that there are often situations when experts who give more reliable assessments find themselves in the minority);
The rule of average assessment - expert opinions are reduced to a certain common denominator. For quantitative assessments, this is quite simple: either a simple or weighted average assessment is determined; for qualitative information, such coordination is more difficult. When agreeing on quantitative estimates, discarding the lowest and highest estimates and averaging the remaining ones can also be used.
When coordinating expert opinions and developing the final examination, it is important to comply with the following requirements: consistency and mutual consistency of examination parameters, validity and consistency of conclusions, completeness of the solution to the task. Compliance with these requirements ensures high quality examination, subject, of course, to the appropriate competence of the experts.
In general, the reliability of the formed examination is determined by two main factors. Firstly, the qualifications of experts, their ability to solve problems of a given level of complexity. Secondly, the qualifications of the research team, its ability to select competent experts, “extract” and accumulate expert knowledge about the problem.
Marketing analysis and forecasting with the help of experts have both undoubted advantages and quite tangible ones." narrow places"Among the advantages, we should especially emphasize the possibility of obtaining unique information that cannot be gleaned from any other sources. The problems of expert surveys are as follows:
1. The complexity of organizing an examination: selecting experts in sufficient quantity and quality and conducting a survey.
2. The difficulty of forming a final examination: coordination of the data obtained, their analysis and interpretation.
3. Possible subjectivity of experts: experts may be captive of their ideas and be reluctant to reconsider their point of view, even if it is incorrect.
4. Possible influence on the result of the chosen form of conducting an expert survey (with an open survey there is a high risk of conformity).
5. The high cost of conducting a survey, because both the remuneration of experts and the costs of organizing and conducting the examination are high.
Due to the high cost and complexity of conducting an expert survey, the choice of this method must be strictly justified. It makes sense to involve experts only to solve non-trivial large-scale problems that require an independent, objective assessment of the situation, as well as to develop solutions that cannot be obtained in any other way.
Expert assessment is the name of a whole system of diagnostic methods that are extremely widely used in management, economic analysis, psychology, marketing and other areas. These methods allow you to characterize, classify, assign a certain rank or rating to events and concepts that cannot be measured quantitatively.
In what cases is expert assessment necessary?
During any research, at any of its stages, method B can be applied management activities it may come in handy:
- At the stage of determining the goals and objectives of the research process.
- During the construction or testing of a hypothesis.
- To clarify the problem situation. To interpret ongoing processes and events.
- To justify the adequacy of the tools used.
- To generate recommendations, as well as to implement many other purposes.
Carrying out expert assessment justified in cases where it is impossible to make a decision based on accurate calculations (for drawing up a psychological portrait, work characteristics, assessing economic uncertainty and risks).
Most often, the use of such assessments becomes important in the situation of choosing one or more options from the proposed set:
- Launch of serial production of one of the developed product variants.
- Selection of astronauts from numerous applicants.
- scientific work which will be financed.
- Selecting an enterprise that will receive an environmental credit.
- Determination of an investment project for investing funds.
Who are the experts and how do they work?
As the name of the method suggests, expert assessment involves the involvement of one or more specialist experts who are competent to make assessments of persons, as well as the processing of their opinions. The selection of experts is carried out taking into account the adequacy of their judgment and experience in this field.
Expert assessment can be expressed both quantitatively and qualitatively. Expert research data is needed by executives, managers and management employees as a basis for decision-making.
The development of an expert assessment is most often carried out by creating a working group that organizes the activities of an expert (or several experts). If more than one person has to be involved, they are combined into an expert commission.
How many experts will be needed?
Depending on the specifics of the task and the capabilities of the enterprise, one or more experts may be invited to conduct an expert assessment. In this case, the expert assessment is called individual or collective.
The assessment used by the teacher to characterize the depth of the student’s knowledge becomes individual. This type also includes a diagnosis made by one doctor. However, in controversial or difficult situations (serious illness, raising the issue of expelling a student), they resort to a collective solution to the issue. Here, symposiums of doctors and the organization of a commission of teachers are necessary.
The same algorithm operates in the army: most often the decision is made by the commander alone, but if necessary, a military council is assembled.
Sequence of the assessment procedure
The sequence of formation of a relevant and objective expert assessment consists of the following stages:
- The activity that needs to be investigated.
- Selection of experts to carry out the procedure.
- Studying existing methods, with the help of which expert assessments will be measured.
- Carrying out the assessment procedure itself.
- Compilation and analysis of information obtained during the assessment.
In this case, it may be necessary to check the input data on which the expert assessment will be based. In some cases, the working group has to change the composition of the expert group or resort to re-measuring the same issues (in order to subsequently compare the resulting assessment with objective data from other sources).
Progress of assessment: characteristics of stages
Competent resolution of organizational issues is of great importance for the successful implementation of the procedure:
- Planning the costs of the event (payment for the services of experts and specialists in analyzing the data obtained, expenses for renting premises, purchasing office supplies).
- Preparation necessary materials(drawing up and printing forms, providing equipment).
- Selecting and instructing an event moderator.
In the process of work, experts must be guided by the allocated regulations, since additional time for making a decision does not affect its accuracy.
When responses from all specialists have been received, an assessment is performed expert opinion. In this case, the degree of consistency of all opinions is taken into account. If there is no clear agreement, the working group must find out the reason for the disagreement, record the formation of several groups of opinions and the lack of consistency as a result of the expert assessment. Then the research error is estimated and a model is built based on the data that was obtained. This is necessary so that later analytical examination can be carried out.
Methods used to conduct individual expert assessment: what is an interview
Among the most effective and frequently used techniques are:
- Analytical method.
- Script writing method.
- Interview.
In accordance with the interview methodology, the forecaster talks with the expert, asking him questions. The subject of conversation becomes the prospects for the development of an object or phenomenon about which we're talking about. The questionnaire program is developed in advance.
The effectiveness and quality of expert assessment directly depend on whether the expert can provide an opinion under limited time conditions.
Carrying out examination by analytical method
When choosing an analytical method to carry out an assessment, the expert specialist must prepare to carefully carry out independent work. He will have to analyze trends, assess the state and possible ways development of the object in relation to which forecasting is applied.
The system of expert assessments involves the study of all information about the object that is available to the expert. The result is formatted as
Main advantage analytical method it becomes that the specialist can demonstrate all his individual abilities.
True, this method is not suitable for analyzing large and complex systems, since the expert may lack knowledge from related fields.
Carrying out expertise by writing scripts
Strictly speaking, this method should not be classified only as an individual assessment method, since it is successfully used for group work.
To use this method, the expert should determine the logic of the processes and phenomena being studied relative to time and different combinations of conditions. Then he will be able to establish the hypothetical sequence of events (their development, transition from the situation to this moment to the predicted state). The scenario reflects all stages of solving the problem, and also provides for the emergence of possible obstacles.
Collective expertise: brainstorming method
To evaluate complex, large-scale, multi-level systems, it is impossible to do without the involvement of several expert specialists.
They can complete the assigned task using one of the following methods:
- Collective generation of ideas (“brainstorming”).
- Method "635".
- Delphi method.
- Commission estimates.
Thanks to collective efforts and a special organization, experts can effectively carry out the most complex procedures, such as expert risk assessment for an investment project or forecasting the activities of various systems.
“Brainstorming” allows you to fully reveal the creative potential of experts. At the first stage, specialists actively generate ideas, then apply destructuring (criticize them, destroy them), put forward counter-ideas and develop a consistent point of view.
The main condition is the absence of criticism at the beginning and the expression of all spontaneously arising ideas.
Specifics of the “635” method
The method received this name because of the technique that experts use when using it: each of the six experts writes down three spontaneously arising ideas on a piece of paper over a period of five minutes.
What is special about the Delphi method?
The purpose of developing this method of expert assessment was the need for a more rigorous and justified procedure that could give an objective and most useful result.
It is used by experts invited to scientific and technical institutes, investment and Insurance companies, as well as in a number of other cases.
The essence of the method is that multi-round individual surveys are conducted (often using questionnaires). Then a computer analysis of expert assessments is performed to form a collective opinion. At the same time, arguments to defend each judgment are identified and summarized.
At the next stage, the results obtained are transferred to experts for adjustments. Their disagreement with the collective judgment must be justified in writing. As a result of repeatedly returning the assessment for adjustment, the working group achieves a narrowing of the range and the development of a consistent judgment regarding the development prospects of the object under study.
What is good about this method:
- The experts taking part in the assessment do not know each other and do not communicate. Thus, their interaction is excluded.
- The results of previous rounds are also of interest and value to the working group.
- It is possible to obtain statistical characteristics of group opinion.
Despite the relatively high cost and duration, this method is becoming the best way to predetermine the development of long-term problematic situations.
Often the assessment is carried out by a specially organized commission (commission method), which for “ round table“consider all aspects of the problem and make an agreed decision. The disadvantage is the influence of participants on each other and distortion of the results. An example would be the expertise of teachers and doctors.
Other methods
The most common methods of performing examination were listed above, but others are also used in the practice of industrial, scientific and research organizations.
Depending on the specifics of the situation that needs to be predicted, as well as on the resources and capabilities of the enterprise, the following can be applied:
- Business game. It allows you to simulate the required number of situations to study the features of a control system or other processes.
- “The Trial” is a re-enactment of a trial in which some experts defend the solutions, while others try to refute them.
- Report method - after the analysis, the expert expresses his opinion in the form of an analytical note or report. This is relevant when it is necessary to carry out relatively simple work (for example, an expert assessment of a car for insurance, taxation or damage compensation).
As a result, it can be noted that the existence large quantity methods and methods of conducting expert assessment allows the head of the enterprise and the working group to choose the most effective option to solve a specific problem.
Often it is necessary to choose among many alternatives, each of which has various advantages. And how can you choose the best one, having the opinion of dozens, or even hundreds of experts?
How to calculate rating computer game, based on critics’ assessments of graphics, gameplay and plot, and the collective choice of a priority task before the appearance of the customer, refers to the methods of expert assessments.
Brief educational program
Expert assessment methods are part of the broad field of decision theory, and itself expert assessment- the procedure for obtaining an assessment of the problem based on the opinion of specialists (experts) for the purpose of subsequent decision-making (choice).In cases of extreme complexity of the problem, its novelty, insufficiency of available information, the impossibility of mathematical formalization of the solution process, one has to turn to the recommendations of competent specialists, excellent knowledgeable about the problem, - to the experts. Their solution to the problem, argumentation, formation of quantitative estimates, processing of the latter by formal methods are called the method of expert assessments.
Exists two groups of expert assessments:
- Individual assessments are based on the use of the opinions of individual experts, independent of each other.
- Collective assessments are based on the use of collective expert opinion.
Ways to measure objects
- Ranging- this is the arrangement of objects in ascending or descending order of some inherent property. Ranking allows you to select the most significant factor from the set of factors being studied.
- Paired comparison- this is the establishment of preference for objects when comparing all possible pairs. Here, as in ranking, there is no need to order all objects; it is necessary to identify a more significant object in each pair or establish their equality.
- Direct assessment. It is often desirable not only to order (rank the objects of analysis), but also to determine how much one factor is more significant than others. In this case, the range of changes in the characteristics of an object is divided into separate intervals, each of which is assigned a certain score (score), for example, from 0 to 10. That is why the direct assessment method is sometimes also called the point method.
a ij is the expert’s assessment of the attribute. n is the number of features, m is the number of experts.
Then, S i is calculated - the average value of the importance of the attribute.
Method for setting weight coefficients (a ij)
- all characteristics are assigned weighting coefficients so that the sum of the coefficients is equal to some fixed number (for example, one, ten or one hundred);
- the most important of all features is given a weighting coefficient equal to some fixed number, and all the others are given coefficients equal to fractions of this number.
- the expert arranges all features in decreasing order of their importance: A1>A2>…>An;
- assigns a value equal to one to the first characteristic: A1=1, and assigns weighting coefficients to the remaining characteristics in fractions of one;
- compares the value of the first attribute with the sum of all subsequent ones.
IN pairwise comparison There is no need, as in ranking, to order all objects; it is necessary to identify a more significant object in each pair or establish their equality. Paired comparison can be carried out with a large number of objects, as well as in cases where the difference between objects is so insignificant that ranking them is practically impossible.
When using the method, a matrix of size n x n is most often compiled, where n is the number of objects being compared. 
When comparing objects, the matrix is filled with elements a ij as follows (another filling scheme can be proposed):
- 2, if object i is preferable to object j (i > j),
- 1, if equality of objects is established (i = j),
- 0 if object j is preferred to object i (i< j).
And now, the best part...
Analysis of the results of expert assessments
Various methods of mathematical statistics are used to analyze the results. Moreover, they can be combined and vary depending on the type of task and the desired result.Formation of a generalized assessment
So, let a group of experts evaluate an object, then x j is the assessment of the jth expert, where m is the number of experts.To form a generalized assessment of a group of experts, average values are most often used. For example, median, which is taken to be such an estimate in relation to which the number of large estimates is equal to the number of smaller ones.
Determining the relative weights of objects
Sometimes it is necessary to determine how important (significant) a particular factor (object) is from the point of view of some criterion. In this case, they say that it is necessary to determine the weight of each factor. Different from formation generalized assessment the fact that it is not the overall assessment of the object that is determined, but the assessment for each of its features.
And
There is a huge variety possible methods processing ratings.
Alternatively, use the Elo rating system for the method paired comparisons.
Moreover, the result may consist of several algorithms, intertwined with others. For example, the algorithm for calculating the expert’s competence coefficient can influence the average statistical assessment of this expert, etc.
Establishing the degree of consistency of expert opinions
If several experts participate in a survey, discrepancies in their assessments are inevitable, but the magnitude of this discrepancy has important. A group assessment can only be considered sufficiently reliable if there is good agreement between the responses of individual experts.To analyze the spread and consistency of estimates, we use statistical characteristics – measures of dispersion or statistical variation.
So, methods for calculating the scatter measure:
Variational scope
Average linear deviation 
Standard deviation 
Dispersion 
Spearman's rank correlation coefficient 
The coefficient (value) can vary in the range from –1 to +1. If the estimates completely coincide, the coefficient equal to one. The coefficient is equal to minus one when there is the greatest discrepancy in expert opinions.
x ij – rank ( importance), assigned to the i-th object by the j-th expert, x ik is the rank assigned to the i-th object by the k-th expert, d i is the difference between the ranks assigned to the i-th object.
Kendell's coefficient of concordance
The coefficient can take values ranging from 0 to 1. With complete agreement of expert opinions, the concordance coefficient is equal to one, while complete disagreement is zero. The most realistic case is the case of partial agreement of expert opinions.
Calculation
The average rank of the set of characteristics is determined:
The deviation d j of the average rank of the j-th characteristic from the average rank of the population is calculated:
The number of identical ranks assigned by experts to the j-th attribute – t q – is determined.
The number of groups of identical ranks is determined - Q. The concordance coefficient is determined by the formula: 
Where 
Speaking about the consistency of expert opinions, it is worth mentioning that ranking does not (or does not always imply) distance. That is, for one expert, A>B>C means that A>>B>C, and for another, A>B>>C. And all sorts of correlations and calculations of average ratings will not help here. Alternatively, consider the consistency index. Something like the number of contradictory closed chains of expert opinions (the first believes that A is better than B, the second that B is better than C, and the third that C is better than A) to the number of all such chains.
Conclusion
The article does not pretend to be a complete multi-stage analysis of evaluation methods and algorithms, only a superficial description of them. Therefore, if you know the methods and algorithms applicable in this case (not described by me), I will be happy to add them to the article. Or any useful thematic literature.So I take my leave. Happy holiday everyone, Ramin. And for those who came to look at the girls - here you go
Expert method.
1.4. Threshold values are determined based on the opinions of expert groups.
The advantage of the method is simplicity and flexibility, since the assessment procedure allows you to analyze heterogeneous indicators in any quantity.
However, the assessment results largely depend on the qualifications and experience of the appraiser. Therefore, the defining moment and main problem This method is the selection of competent employees and/or the search for experts who will be tasked with assessing potential.
Expert method - This is a set of logical and mathematical-statistical methods and procedures associated with the activities of experts in processing the information necessary for analysis and decision-making.
IN practical activities on studying and forecasting demand method expert assessments can be used to solve the following problems: 1) Development of medium- and long-term forecasts of the group structure of demand for consumer goods. 2) Forecasting the intra-group (in the expanded assortment) structure of demand for the upcoming business year. 3) Identification of groups of potential consumers. 4) Assessment of the amount of unsatisfied demand by groups, types and varieties of goods.
An expert is a competent person who has deep knowledge about the subject or object of research.
How to form a group of experts? At the very first stage of selection, it is advisable to use two criteria as criteria: occupation and work experience in the profile of interest to us. If necessary, the level, nature of education, and age are also taken into account. The central criterion for selecting experts is their competence. To determine this, two methods are applicable, with varying degrees of accuracy: self-assessment of experts and collective assessment of the authority of experts.
The simplest and most convenient form of self-assessment of experts is a cumulative index, calculated based on the experts’ assessment of their knowledge, experience and abilities on a ranking scale with the positions “high”, “medium” and “low”. In this case, the first position is assigned the numerical value “1”, the second – “0.5”, the third – “0”. In this case, the cumulative index - the coefficient of the expert’s competence level is calculated using the formula.
The competence level coefficient ranges from 0 to 1. Usually, it is customary to include in a group of experts those with a competence index of at least the average of 0.5 and above -1.
The collective assessment method is used to form a group of experts in the case where they have an understanding of each other as specialists. This situation is typical for scientists, creative figures, politicians, and economists.
Forecast. It most clearly demonstrates the difference between expert assessment and information obtained as a result of a mass survey. It lies in the desire for consistency, uniformity of judgments and assessments expressed by experts. Indeed, is it possible to use for practical purposes, say, the opinion of thirty experts if they include 5-7 mutually exclusive prognostic assessments? Further, the larger the population of respondents, the higher the reliability of data in a mass survey, as well as some average statistical indicators. In principle, predictive expert assessment is feasible for any social processes and phenomena.
In applied sociology, a number of methods of expert surveys have been developed that are used to obtain a predictive assessment. At the same time, it is appropriate to note that some technical and methodological techniques widely used in mass surveys lose their significance when surveying such a specific audience as experts. As a rule, mass surveys are anonymous. In expert surveys, this loses its meaning, because experts must be fully aware of the tasks that are being solved during the study with their help. Therefore, there is no need to use indirect or Control questions, tests or any other techniques aimed at identifying the “hidden” positions of the respondent. Moreover, the use of such techniques can cause significant damage to the quality of expert assessment. An expert in the full sense of the word is an active participant in scientific research. And an attempt to hide the purpose of the study from him, thus turning him into a passive source of information, is fraught with the loss of his trust in the organizers of the study.
The main tool for an expert survey is a questionnaire or interview form developed using a special program.
Unlike a mass survey, the program of a predicted survey of experts is not so detailed and is predominantly conceptual in nature. First of all, it clearly formulates the phenomenon to be predicted and provides possible options for its outcome in the form of hypotheses.
Quite often used in applied sociology is such a method of expert forecasting as the “Dolphin technique”. It consists in developing consistent opinions by repeating a survey of the same experts many times. After the first survey and generalization of the results, its results are communicated to the participants of the expert group. Then a repeat survey is conducted, during which the experts either confirm their point of view or change the assessment in accordance with the majority opinion. This cycle contains 3-4 passes. During such a procedure, an assessment is developed, but the researcher, of course, should not ignore the opinion of those who, after repeated surveys, remained at their point of view.
Assessing the degree of reliability of the results of mass surveys. In production management decisions With the help of sociological research, the question often arises about the reliability of mass research and, accordingly, about the legitimacy of the conclusions formulated on their basis. In short, we are talking about assessing the competence of the opinions expressed by respondents.
For this purpose, an expert questionnaire is compiled, which includes mainly closed questions, identical in structure to the questions formulated in the respondent’s questionnaire. The expert’s task is, taking into account the objective situation and factors of interest to the researcher, to give an unbiased, comprehensively balanced assessment on the questions posed.
Certification of team members. IN last years In the practice of studying the state of ideological and educational work, a type of expert assessment method called certification is widely used. In this case, team leaders take on the role of experts, public organizations or a special certification commission.
STEPS:
1. Statement of the problem
2. Technology for organizing expert analysis
3. Formation of an expert commission
4. Organization of an expert survey
5. Formal methods for describing object preferences
6. Formal methods for determining preferences
7. Mathematical methods processing of examination results
8. Assessing expert agreement
During management production system Situations constantly arise when managers different levels(from master to minister) are faced with the need to choose one of several possible courses of action. Development and decision-making is a key procedure in the activities of a manager, which determines the entire further course of the management process, especially the final result of management activities.
Many factors that determine or influence the choice of a solution are by their nature not amenable to quantitative characteristics, others practically cannot be measured. All this made it necessary to develop special methods, facilitating the choice of management decisions in complex technical, organizational, economic problems (methods of operations research, expert assessments, etc.).
1.4. Ranking as method of collective expert assessments. The essence of the procedure is as follows: experts (specialists in the field under study) are asked to rank (arrange phenomena in ascending (descending) order of importance) of certain phenomena (price-forming factors). In this case, each phenomenon (factor) is assigned its own rank (ordinal place of the phenomenon (factor) in the general sequence of factors). Ranks are usually denoted by ordinal numbers in the natural series (1, 2, 3, 4......). In this case, rank 1 is assigned to the most significant factor.
The sum of ranks assigned by an expert for all factors can be calculated using the following formula:
ri – rank assigned to the i-th factor;
m – number of studied phenomena (factors).
The sums of ranks assigned by each expert must be equal. If an expert assigns equal ranks to two (three and so on) different phenomena, i.e. considers two phenomena (factors) to be equivalent (related ranks), then the sums of ranks assigned by each expert will not coincide
which in turn does not allow further calculations.
To avoid violation of the above equality, the appraiser conducting the examination needs to recalculate such ranks into the so-called standardized ranks. The calculation is carried out by dividing the sum of places occupied by related ranks by their number. An example of the calculation is presented in Table 1.
As a result, we obtain a normal ranking for which the main ranking conditions (1) are met.
Next, for each phenomenon, the sum of the ranks assigned to each expert is calculated. The phenomenon (factor) that received the smallest sum of ranks is assigned a corresponding rank of 1, etc. If at this stage for some phenomena the sums of ranks coincide, then it is necessary to carry out the procedure of standardizing the ranks again, but according to the ranks obtained by summing up the expert assessments. An example of ranking is presented in Table 2. It should be noted that the number of experts (d) must be at least 1 greater than the number of factors under study (m).
| Factors | ||||||||
| Expert | sum | |||||||
| Standardized ranks | ||||||||
| 3,5 | 3,5 | |||||||
| 4,5 | 4,5 | |||||||
| 4,5 | 4,5 | |||||||
| Sum of ranks (r1) | 23,5 | 31,5 | ||||||
| Factor ranks (R) | ||||||||
| Standardized final ranks | 1,5 | 1,5 | 6,5 | 6,5 |
Obtaining ranks allows for further calculations of weighting coefficients for each pricing factor when making adjustments in qualimetric models and other methods of the comparative approach. However, the final conclusion about the possibility of using the collective opinion of experts is possible after determining the consistency of expert opinions. The consistency of opinions is checked using the coefficient of concordance (agreement). The coefficient is calculated using the following formula:
Where,
m – number of factors being assessed;
d – number of experts;
Ri – sum of ranks for the i-th factor (phenomenon);
The closer the coefficient value is to 1, the higher the consistency. The acceptability of the obtained expert assessments is determined by comparing the significance criterion x2fact=d*(m-1)*W with the table value x2tab. with degrees of freedom (m-1) and a given probability (for example: Po = 0.05, P=1-Po). If the value obtained by calculation is equal to or higher than the table one, then the concordance coefficient is significant and with a reliability of 0.95, expert opinions are consistent. Table values of the significance criterion are presented in Table 3.
| m-1 | P0 | m-2 | P0 | ||||
| 0,05 | 0,01 | 0,001 | 0,05 | 0,001 | |||
| 3,84 | 6,63 | 10,83 | 26,3 | 39,25 | |||
| 5,99 | 9,21 | 13,81 | 27,59 | 33,41 | 40,79 | ||
| 7,81 | 16,27 | 28,87 | 34,8 | 42,31 | |||
| 9,49 | 13,28 | 18,46 | 30,14 | 36,19 | 43,82 | ||
| 11,07 | 15,09 | 20,52 | 31,41 | 37,57 | 45,31 | ||
| 12,59 | 16,81 | 22,46 | 32,67 | 38,93 | 46,8 | ||
| 14,07 | 18,47 | 24,32 | 33,92 | 40,29 | 48,27 | ||
| 15,51 | 20,09 | 26,12 | 35,17 | 41,63 | 49,73 | ||
| 16,92 | 21,67 | 27,88 | 36,41 | 42,98 | 51,18 | ||
| 18,31 | 23,21 | 29,59 | 37,65 | 44,31 | 52,62 | ||
| 19,67 | 24,72 | 31,26 | 38,88 | 45,64 | 54,05 | ||
| 21,03 | 26,22 | 32,91 | 40,11 | 46,96 | 55,48 | ||
| 22,37 | 27,69 | 34,53 | 41,34 | 48,28 | 56,89 | ||
| 23,68 | 29,14 | 36,12 | 42,56 | 49,59 | 58,3 | ||
| 30,58 | 37,7 | 43,77 | 50,89 | 59,7 |
x2fact = 44.016 > x2table = 12.59
Thus, the expert opinions given in our example are quite consistent and can be used in further calculations.
A weighting factor can be assigned to each factor using following formula:
ri – final factor rank.
In our example, the weights would be distributed as follows (Table 4).
| Factors | ||||||||
| sum | ||||||||
| Rank | 1,5 | 1,5 | 6,5 | 6,5 | ||||
| Weight | 0,232 | 0,179 | 0,143 | 0,232 | 0,107 | 0,054 | 0,054 |
The presented method of distributing weights does not pretend to be objective, and has a number of disadvantages, in particular: the distribution of weights is based on the ranking of factors, which in turn allows factors to be distributed only with a given interval, i.e. the method gives the answer that factor “A” is more important than “B”, and “B” is more important than “C”. However, according to this, factor “A” is twice as important as factor “C,” which may not be true.
The expert method for determining quality indicators is based on taking into account the opinions of specialist experts. An expert is a specialist who is competent in solving specific task(from the Latin word expertus - experienced). This method is used in cases where quality indicators cannot be determined by other methods due to insufficient information, the need to develop special technical means and so on.
The expert method is a combination of several various methods, which are its modifications. Well-known varieties of the expert method are used where the basis of the decision is the collective decision of competent people (experts). For example, solutions various advice, conferences, meetings, commissions, as well as examiners when assessing students’ knowledge, etc. - These are decisions made by expert methods. When using the expert method, the following factors are important: the competence of experts when forming an expert group; methods of interviewing experts; methods of information exchange; types of scales used for assessment. For expert assessment, it is advisable to use scales with an odd number of gradations, in which, in addition to the lower and upper, there is an average (satisfactory) level.
The expert method can be used in forming a general assessment (without detail) of the level of product quality, as well as in solving many specific issues related to the determination of quality indicators. The areas of application of the expert method are as follows:
grade regulatory documents for products;
generalized assessment of product quality;
classification of goods being valued;
determination of the range of quality indicators of the products being assessed;
determination of weight coefficients for quality indicators:
selection of basic samples and determination of the values of basic quality indicators;
measurement and assessment of quality indicators using the senses (organoleptically);
assessment of single indicators, the values of which are determined by measurement or calculation methods;
product identification;
determination of a complex quality indicator based on a set of individual (generalized and group indicators);
assessment of product competitiveness;
product certification.
The expert method is not used if quality can be assessed by other methods with greater accuracy or less cost.
The results of the expert assessment have elements of uncertainty and unreasonableness. The reliability of the assessment results depends on the competence and qualifications of the experts. The expert’s competence consists of professional and qualimetric competence. Professional competence provides knowledge:
development history of the products being evaluated, changes in their properties and quality indicators;
conditions for design (construction) and production (processing) of products;
values of quality indicators of domestic and foreign analogues;
prospects for product development, reflected in research works, patents, and design developments;
consumer requirements, conditions and nature of consumption (operation) and disposal.
Qualimetric competence requires a clear understanding of the approach to assessing the quality of goods, methods of quality assessment, issues of constructing and applying rating scales, determining subjective probabilities and distinguishing a sufficient number of gradations of the assessed object.
An expert's qualifications are determined not only by knowledge of the subject of discussion, but also by the ability to express clear, unambiguous judgments. The specific capabilities of the expert are taken into account. For example, in the food industry, when assessing the quality of food products, they take into account the expert’s ability to perceive taste, smell, etc., as well as his state of health. Experts assessing aesthetic and ergonomic quality indicators must be knowledgeable in the field of artistic design.
When using the expert method to assess quality, a working and expert group is formed. The working group organizes the procedure for interviewing experts, collects questionnaires, processes and analyzes expert assessments.
The expert group is formed from highly qualified specialists in the field of creation and use of the products being assessed: commodity experts, marketers, designers, constructors, technologists, etc. The number of experts included in the group depends on the required accuracy of average assessments, the permissible labor intensity of assessment procedures, the capabilities of group management and organization , in which a group is formed, but the group must have at least 7-12 people. If it is necessary to increase the accuracy of quality assessments, the composition of the group can be increased to 15 - 20 experts.
It is desirable that the expert group is formed not for one examination, but as a permanently functioning body with a fairly stable composition of experts.
During the group’s work, experts, based on an analysis of the results of previous work, develop general approaches and principles for assessing product quality, which increases the effectiveness of the assessment.
When assessing quality, experts are asked to construct a hierarchical nomenclature of quality indicators, and it is advisable to start the assessment from the level of consideration at which there are single indicators for which objective assessment methods exist.
When constructing a nomenclature of quality indicators, it is advisable to fulfill the following four conditions:
the characteristic by which any complex indicator is divided into n indicators below the level (classification characteristic) must be the same for all n indicators. This provides experts best opportunities to compare indicators when determining weight coefficients (the next operation after constructing the nomenclature);
when determining weight coefficients, the expert compares the importance various indicators qualities included in a homogeneous group. Since this operation becomes difficult with a significant number of indicators, which reduces the objectivity of the results, the number of indicators included in a homogeneous group should not exceed 10;
if a quality indicator is repeated at two or more levels, then its weight coefficient is considered overestimated. Therefore, repetition of indicators is undesirable;
the number of quality indicators included in homogeneous groups: at the same level, the nomenclature of quality indicators should not differ sharply, since an increase in the number of indicators can lead to a decrease in the values of the weight coefficients. For example, in the nomenclature of quality indicators at the first level there are three indicators (S = 1, 2, 3), at the second level each of them consists of a certain number of indicators - respectively q, I, p. It is desirable that the condition q I p be satisfied or that these numbers be sufficiently close.
Possible errors can be reduced by including explanatory note necessary clarifications.
The work of experts consists of performing two independent but interrelated operations - adjusting (clarifying) the nomenclature of quality indicators and determining the weighting coefficients of these indicators.
After studying the presented range of quality indicators, each expert comes to one of the following conclusions:
The nomenclature contains quality indicators, the weight of which is insignificant. They can be ignored when assessing quality (the expert crosses out these quality indicators);
the nomenclature does not include sufficiently important indicators (the expert enters these indicators into the nomenclature);
indicators that can be considered unimportant should be deleted and at the same time the nomenclature should be supplemented with indicators that are quite important;
The nomenclature of quality indicators is compiled correctly.
The expert writes down the reasons for his actions on a separate sheet attached to the nomenclature, or states them to the technical worker who fills out this sheet.
After analyzing the opinions of experts by members of the working group, the nomenclature is adjusted and again sent to the experts to determine the weighting coefficients of quality indicators.
The determination by experts of the weighting coefficients of quality indicators begins with ranking, when each quality indicator is assigned a certain rank. If a homogeneous group includes four or more indicators, experts pre-rank them: rank 1 is assigned to the most important indicator, rank 2 to the next most important, etc. If the expert believes that the weight of two or more indicators is the same, then he assigns them the same ranks. If there are less than four indicators, the ranking operation can be omitted.
Each expert becomes familiar with the weight coefficients assigned by other experts and their justifications. If the expert group is formed from employees of various organizations who are difficult to gather for joint work, then each expert attaches a brief rationale to the completed questionnaire. Since this procedure is very labor-intensive, it is recommended to use it when the number of indicators included in block diagram, is relatively small (about 10... 15). Otherwise, experts are invited to provide justification for only some of the weight coefficient values at their discretion. To simplify the procedure, you can refuse justifications and request them only if necessary.
Each expert gets acquainted with the anonymous opinions of other experts and again puts down the values of the weight coefficients.
At the expert group meeting, an open discussion of all weighting factors is held. All experts have the opportunity to briefly justify their judgments about the significance of the weight coefficient of each indicator and criticize other opinions. To eliminate the possible influence of official position on the opinions of experts, it is desirable that experts speak in sequence from junior to senior (by official position). After the discussion, the experts record in the documents the values of the weight coefficient of the quality indicator and move on to discussing the next indicator.
Based on the results of assessing weight coefficients, the consistency of expert opinions is assessed using coefficients of variation, concordance, etc.
The values of the coefficients of variation are determined by: the number of levels in the nomenclature of quality indicators, the variety of consumer opinions, the competence of experts, etc. When determining weight coefficients, we can recommend the following values of coefficients of variation, obtained based on an analysis of the results of the work of various expert groups: V< 0,10 - согласованность высокая; V= 0,11 ...0,15 - согласованность выше средней; V=0,16...0,25 - согласованность средняя; V= 0,26...0,35 - согласованность ниже средней; V >0.35 - consistency is low.
If the consistency of expert opinions is average or above average, then we can proceed to assessing the consistency of the expert group’s opinions on the weight of all indicators. If consistency is below average, additional analysis is required. The reasons for the low consistency of expert opinions can be subjective and objective. Subjective ones include: insufficient awareness of experts about the quality indicator, the weight coefficient of which is determined; unclear understanding of the problem being solved; arithmetic errors of experts, etc.
Objective reasons - the classification of consumers, on which experts rely when determining weight coefficients, is insufficient. Therefore, it is necessary to define the conditions of consumption more clearly.
To identify the reason for low consistency, the weight coefficients of this indicator are re-determined with discussion and the coefficient of variation is again calculated. If after this procedure the coefficient of variation has not improved (or has improved slightly), then the obtained values of the weight coefficients should be discussed with justification of expert opinions and clarification of the consumer group.
The purpose of the discussions is to strive to develop a common understanding among all experts of the nature of the work and issues, to identify the true opinion of each expert, and not to seek to bring together the values of the weight coefficients that experts assign. Thus, the coefficient of variation to a large extent is not a measure of “disadvantage” in opinions, but a measure of the differences in the requirements of individual groups of consumers for the products being evaluated.
