[section separator="true"]
[section-item 9]
[row]
[column 12]
[toc-this]
Please see also pages on [link title="deciding%20to%20use%20a%20survey" link="%2Faware%2FPA%2FPages%2FPlanning%2FDeciding-using-survey.aspx" /]
and [link title="planning%20and%20setting%20up%20a%20survey" link="%2Faware%2FPA%2FPages%2FExamination%2Fsurveys.aspx" /]
.
Principles
In order to make the most out of survey data we analyse it through the use of appropriate methods.
We document the analysis to ensure it is reproducible.
We use the results as [link title="evidence" link="%2Faware%2FGAP%2FPages%2FAudit-evidence.aspx" /]
and present them in a fair, balanced and transparent way in our reports.
Instructions
Prepare the data
Examine the raw data that comes out of the survey. This may involve spotting and resolving data inconsistencies or errors in order to ensure a high level of data quality. It is often the most time-consuming part of the analysis. The amount of data preparation and the particular techniques you have to employ will depend on the analysis you envisage and the nature of the data set. In the process, you should conserve as much of the original data as possible while ensuring that the data set is suitable to allow you to employ the envisaged analytical methods to give relevant and reliable results.
Proceed as follows, making sure you document the modifications you make to the original data:
- back-up the original data set before you start modifying it;
- screen your data for inconsistent, invalid, missing or outlier data. The most common issues are that dates or numerical values were
[link new-window title="not%20recognised" link="https%3A%2F%2Fsupport.microsoft.com%2Fen-us%2Foffice%2Fconvert-numbers-stored-as-text-to-numbers-40105f2a-fe79-4477-a171-c5bad0f0a885" icon="external-link" /]
properly. Check also for [link new-window title="duplicate" link="https%3A%2F%2Fsupport.microsoft.com%2Fen-us%2Foffice%2Ffind-and-remove-duplicates-00e35bea-b46a-4d5d-b28e-66a552dc138d" icon="external-link" /]
rows and remove them. Identify variables with a high share of missing data, you might remove such variables from the file for further analysis. Document any of the modifications either manually if you work in Excel or dedicated tools generate a log of procedures, which you should save to your audit file;
- transform variables into formats which are suitable for your analytical procedures, e.g. the survey tool provides you with textual answers to rating questions. If you keep the textual answers in your data set, you can still analyse the frequency distribution by preparing a bar chart, but such a format will prevent you from analysing relationship(s) with other variables, because this type of analysis requires numerical values;
- explore quantitative data using descriptive statistics. The aim is to get a broad idea of how each variable is distributed. As a minimum, check the mean, median and modal values, because these can reveal invalid data points. There is a range of other techniques available, for instance, consult
[link new-window title="Statistical%20terms%20and%20concept%20by%20Australian%20Bureau%20of%20Statistics" link="https%3A%2F%2Fwww.abs.gov.au%2Fstatistics%2Funderstanding-statistics%2Fstatistical-terms-and-concepts" icon="external-link" /]
or [link new-window title="The%20beginner%E2%80%99s%20guide%20on%20statistical%20analysis%20on%20Scribbr" link="https%3A%2F%2Fwww.scribbr.com%2Fstatistics%2Fdescriptive-statistics%2F" icon="external-link" /]
; and
- code the answers to open-ended questions.
Code answers to open-ended questions
Categorise the answers to open-ended questions in order to be able to aggregate the data or analyse it alongside the rest of the survey data.
This will involve the following steps:
- establish a categorisation scheme, which describes the relevant coding categories through a definition or set of examples. Assign to each category a unique label or number;
- code the answers;
- test the reliability of the coding. For instance, another person in the team could code a sub-sample of answers and you compare the results. The higher the degree of agreement between the two coding sets, the more reliable the coding approach is likely to be; and
- merge the codes you assigned to each response with the other survey data for further analysis.
Choose the right analytical method
Your analytical method will depend on what you aim to prove or disprove, and the type and nature of the data.
Analytical aims
Basic one-dimensional analysis by type of data
For nominal data, use frequencies or percentages. Do not calculate mean or standard deviation on such data. Use a pie chart if the number of categories is small and they are not similar in size, otherwise use a bar [link title="chart" link="%2Faware%2FGAP%2FPages%2Fcharts.aspx" /]
.
For ordinal data, use frequencies and percentages. Use a histogram, which presents the data in the natural order of the categories.
For interval or ratio data, use mean and standard deviation. If you want to communicate the full spectrum of values, use a histogram or boxplot.
Advanced methods
The types of variables you have usually determine what type of statistical test you can use. When identifying the suitable multi-dimensional analytical method, refer to the literature on the subject or online sources, e.g. [link new-window title="Scribbr" link="https%3A%2F%2Fwww.scribbr.com%2Fstatistics%2Fstatistical-tests%2F" icon="external-link" /]
. If you need help with multi-dimensional analysis, consult the DATA team.
Compare with results of other methods
If the other method led to results different from the survey results, treat such a situation as [link title="two%20inconsistent%20pieces%20of%20audit%20evidence" link="%2Faware%2FGAP%2FPages%2FAudit-evidence.aspx%23Corroboration" /]
. Possible courses of action include disregarding the responses to the respective question or undertaking a further audit procedure.
Document in Assyst
Include in your [link title="documentation" link="%2Faware%2FGAP%2FPages%2FDocumenting-an-audit.aspx" /]
:
Report and clear the essential only
Include in the [link title="scope%20and%20approach" link="%2Faware%2FPA%2FPages%2FReporting%2FScope-and-approach.aspx" /]
section:
- the objective of the survey or topics covered; and
- the target population, the distribution method and the response rate, including
- any limitations to the type and strength of evidence the results provide.
This allows the reader to place the results in their correct context, and helps them to gauge the reliance they wish to place on them.
When you use the data derived from the survey in the main text to support your findings, indicate the exact questions asked and the number of responses received.
If the survey results represent an important source of evidence, include a summary of the survey design and the results in an annex to the report.
[toggles]
[toggle title="Examples%20of%20additional%20information%20useful%20for%20readers%E2%80%99%20understanding%20if%20the%20survey%20results%20provide%20an%20important%20element%20of%20evidence"]
[/toggle]
[/toggles]
[link title="Clear" link="%2Faware%2FPA%2FPages%2FExamination%2FClearing.aspx" /]
with the auditee(s) the survey results that concern them and will be used in the report. Do not share individual responses with the member states’ authorities, unless you envisaged so and informed the respondents about it in advance. Do not share replies provided by entities in member states with the respective supreme audit institution.
Help improve our survey practice
Report back to the DQC survey support team your experience of designing and running the survey, including good practice and any problems you encountered and how you solved them. This will enable us to build a knowledge base and contribute to improving the guidance.
References
- Züll, Cornelia (2015)
[link new-window title="Offene%20Fragen" link="https%3A%2F%2Fwww.gesis.org%2Ffileadmin%2Fupload%2FSDMwiki%2FOffene_Fragen_Zuell_08102015_1.1.pdf" icon="external-link" /]
[link new-window title="The%20Beginner's%20Guide%20to%20Statistical%20Analysis" link="https%3A%2F%2Fwww.scribbr.com%2Fcategory%2Fstatistics%2F" icon="external-link" /]
Scribbr.com
[link new-window title="Statistical%20terms%20and%20concepts" link="https%3A%2F%2Fwww.abs.gov.au%2Fstatistics%2Funderstanding-statistics%2Fstatistical-terms-and-concepts" icon="external-link" /]
by Australian Bureau of Statistics
[/toc-this]
[/column]
[/row]
[/section-item]
[section-item 3]
[row]
[column 12][/column]
[/row]
[row]
[column 12][/column]
[/row]
[row]
[column 12]
[toc fixed="true" selectors="h2%2Ch3" class="basic-toc" /]
[/column]
[/row]
[/section-item]
[/section]