Pardee Wiki - User contributions [en]

UNSD Environmental Indicators

2023-11-20T17:45:21Z

Sami.McKinsey: Created page with "== Summary == UNSD Environmental Indicators disseminate global environment statistics on ten indicator themes compiled from a wide range of data sources. The themes and indicator tables were selected based on the current demands for international environmental statistics and the availability of internationally comparable data. Indicator tables, charts and maps with relatively good quality and coverage across countries, as well as links to other international sources, are..."

== Summary ==
UNSD Environmental Indicators disseminate global environment statistics on ten indicator themes compiled from a wide range of data sources. The themes and indicator tables were selected based on the current demands for international environmental statistics and the availability of internationally comparable data. Indicator tables, charts and maps with relatively good quality and coverage across countries, as well as links to other international sources, are provided under each theme.

Environment statistics is still in an early stage of development in many countries, and data are often sparse. The indicators selected here are those of relatively good quality and geographic coverage. Information on data quality and comparability is given at the end of each table together with other important metadata.

IFs uses UNSD Environmental Indicators to pull data related to wastewater and how populations are connected to wastewater, such as SeriesWasteWaterTreat%, SeriesWasteWaterSecTreat%, and SeriesWasteWaterCollect%.

== Pulling Instructions ==
Step 1) Navigate to the [https://unstats.un.org/unsd/envstats/qindicators UNSD Environmental Indicators homepage.]
[[File:UNSD IMG 1.png|center|thumb|840x840px|UNSD Environmental Indicators Homepage]]

Step 2) Scroll down to see the different themes that the UNSD has data series on. Themes are listed on the left and the datasets under that theme appear in the center.
[[File:UNSD IMG 2.png|center|thumb|833x833px|UNSD Datasets]]

Step 3) Most data for IFs will be under "Inland Water Resources". Select that option and a new list of data will load.
[[File:UNSD IMG 3.png|center|thumb|822x822px|Select Desired Topic]]

Step 4) Click directly on the desired series. This example will use "Population connected to wastewater treatment".
[[File:UNSD IMG 4.png|center|thumb|815x815px|Inland Water Resources Data]]

Step 5) An Excel file will immediately download. Now you can format the data to upload it into IFs. To import data into IFs, please follow the instructions found in the [[Importing data (general instructions)|Importing Data (general instructions)]] page.

File:UNSD IMG 4.png

2023-11-20T17:36:10Z

Sami.McKinsey:

Inland Water Resources Data

File:UNSD IMG 3.png

2023-11-20T17:24:11Z

Sami.McKinsey:

Select Desired Topic

File:UNSD IMG 2.png

2023-11-20T17:19:23Z

Sami.McKinsey:

UNSD Datasets

File:UNSD IMG 1.png

2023-11-20T17:17:49Z

Sami.McKinsey:

UNSD Environmental Indicators Homepage

IRF World Road Statistics

2023-11-20T17:00:19Z

Sami.McKinsey:

== Data Source Summary ==
The International Road Federation (IRF) maintains a database known as World Road Statistics (WRS), comprising over 200 series for over 200 countries. The database is a tremendous resource for infrastructure data, specifically transportation network data. WRS contains information on total road length, but also contains filters to view paved vs. unpaved roads, the kinds of traffic (commercial vs. private), the vehicles using the roadway, accidents and the consequences of them, and more.

To IFs, the IRF provides an important window into the transportation infrastructure of a country. A robust, paved road network makes it easier for a country's exporters to access ports, and for its citizens to find jobs if they are scarce locally. IFs uses three preprocessors from this data source: RoadsPavedKm, the total length of paved roads in a country measured in kilometers; RoadsTotalNetwork, the total length of roads paved or unpaved; and RoadsPaved%, the proportion of the total network which is paved.

=== Pulling Instructions: ===
Step 1) Navigate to https://datawarehouse.worldroadstatistics.org/users/login. If necessary, register. The registration process is short, and provides immediate access to the data warehouse[[File:IRF world Road.png|center|IRF World Road Statistics Login Page|thumb|735x735px]]Step 2) Select "Country Comparison - Multiple Years" under Chart Type.
[[File:WRS IMG 2.png|center|thumb|736x736px|Select Country Comparison (Multiple Years)]]
Step 3) Manually select each continent in the selection box.
[[File:WRS IMG 3.png|center|thumb|734x734px|Select each continent manually]]
Step 4) In the countries box, select "all countries".
[[File:WRS IMG 4.png|center|thumb|741x741px|Select "All countries"]]

Step 5) Under themes, select an option. Most IFs preprocessors will be under "Road Network". This example will continue with that selection.
[[File:WRS IMG 5.png|center|thumb|745x745px|Select desired option under "Themes"]]
Step 6) Under "Metrics", select desired series.

* For the preprocessor RoadPavedKm, select "Total Road Network- All Road Types - Paved"
* For the preprocessor RoadsTotalNetwork, select "Total Road Network- All Road Types - Total"
* For the preprocessor RoadsPaved%, select "Paved Network Ratio"

[[File:WRS IMG 6.png|center|thumb|749x749px|Select desired series under "Metrics"]]

Step 7) After those four selections are complete, the Country comparison page will look like this. Select "Show Graph".
[[File:WRS IMG 7.png|center|thumb|737x737px|Country completion page after selection]]

Step 8) A new page will populate.
[[File:WRS IMG 8.png|center|thumb|740x740px|WRS Graph Page]]

Step 9) Click on the three bar menu in the upper right corner of the graph, then "download as CSV" or "download as XLS".
[[File:Screenshot (3).png|center|thumb|733x733px|Download the data series]]

Step 10) An Excel sheet will download. The downloaded spreadsheet is in time series data. If necessary, pivot the Excel sheet to bring it into IFs import format. Now you can format the data to upload it into IFs. To import data into IFs, please follow the instructions found in the [[Importing data (general instructions)|Importing Data (general instructions)]] page.

IRF World Road Statistics

2023-11-20T16:59:58Z

Sami.McKinsey: /* Pulling Instructions: */

== Data Source Summary ==
The International Road Federation (IRF) maintains a database known as World Road Statistics (WRS), comprising over 200 series for over 200 countries. The database is a tremendous resource for infrastructure data, specifically transportation network data. WRS contains information on total road length, but also contains filters to view paved vs. unpaved roads, the kinds of traffic (commercial vs. private), the vehicles using the roadway, accidents and the consequences of them, and more.

To IFs, the IRF provides an important window into the transportation infrastructure of a country. A robust, paved road network makes it easier for a country's exporters to access ports, and for its citizens to find jobs if they are scarce locally. IFs uses three preprocessors from this data source: RoadsPavedKm, the total length of paved roads in a country measured in kilometers; RoadsTotalNetwork, the total length of roads paved or unpaved; and RoadsPaved%, the proportion of the total network which is paved.

=== Pulling Instructions: ===
Step 1) Navigate to https://datawarehouse.worldroadstatistics.org/users/login. If necessary, register. The registration process is short, and provides immediate access to the data warehouse[[File:IRF world Road.png|center|IRF World Road Statistics Login Page|frame]]Step 2) Select "Country Comparison - Multiple Years" under Chart Type.
[[File:WRS IMG 2.png|center|thumb|736x736px|Select Country Comparison (Multiple Years)]]
Step 3) Manually select each continent in the selection box.
[[File:WRS IMG 3.png|center|thumb|734x734px|Select each continent manually]]
Step 4) In the countries box, select "all countries".
[[File:WRS IMG 4.png|center|thumb|741x741px|Select "All countries"]]

Step 5) Under themes, select an option. Most IFs preprocessors will be under "Road Network". This example will continue with that selection.
[[File:WRS IMG 5.png|center|thumb|745x745px|Select desired option under "Themes"]]
Step 6) Under "Metrics", select desired series.

* For the preprocessor RoadPavedKm, select "Total Road Network- All Road Types - Paved"
* For the preprocessor RoadsTotalNetwork, select "Total Road Network- All Road Types - Total"
* For the preprocessor RoadsPaved%, select "Paved Network Ratio"

[[File:WRS IMG 6.png|center|thumb|749x749px|Select desired series under "Metrics"]]

Step 7) After those four selections are complete, the Country comparison page will look like this. Select "Show Graph".
[[File:WRS IMG 7.png|center|thumb|737x737px|Country completion page after selection]]

Step 8) A new page will populate.
[[File:WRS IMG 8.png|center|thumb|740x740px|WRS Graph Page]]

Step 9) Click on the three bar menu in the upper right corner of the graph, then "download as CSV" or "download as XLS".
[[File:Screenshot (3).png|center|thumb|733x733px|Download the data series]]

Step 10) An Excel sheet will download. The downloaded spreadsheet is in time series data. If necessary, pivot the Excel sheet to bring it into IFs import format. Now you can format the data to upload it into IFs. To import data into IFs, please follow the instructions found in the [[Importing data (general instructions)|Importing Data (general instructions)]] page.

File:Screenshot (3).png

2023-11-20T16:59:14Z

Sami.McKinsey:

Download the series

File:WRS IMG 8.png

2023-11-20T16:57:52Z

Sami.McKinsey:

WRS Graph Page

File:WRS IMG 7.png

2023-11-20T16:50:15Z

Sami.McKinsey:

Final Country Comparison page

File:WRS IMG 6.png

2023-11-20T16:47:13Z

Sami.McKinsey:

Select desired option under "Metrics"

File:WRS IMG 5.png

2023-11-20T16:44:58Z

Sami.McKinsey:

Select option under "Themes"

File:WRS IMG 4.png

2023-11-20T16:43:14Z

Sami.McKinsey:

Select "All Countries"

File:WRS IMG 3.png

2023-11-20T16:40:12Z

Sami.McKinsey:

Select each continent manually

File:WRS IMG 2.png

2023-11-20T16:37:16Z

Sami.McKinsey:

WRS Country Comparison Page

AQUASTAT, FAO

2023-11-02T23:34:49Z

Sami.McKinsey: Updated with screenshots from new Aquastat page.

__FORCETOC__

== SUMMARY ==
[https://www.fao.org/aquastat/en/ AQUASTAT] is the UN's Food and Agriculture Organization (FAO) global information system on water resources and agricultural water management. It collects, analyzes, and provides free access to over 180 variables and indicators by country and year from 1960. AQUASTAT plays an important role in monitoring of the UN's Sustainable Development Goal 6 that sets out to "ensure availability and sustainable management of water and sanitation for all". Additionally, AQUASTAT's new dissemination system allows users to download up to 100,000 data points and the data is made available yearly.

The data team uses AQUASTAT for a number of series including SeriesDesalinatedWater, SeriesLandCultivatedArea, and SeriesLandIrWaterLogged to name a few. To pull data, please follow the instructions below.

== GENERAL STEPS TO PULL DATA FROM AQUASTAT ==

Step 1.) Navigate to the home page of [https://www.fao.org/aquastat/en/ AQUASTAT FAO's Global Information System on Water and Agriculture],

Step 2.) Near the top of the page, click on the tab labeled "'''Database'''"
[[File:Aquastat IMG STEP 01.jpg|center|thumb|576x576px|AQUASTAT's Homepage]]

Step 3.) On the left hand side of the page, click on the tab labeled "'''Country Statistics'''"
[[File:Aquastat IMG STEP 03.jpg|center|thumb|574x574px|AQUASTAT's Database Homepage]]

Step 4.) On the left hand side of the page, click on the tab labeled "'''Database'''", under "Country Statistics"

THIS WILL OPEN A PAGE ON ANOTHER TAB
[[File:Aquastat IMG STEP 04 Actual.jpg|center|thumb|575x575px|AQUASTAT's Country Statistics Page]]
[[File:Aquastat 1.png|center|thumb|575x575px|AQUASTAT Homepage]]

Step 5.) Under the "Areas" section on the left side, select '''(World)''' to select all countries. You can also select different regions.
[[File:Aquastat4.png|center|thumb|575x575px|Select (World)]]

Step 6.) Under the "Year" section on the left, select each year to see it represented in the data.
[[File:Aquastat 3.png|center|thumb|575x575px|Select Years]]

Step 7.) To find your specific variable, click which "Variable Group" it is in, then the "Variable Subgroup", and finally your "Variable". You can also start typing your variable name in the "Select Variables" space and it will display that variable and similar ones.
[[File:Aquastat2.png|center|thumb|576x576px|The Variable Group of your Series]]
Step 8.) Download the data by clicking "Share/Download". You will then be given download options. Click "Excel". An Excel file will download. Now you can format the data to upload it into IFs. To import data into IFs, please follow the instructions found in the [[Importing data (general instructions)|Importing Data (general instructions)]] page.
[[File:Aquastat5.png|center|thumb|574x574px|Select "Download"]]

== SPECIFIC VARIABLE EXAMPLE: SeriesIrrigatedCropIntensity ==
Step 9.) On the left hand side, under "Year", select desired years of data. This example will use years 2000-2020. "'''Irrigation and drainage development'''"
[[File:Aquastat 7.png|center|thumb|629x629px|Select Years]]

Step 10.) Start typing '''"Irrigated cropping intensity"''' into the "Select Variable" box. '''"Irrigated cropping intensity"''' will be under variable group "Irrigation and drainage development" and variable subgroup "Irrigated crop area and cropping intensity". Select the variable and a blue check mark will appear.
[[File:Aquastat 6.png|center|thumb|627x627px|Select the Variable]]
Step 11.) To select all countries, make sure "World" is selected under "Area".
[[File:Aquastat 9.png|center|thumb|622x622px|Select "World"]]

Step 11.) Data will now populate the page on the right side.
[[File:Aquastat 10.png|center|thumb|617x617px|Data populates page]]

Step 12.) Select "Share/Download" and then select "Excel".
[[File:Aquastat 8.png|center|thumb|621x621px|Click "Download"]]

Step 13.) An Excel sheet will download. Now you can format the data to upload it into IFs. To import data into IFs, please follow the instructions found in the [[Importing data (general instructions)|Importing Data (general instructions)]] page
[[File:AQUASTAT Excel page.png|thumb|575x575px|center]]

File:Aquastat 8.png

2023-11-02T23:33:39Z

Sami.McKinsey:

Download Data

File:Aquastat 10.png

2023-11-02T23:31:38Z

Sami.McKinsey:

Data Display

File:Aquastat 9.png

2023-11-02T23:29:42Z

Sami.McKinsey:

Select "World"

File:Aquastat 6.png

2023-11-02T23:15:35Z

Sami.McKinsey:

Aquastat Variable Selection

File:Aquastat 7.png

2023-11-02T23:13:43Z

Sami.McKinsey:

Aquastat Year Selection

File:Aquastat5.png

2023-11-02T22:52:59Z

Sami.McKinsey:

Aquastat Download

File:Aquastat2.png

2023-11-02T22:47:13Z

Sami.McKinsey:

AQUASTAT variables

File:Aquastat 3.png

2023-11-02T22:40:55Z

Sami.McKinsey:

AQUASTAT Years

File:Aquastat 1.png

2023-11-02T22:31:37Z

Sami.McKinsey:

Aquastat Homepage

File:Aquastat4.png

2023-11-02T22:30:22Z

Sami.McKinsey:

Aquastat Area

Vetting data

2023-09-25T19:11:51Z

Sami.McKinsey:

'''Vetting Data'''

Vetting is the process of checking to assure the quality of the data we import into IFs. Even when data is imported through an automated application there could be errors or missing data in the imported series. Vetting the new data you are bringing into IFs requires you to compare the new series to the existing historical series in IFs.

There are a number of reasons for vetting data. One reason is that a country (or a number of countries) could be missed when importing due to not properly concording countries with the proper country concordance table. Make sure the IFs country concordance list (i.e., the relevant column in the '''Country '''Translation table in IFs.mdb) is correct and up to date and you are using the correct concordance list as listed in the DataDict. Additionally, make sure there are no missing years in the new data. Sometimes the year column is missed because the data is formatted as text and not as numbers, or the source simply did not provide an update for that year.  If this is the case, be sure to blend in the missing years in the IFs vetting tool. One more general issue is around the units. Make sure you perform any unit conversion required before you import the data. Comparing the new data with the current IFs series will give you an idea on any unit conversion that might be needed.

 

'''Vetting Data''' '''Checklist:'''

*Data in Access file and source Excel file match
*Use IFs vetting tool to compare new data to historical data
*Large discrepancies between new and old data are documented
*Check to see if there are big spikes in the new, imported data
*Blend countries/years with missing data in IFs vetting tool
*Zeroes in Access file are actually zeroes and not null values
*No missing countries or years
*DataDict contains no errors, Original Source has website name and Name in Source contains variable source
*Initials added to both Access and Excel file names, as well as in the DataDict notes
*Send Word file with vetting notes, along with original files to puller and project leads

'''Process:'''

#After the data series are imported into IFs, the IFsDataImport.mdb file is passed to another RA (assigned as a vetter) who brings the data from the IFsDataImport.mdb file into the model via the "Vet Imported Data" feature in the Extended Features menu
#The vetter uses the vetting tool to determine if there are any remarkable inconsistencies between the old and new data. This is a bit subjective, but necessarily so as the threshold for concern will be different depending on which series from which source is being vetted
#If significant errors are found that need to be corrected, and these errors had occurred during the import process, these are documented in the RA's vetting notes and the files are passed back to the original data puller to correct and re-send. 
#If no errors are found, the vetter blends columns for new years and preserves historical data points as appropriate, and saves the changes in the IFsDataImport file. 
#The vetter passes the IFsDataImport file (renamed to reflect series/batch update name and completion date) back to the project lead and data team supervisor
#Data team supervisor stores data import file for consolidation process

 

'''You can ensure data quality by following certain procedures''':

#For an updated table, open the existing and updated table side by side. Compare data points for all years and all countries to make sure there are no, or very small, differences.
#For all series, be sure to check large and important countries like USA, China or Germany.
#Check countries with similar name (like the two Congos and two Koreas), as these can sometimes get mixed up.  
#Check for zeros and make sure they are actually zeros in the source data. We take no data as an empty cell. If we have zeros that need to be data and must be a feasible value (e.g., GDP cannot be 0).
#Check the variable definition and make sure it makes sense and is in fact the right definition.
#Make sure percentages are below 100 (there are cases when percentages can be above 100, e.g., gross enrollment rate).
#Check values: GDP growth rate of more than 10% should raise a flag. For instances like this, check against the source data.
#Creating line graphs for the countries in a series is a great way to quickly check for transients. This can easily be done in Excel or Tableau.

 

'''Using the IFs Vetting Tool'''

IFs has a feature to do some basic initial vetting and is a tool that should be used in every vetting process. The vetting tool can be found by the following:

'''IFs -> MainMenu->Extended Features -> Vet Imported Data'''.
[[File:Vetting IMG 1.png|center|thumb|877x877px|IFs Main Page]]

The Compare Imported Table With Existing Table screen is then opened. Currently, the .NET version of IFs only compares tables that have been imported previously, as opposed to being able to open Access files. Make sure you [[Importing data (general instructions)|import data]] into IFs before vetting. Select a series from "New Imported Series" you would like to vet.
[[File:Vetting IMG 2.jpg|center|thumb|863x863px|Compare Imported Table with Existing Table]]
This will then populate the grids below with the new data (the data you are importing) and the old, historical data that is in IFs. The old data is in the first grid and the new data is in the second grid, as shown below.
[[File:Vetting IMG 3.jpg|center|thumb|861x861px|Vetting Data Page]]

Here you have a lot of different options to compare the new data with the old. The vetting tool will automatically mark any zeros in the table. You will want to double check to see if, in fact, these are suppose to be zeros or if they are just null values. If they are just null values, you can click the button, '''Delete Zeros.'''

Next, you need to click the button, '''Mark Differences between New and Old Data for same country-year. '''This will allow you to easily observe any differences, as the vetting tool will highlight them. You can also change the threshold that it will mark. The default is that it will mark any more than 10% difference. As you can see in the screenshot below, the values for Afghanistan in 2014 and 2015 are marked. You will want to go through all of the countries in each series you are importing to look at these marked differences and write down any that are very large.

You can also click the button '''Mark''' '''Year to Year Jumps in New Data, '''which is helpful to observe any large changes year-to-year.
[[File:Vetting IMG 4.jpg|center|thumb|851x851px|Vetting Methods]]

'''Blending'''

This vetting tool is also used to merge country and year columns between updated and existing Access tables. This is a necessity when blending to preserve the existing historical IFs values when there is no corresponding value in the new data series. For example, often times there will be an update from a source but it is missing multiple countries we already have data for. In this case, you can use the blending tool to blend in any historical country values that are not in the new data. The same can be done for missing years in the new data.  

Once you have thoroughly vetted a data series, you will want to blend the new and historical series together. This ensures that we do not lose any values from the historical series. To do this for the Country values, you will want to check the '''Select All '''box in the bottom right hand corner of the screen. This will automatically check all of the countries, as seen below. Once you have check this box, press the '''Blend Data Points '''button, and click '''Yes. '''This will update the new Access file with missing Country observations.

You will also want to do the same process for the '''Blend Columns (Years) '''button, if there are any missing years. This ensures that we retain the historical values.
[[File:Vetting IMG 5.jpg|center|thumb|848x848px|Blending Data in IFs]]

Vetting data

2023-09-25T19:06:09Z

Sami.McKinsey: Updated pictures to .NET version

'''Vetting Data'''

Vetting is the process of checking to assure the quality of the data we import into IFs. Even when data is imported through an automated application there could be errors or missing data in the imported series. Vetting the new data you are bringing into IFs requires you to compare the new series to the existing historical series in IFs.

There are a number of reasons for vetting data. One reason is that a country (or a number of countries) could be missed when importing due to not properly concording countries with the proper country concordance table. Make sure the IFs country concordance list (i.e., the relevant column in the '''Country '''Translation table in IFs.mdb) is correct and up to date and you are using the correct concordance list as listed in the DataDict. Additionally, make sure there are no missing years in the new data. Sometimes the year column is missed because the data is formatted as text and not as numbers, or the source simply did not provide an update for that year.  If this is the case, be sure to blend in the missing years in the IFs vetting tool. One more general issue is around the units. Make sure you perform any unit conversion required before you import the data. Comparing the new data with the current IFs series will give you an idea on any unit conversion that might be needed.

 

'''Vetting Data''' '''Checklist:'''

*Data in Access file and source Excel file match
*Use IFs vetting tool to compare new data to historical data
*Large discrepancies between new and old data are documented
*Check to see if there are big spikes in the new, imported data
*Blend countries/years with missing data in IFs vetting tool
*Zeroes in Access file are actually zeroes and not null values
*No missing countries or years
*DataDict contains no errors, Original Source has website name and Name in Source contains variable source
*Initials added to both Access and Excel file names, as well as in the DataDict notes
*Send Word file with vetting notes, along with original files to puller and project leads

'''Process:'''

#After the data series are imported into IFs, the IFsDataImport.mdb file is passed to another RA (assigned as a vetter) who brings the data from the IFsDataImport.mdb file into the model via the "Vet Imported Data" feature in the Extended Features menu
#The vetter uses the vetting tool to determine if there are any remarkable inconsistencies between the old and new data. This is a bit subjective, but necessarily so as the threshold for concern will be different depending on which series from which source is being vetted
#If significant errors are found that need to be corrected, and these errors had occurred during the import process, these are documented in the RA's vetting notes and the files are passed back to the original data puller to correct and re-send. 
#If no errors are found, the vetter blends columns for new years and preserves historical data points as appropriate, and saves the changes in the IFsDataImport file. 
#The vetter passes the IFsDataImport file (renamed to reflect series/batch update name and completion date) back to the project lead and data team supervisor
#Data team supervisor stores data import file for consolidation process

 

'''You can ensure data quality by following certain procedures''':

#For an updated table, open the existing and updated table side by side. Compare data points for all years and all countries to make sure there are no, or very small, differences.
#For all series, be sure to check large and important countries like USA, China or Germany.
#Check countries with similar name (like the two Congos and two Koreas), as these can sometimes get mixed up.  
#Check for zeros and make sure they are actually zeros in the source data. We take no data as an empty cell. If we have zeros that need to be data and must be a feasible value (e.g., GDP cannot be 0).
#Check the variable definition and make sure it makes sense and is in fact the right definition.
#Make sure percentages are below 100 (there are cases when percentages can be above 100, e.g., gross enrollment rate).
#Check values: GDP growth rate of more than 10% should raise a flag. For instances like this, check against the source data.
#Creating line graphs for the countries in a series is a great way to quickly check for transients. This can easily be done in Excel or Tableau.

 

'''Using the IFs Vetting Tool'''

IFs has a feature to do some basic initial vetting and is a tool that should be used in every vetting process. The vetting tool can be found by the following:

'''IFs -> MainMenu->Extended Features -> Vet Imported Data'''.
[[File:Vetting IMG 1.png|center|thumb|877x877px|IFs Main Page]]

The Compare Imported Table With Existing Table screen is then opened. Currently, the .NET version of IFs only compares tables that have been imported previously, as opposed to being able to open/compared Access files. Make sure you [[Importing data (general instructions)|import data]] into IFs before vetting. Select which "New Imported Series" you would like to vet.
[[File:Vetting IMG 2.jpg|center|thumb|863x863px|Compare Imported Table with Existing Table]]
This will then populate the grids below with the new data (the data you are importing) and the old, historical data that is in IFs. The old data is in the first grid and the new data is in the second grid, as shown below.
[[File:Vetting IMG 3.jpg|center|thumb|861x861px|Vetting Data Page]]

Here you have a lot of different options to compare the new data with the old. The vetting tool will automatically mark any zeros in the table. You will want to double check to see if, in fact, these are suppose to be zeros or if they are just null values. If they are just null values, you can click the button, '''Delete Zeros.'''

Next, you need to click the button, '''Mark Differences between New and Old Data for same country-year. '''This will allow you to easily observe any differences, as the vetting tool will highlight them. You can also change the threshold that it will mark. The default is that it will mark any more than 10% difference. As you can see in the screenshot below, the values for Afghanistan in 2014 and 2015 are marked. You will want to go through all of the countries in each series you are importing to look at these marked differences and write down any that are very large.

You can also click the button '''Mark''' '''Year to Year Jumps in New Data, '''which is helpful to observe any large changes year-to-year.
[[File:Vetting IMG 4.jpg|center|thumb|851x851px|Vetting Methods]]

'''Blending'''

This vetting tool is also used to merge country and year columns between updated and existing Access tables. This is a necessity when blending to preserve the existing historical IFs values when there is no corresponding value in the new data series. For example, often times there will be an update from a source but it is missing multiple countries we already have data for. In this case, you can use the blending tool to blend in any historical country values that are not in the new data. The same can be done for missing years in the new data.  

Once you have thoroughly vetted a data series, you will want to blend the new and historical series together. This ensures that we do not lose any values from the historical series. To do this for the Country values, you will want to check the '''Select All '''box in the bottom right hand corner of the screen. This will automatically check all of the countries, as seen below. Once you have check this box, press the '''Blend Data Points '''button, and click '''Yes. '''This will update the new Access file with missing Country observations.

You will also want to do the same process for the '''Blend Columns (Years) '''button, if there are any missing years. This ensures that we retain the historical values.
[[File:Vetting IMG 5.jpg|center|thumb|848x848px|Blending Data in IFs]]

File:Vetting IMG 5.jpg

2023-09-25T19:05:21Z

Sami.McKinsey:

Blending Data in IFs

File:Vetting IMG 4.jpg

2023-09-25T18:58:32Z

Sami.McKinsey:

Ways to Vet in IFs

File:Vetting IMG 3.jpg

2023-09-25T18:30:46Z

Sami.McKinsey:

Vetting Data

File:Vetting IMG 2.jpg

2023-09-25T18:23:47Z

Sami.McKinsey:

Compare Imported Table with Existing Table

Vetting data

2023-09-25T02:47:00Z

Sami.McKinsey:

'''Vetting Data'''

Vetting is the process of checking to assure the quality of the data we import into IFs. Even when data is imported through an automated application there could be errors or missing data in the imported series. Vetting the new data you are bringing into IFs requires you to compare the new series to the existing historical series in IFs.

There are a number of reasons for vetting data. One reason is that a country (or a number of countries) could be missed when importing due to not properly concording countries with the proper country concordance table. Make sure the IFs country concordance list (i.e., the relevant column in the '''Country '''Translation table in IFs.mdb) is correct and up to date and you are using the correct concordance list as listed in the DataDict. Additionally, make sure there are no missing years in the new data. Sometimes the year column is missed because the data is formatted as text and not as numbers, or the source simply did not provide an update for that year.  If this is the case, be sure to blend in the missing years in the IFs vetting tool. One more general issue is around the units. Make sure you perform any unit conversion required before you import the data. Comparing the new data with the current IFs series will give you an idea on any unit conversion that might be needed.

 

'''Vetting Data''' '''Checklist:'''

*Data in Access file and source Excel file match
*Use IFs vetting tool to compare new data to historical data
*Large discrepancies between new and old data are documented
*Check to see if there are big spikes in the new, imported data
*Blend countries/years with missing data in IFs vetting tool
*Zeroes in Access file are actually zeroes and not null values
*No missing countries or years
*DataDict contains no errors, Original Source has website name and Name in Source contains variable source
*Initials added to both Access and Excel file names, as well as in the DataDict notes
*Send Word file with vetting notes, along with original files to puller and project leads

'''Process:'''

#After the data series are imported into IFs, the IFsDataImport.mdb file is passed to another RA (assigned as a vetter) who brings the data from the IFsDataImport.mdb file into the model via the "Vet Imported Data" feature in the Extended Features menu
#The vetter uses the vetting tool to determine if there are any remarkable inconsistencies between the old and new data. This is a bit subjective, but necessarily so as the threshold for concern will be different depending on which series from which source is being vetted
#If significant errors are found that need to be corrected, and these errors had occurred during the import process, these are documented in the RA's vetting notes and the files are passed back to the original data puller to correct and re-send. 
#If no errors are found, the vetter blends columns for new years and preserves historical data points as appropriate, and saves the changes in the IFsDataImport file. 
#The vetter passes the IFsDataImport file (renamed to reflect series/batch update name and completion date) back to the project lead and data team supervisor
#Data team supervisor stores data import file for consolidation process

 

'''You can ensure data quality by following certain procedures''':

#For an updated table, open the existing and updated table side by side. Compare data points for all years and all countries to make sure there are no, or very small, differences.
#For all series, be sure to check large and important countries like USA, China or Germany.
#Check countries with similar name (like the two Congos and two Koreas), as these can sometimes get mixed up.  
#Check for zeros and make sure they are actually zeros in the source data. We take no data as an empty cell. If we have zeros that need to be data and must be a feasible value (e.g., GDP cannot be 0).
#Check the variable definition and make sure it makes sense and is in fact the right definition.
#Make sure percentages are below 100 (there are cases when percentages can be above 100, e.g., gross enrollment rate).
#Check values: GDP growth rate of more than 10% should raise a flag. For instances like this, check against the source data.
#Creating line graphs for the countries in a series is a great way to quickly check for transients. This can easily be done in Excel or Tableau.

 

'''Using the IFs Vetting Tool'''

IFs has a feature to do some basic initial vetting and is a tool that should be used in every vetting process. The vetting tool can be found by the following:

'''IFs -> MainMenu->Extended Features -> Vet Imported Data'''.
[[File:Vetting IMG 1.png|center|thumb|877x877px|IFs Main Page]]

The Compare Imported Table With Existing Table screen is then opened.

[[File:2019-07-25 (2).png|900x500px]]

Here, go to '''File>Open Database with Imported Tables, '''and open the Access file with the data series that you want to import into IFs. This Access file contains data series tables and a dictionary table with a list of new and/or updated data variables. This will then populate the top grid with every series within that Access file.

[[File:2019-07-25 (3).png|900x500px]]

Next, double click on one of the series (rows) in the top grid to display the data. This will then populate the new data (the data you are importing) and the old, historical data that is in IFs. The old data is in the first grid and the new data is in the second grid, as shown above.

Here you have a lot of different options to compare the new data with the old. The vetting tool will automatically mark any zeros in the table. You will want to double check to see if, in fact, these are suppose to be zeros or if they are just null values. If they are just null values, you can click the button, '''Delete Zeros.'''

Next, you need to click the button, '''Mark Differences between New and Old Data for same country-year. '''This will allow you to easily observe any differences, as the vetting tool will highlight them. You can also change the threshold that it will mark. The default is that it will mark any more than 10% difference. As you can see in the screenshot below, the values for Afghanistan in 2014 and 2015 are marked. You will want to go through all of the countries in each series you are importing to look at these marked differences and write down any that are very large.

[[File:2019-07-25 (4).png|900x500px]]

You can also click the button '''Mark''' '''Year to Year Jumps in New Data, '''which is helpful to observe any large changes year-to-year.

'''Blending'''

This vetting tool is also used to merge country and year columns between updated and existing Access tables. This is a necessity when blending to preserve the existing historical IFs values when there is no corresponding value in the new data series. For example, often times there will be an update from a source but it is missing multiple countries we already have data for. In this case, you can use the blending tool to blend in any historical country values that are not in the new data. The same can be done for missing years in the new data.  

Once you have thoroughly vetted a data series, you will want to blend the new and historical series together. This ensures that we do not lose any values from the historical series. To do this for the Country values, you will want to check the '''Select All '''box in the bottom right hand corner of the screen. This will automatically check all of the countries, as seen below. Once you have check this box, press the '''Blend Data Points '''button, and click '''Yes. '''This will update the new Access file with missing Country observations.

You will also want to do the same process for the '''Blend Columns (Years) '''button, if there are any missing years. This ensures that we retain the historical values.

[[File:2019-07-25 (5).png|900x500px]]

 

File:Vetting IMG 1.png

2023-09-25T02:42:56Z

Sami.McKinsey:

Vetting Page

Importing data (general instructions)

2023-09-25T02:19:00Z

Sami.McKinsey:

= Introduction =

This section describes the import and update processes for data tables in IFsHistSeries.Mdb. Updating World Value Survey data (IFsWVSCohort.MDB) requires following a slightly different procedure.

Data import for IFs is a <u>three step</u> process:
<ol style="list-style-type:lower-roman;">
<li>Extraction, preparation and blending of source data (and meta-data) into MS Access tables formatted for IFs database. This step might involve a blending process when data is updated.</li>
<li>Vetting of MS Access tables containing extracted data (and meta-data)</li>
<li>Merging of the MS Access table (and meta-data) to IFs master database (IFsHistSeries.Mdb   and Datadict.mdb)</li>
</ol>

In this section we shall discuss how to extract data from external sources, prepare extracted data for IFs database and blend extracted data with existing data. When updating an existing data table we usually preserve data not reported in the more recent source to maintain consistency between old and new data, a process we call <u>blending</u>.

There are three ways to import data from external sources and prepare it for IFs as a new series or as an update of an existing series:
<ol style="list-style-type:lower-roman;">
<li>Manual import, which basically involves reorganizing external data into an IFs template spreadsheet and copying that data to an MS Access table   </li>
<li>Automated import that uses IFs software application to extract data from Excel spreadsheets and prepare Access tables using a Country Concordance table between external data source and IFs.</li>
<li>Batch update, another IFs application feature that automates updating all IFs tables from the same external source as long as external data is available in a required format.</li>
</ol>

= Manual Import Using IFs Template Spreadsheet =

The steps for the manual import are:
<ol style="list-style-type:lower-roman;">
<li>Enter or reorganize external data into IFs template spreadsheet. The IFs team uses an Excel spreadsheet template containing 186 IFs countries, country FIPS code and year columns going from 1960 to 2014. The name of the Excel file is "Template Data Form 6.XLS." This file can be found in the IFs\Data folder. (Template spreadsheets can also be generated from IFs application's import screen)</li>
<li>Delete year column in the template spreadsheet which do not have any data.</li>
<li>Create an MS Access table with the same structure (i.e., containing the same year columns) as the template spreadsheet. This new Access table needs to be created inside "IFsDataImport.Mdb" file in the IFs\Data folder. If you are updating an existing table, you should give the table the same name as the existing table. For a new table, prefix the table name with "Series" and follow the naming convention described in section 2.2.</li>
</ol>

<u>Tip</u>: Rather than creating an altogether new Access table, take an existing data table, i.e., any of the IFsHistSeries.mdb tables, copy it to a new name and change the structure by going to "Design View" in MS Access. In fact, it is best to start from an existing table so that the data field specifications are strictly followed. To add a year to the new table, insert an empty column (in the design view), copy an existing year column, paste it on the empty column and change the column (or field) name to the label of the year you need. (If you are using an existing Access table to create a new table, please delete the Earliest and MostRecent columns from the Access table as this will be populated automatically by IFs application.)
<ol style="list-style-type:lower-roman;">
<li>Copy and paste the data from Excel template spreadsheet to Access table. Make sure you sort both the spreadsheet and the Access table alphabetically by country name before you copy and paste.</li>
<li>Add a row to the Datadict table inside IFsDataImport.Mdb file. If you are importing new data, create an empty row and add meta-data (please be as complete as possible). If you are updating data, copy the row from master copy of Datadict (DataDict.Mdb) and edit as needed (you must edit last IFs update)</li>
</ol>

<u>Tip: </u>You can skip steps iii, iv and v and use IFs' automated import (see next section for description) feature instead

'''Passing Extracted Data for Vetting'''

Extracted data, saved in <u>Data\IFsDataImport.Mdb, </u>need to be sent to the person who will do the vetting. Please also pass all source information (Excel spreadsheet data or pdf file/s or the website from which data is taken). Some initial scrutiny of the data before it is sent for vetting is recommended.

= Automated Import Using ImportXLS: Single Import =

External data available in Excel format known to IFs can be extracted into Access tables using IFs application's automated ImportXLS (the process works only for Excel files and hence the name) feature that work by importing one time series at a time. If you have subnational models broken out in IFs on your computer, you need to switch back to the full 186 country model. Instructions on how to do this can be found in the [[SubRegionalization_Handbook|SubRegionalization Handbook]]. This process will take around five minutes, so can be done as frequently as needed. 

This feature can be accessed through IFs menu system:

IFs -> MainMenu->Extended Features -> Import Data (Single Series).
[[File:Importing IMG 1.jpg|center|thumb|872x872px|IFs Main Menu]]

This opens up a form shown below:
[[File:Importing IMG 3.jpg|center|thumb|869x869px|Single Import Page]]

The form is self-explanatory for the most part. You have to start by opening the source Excel file (select "Choose File") and selecting the spreadsheet which contains source. Then click "Refresh Sheets" and choose the desired Excel sheet. You then need to choose one from the six options showing the format of the data organization in the source spreadsheet (see the right side of the import screen). The default of these six options (the one at the bottom) works in most cases, however you can compare each option to your data by selecting a data organization option and viewing the layout in the image directly below the final option.

The list of steps to follow when importing data can be found in the "Help" menu option, which opens up a dialog box with the basic list.

[[File:ImportDataInstructions.png|RTENOTITLE]]

This list is fleshed out here in more detail and should be followed carefully:

'''1. Open the Excel file containing formatted source data (option found on top bar)'''

'''2. Select one spreadsheet when the source file contains more than one'''

'''3. Select source data format using illustrated options'''

    Each option can be previewed by selecting it and seeing the visualization directly below

'''4. Identify the Excel cells containing country and year labels.'''

If you do not remember when it starts, you can click ''View Excel File ''to check - the file must be closed after viewing

The ''Series Name Start at'' field will only appear if you have selected a ''Multiple Series'' option in the ''Choose Excel Source Data Format'' field. In this case, you choose the column of the data series          you want to import. 

'''5. Data Conversion Operator and Factor'''

'''????'''

'''6. Choose aggregation rule'''

An aggregation rule tells the model how to aggregate country level data up to different country groupings that can be selected in the model. For instance, if you choose SubSaharan Africa as a region, the model needs to know whether to add, average, use a weighted average for all countries in the grouping. 

There are five different aggregation rules to choose from: POP, GDP, AVG, SUM, LND. If the dataset is being updated, the aggregation rule should populate. If you are importing a new series, you have to choose the aggregation rule. You can look in the Data Dict at similar series to get an idea of what should be chosen or below are brief explanations of each, along with a few examples of datasets that use each rule.

     1. POP (population): uses a weighted average in which population of a country is the weight; used in demographics, health, and education series; used in series with per capita designations or when the variable is based on a percent of households (HHs)

     2. GDP: uses a weighted average in which GDP at MER of a country is the weight; used in economic series and when price is a consideration in the dataset

     3. AVG (average): takes the average of each country's data; used in indices 

     4. SUM: sums each country's data; used in many energy and labor series and when the data is a count

     5. LND (land): uses a weighted average using land cover as weight

'''7. Choose Country Concordance table '''

Country Concordance lists are used in converting external data to IFs format. "Country Concordance" options are shown in a dropdown list at the bottom of the screen. The prepackaged lists that you see in the dropdown were built over the years from the major international databases like World Bank's WDI, UNESCO, IEA etc. If you do not see a list that you can use you need to either use a manual import described above in section 3.1.1 or leave ImportXLS for now, build and update a country list and then come back to import. Unless a data source has a large number of tables that we will be regularly updating we do not spend the time in building and updating a new country concordance table. As a general rule, decimals should be truncated after the 4th decimal place. 

'''8. Identify missing values, if any. Data source and names can be put in later. '''

'''9. Enter a name and a definition for the imported table. If the variable is new, the name must be different from any existing table name. '''

If you are importing a new variable into the model, ask the person who assigned you this task for a specific name. If not provided at first, this can be added in the Data Dict after the import is complete. If you would like to update an existing table rather than import a new one, you can select the name of the table from a dropdown box just above the variable name box. Selecting an existing table will automatically bring in dictionary information from the master copy of the data dictionary.

'''10. Click Import when you are ready. '''

'''11. Click on View Excluded after importing the data'''

You'll see the countries that were not imported, if any. This may have happened because the IFs model does not have those countries in the model or because the name was incorrect and did not concord. If the name was incorrect, go back into the source file and change the country name and re-import the data. 

'''12. After accepting imported data, you'll see a form to enter data dictionary information. '''

Here are some tips on what to enter in the different fields in the data dict that do not auto-populate from the import: 

''Series ''& ''Cohort'': the entry for Series is Yes and for Cohort is No

''Source Definition'': the exact definition from the data source

''Extended Source Definition'': any information in addition to the source definition, such as any calculations or modifications you or the creators of the dataset performed on the data 

''Name in Source'' & ''Code in Source'': only relevant for batch pulls since there are multiple series in the pull

''Notes'': enter your initials as data puller/importer and the data vetter will then enter their initials

''Forecast'': click Yes if the series imported is a forecast and not a historical data series
[[File:Importing IMG 4.jpg|center|thumb|912x912px|Data Dictionary]]

'''13. Imported data will go to IFs\Data\IFsDataImport.mdb.'''

This is the file you will send to the person responsible for vetting and merging the data into the model.

'''14. Dictionary information will go to IFs\Data\IFsDataDict.mdb. '''

Any changes that need to be made to the data dict can be made directly here after the data is imported.

== Passing Extracted Data for Vetting ==

Extracted data will be saved in <u>Data\IFsDataImport.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.

= Batch Update Using ImportXLSBatch =

Another form has been created that allows import from an Access table. This needs to be described, and shown.

Batch update, another IFs application feature, allows updating all IFs tables pertaining to a certain external source at once. This works only for those sources where data is available in a recognized format and each source series is identified with a fixed code that IFs data dictionary can save and match during subsequent updates. At this moment, batch update works for AQU, WDI, UNESCO (UIS), IMF and FAO.

The batch update feature can be reached through IFs main menu:

IFs -> MainMenu->Extended Features -> Manage Country Data->Historic Data File Processing-> Update Data in Batch.

The form that opens up is shown below. The grid on the right of the form will be empty initially, but fills up as soon as the user selects a source name from the dropdown box sitting above the grid. You can select all series for batch update (a checkbox is available for this) or decide to update the ones that you select by clicking on the "Update" column in the grid. There is an additional option of selecting only those variables per data source which are used in the Pre-Processor, which can be selected by checking the ‘Pre-Proc’ box on the batch import form seen below. Of course, the import process will update only those series that are found in the source Excel that you'll have to open using the menu at the top.

Batch update feature works by extracting multiple data series from a single spreadsheet at a time. From the box listing all spreadsheets in the workbook that you just opened, you will have to select the one that contains all the series dat.

Before using the batch import process, new variables must be added to the datadict and the source column of the datadict must be formatted for batch import by including one of the qualified series listed above (FAO, IMF, AQU, etc). Additionally, each data series in the external spreadsheet is matched with an IFs data table through a "series code" that is previously collected in the IFs "datadict" table. For WDI, these codes appear as combination of two to four letter code sections separated by periods (e.g., AG.PRD.FOOD.XD). For UNESCO education data, we take series names used by UIS, exactly as they type it, as series codes. For FAO data, Element code and Item code used by FAO are used by IFS (accordingly, there will be two series code  boxes in the batch update form when you are updating FAO data; identify Element code as the first one and Item code as the second one).

The decimal places box is used for specifying the decimal places that will be kept in extracted tables. As a general rule, decimals should be truncated after the 4th decimal place. The setting works for those tables which do not have such setting established already (as shown in the grid above).

[[File:BatchImportWizard.png|RTENOTITLE]]

The steps are explained in a Help dialog shown below:

[[File:BatchImportInstructions.png|RTENOTITLE]]

Clicking the "Import Batch" button will import (and update) the series that are selected in the grid one at a time and show a progress bar as this is being done.

Imported tables can be subjected to automated blending and vetting using the new vetting/comparison interface described later in this document.

== Source Data for Batch Update ==

World Development Indicator (WDI) data is released annually and is usually available as a single Excel file at their website:

[http://data.worldbank.org/data-catalog/world-development-indicators/ http://data.worldbank.org/data-catalog/world-development-indicators/]

FAO and UIS data do not come in single file as WDI. What we need to do is download multiple series on related issues (e.g., all cereal production data or all primary education data) as single Excel files. (Kate McGrath has prepared instructions on getting FAO and UIS data from their websites).

IMF data is available only by accessing our IMF e-Library account, registered to Mohammod Irfan.

== Passing Extracted Batch Data for Vetting ==

Extracted data will be saved in <u>Data\IFsDataImportBatch.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.
<div><div id="_com_1"></div></div>
= Automated Import Using ImportXLS: Province Import =

Importing data for a provincial model can be done with the Province Data Import Feature. IFs does not need to be sub-regionalized to use this feature.

The batch update feature can be reached through IFs main menu:

IFs -> MainMenu->Extended Features -> Manage Country Data->Historic Data File Processing-> Import Data from XLS Provinces

The form that opens up is shown below. The format reflects the single series import process already described. A recognized excel format must be selected from the list on the right. The location of Province names, Years, and Series from your chosen excel data sheet must be designated. The provincial country concordance table must also be selected from the drop down list. The form contains the option to view the excel sheet you are importing, and to view the country concordance tables. Lastly, the variable you are importing needs to be selected from the drop down list. Existing variables will populate the definition field. New variables will need to be named using the Ifs naming convention reviewed above, and defined.

== Passing Extracted Data for Vetting ==

Extracted data will be saved in <u>Data\IFsDataImport.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.

File:Importing IMG 3.jpg

2023-09-25T02:12:34Z

Sami.McKinsey:

Single Import Page

File:Importing IMG 4.jpg

2023-09-25T02:10:03Z

Sami.McKinsey:

Data Dict

File:Importing IMG 2.jpg

2023-09-25T02:00:47Z

Sami.McKinsey:

Single Import Page

File:Importing IMG 1.jpg

2023-09-25T01:53:54Z

Sami.McKinsey:

IFs Main Menu

Fiscal Monitor, IMF

2023-09-21T00:05:18Z

Sami.McKinsey:

=== IMF Fiscal Monitor Summary ===
The International Monetary Fund's Fiscal Monitor surveys and analyzes the latest public finance developments. Country-specific data and projections for key fiscal variables are based on the most recent World Economic Outlook database, unless indicated otherwise, and compiled by the IMF staff. Historical data and projections are based on the information gathered by IMF country desk officers in the context of their missions and through their ongoing analysis of the evolving situation in each country.

Datasets are updated on a continual basis as more information becomes available. IMF staff estimates serve as proxies when complete information is unavailable and therefore Fiscal Monitor data may differ from other sources' official data, even the IMF's International Financial Statistics. All fiscal data refer to the general government where available and to calendar years, except 32 countries, for which they refer to the fiscal year.

The data team uses IMF Fiscal Monitor for a number of series, including but not limited to SeriesGovtGenExp%GDPFM, SeriesGovtGenGDebt%GDPFM, SeriesGovtGenNDebt%GDPFM, SeriesGovtGenRev%GDPFM. To pull data, please follow the instructions below.

=== Fiscal Monitor Pulling Steps ===
Step 1. Navigate to [https://www.imf.org/external/datamapper/datasets/FM IMF Fiscal Monitor] site. Make sure the site is updated by checking the date next to “Fiscal Monitor”
[[File:IMF FM IMG 1.jpg|center|thumb|603x603px|Homepage of IMF Fiscal Monitor Database]]

Step 2. Scroll down past the text until you reach '''“Fiscal Indicators”'''. Click on the arrow on the right side below “Fiscal Indicators” to see additional data.
[[File:IMF FM IMG 2.jpg|center|thumb|609x609px|Fiscal Monitor Indicators]]

Step 3. This example will select '''“Revenue”''' from the indicators.
[[File:IMF FM IMG 3.jpg|center|thumb|611x611px|Fiscal Monitor Indicators Example]]

Step 4. After selecting the indicator, a new webpage will populate.
[[File:IMF FM IMG 4.jpg|center|thumb|618x618px|Revenue Indicator Example]]

Step 5. Scroll down until you reach the bottom of the webpage. There are a few download options but for most cases, you should select '''“EXCEL FILE”''' under '''“All Country Data”'''. This option will also download all available years.
[[File:IMF FM IMG 5.jpg|center|thumb|630x630px|Revenue Indicator Download]]

Step 6. An Excel sheet will download. Now you can format the data to upload it into IFs. To import data into IFs, please follow the instructions found in the [[Importing data (general instructions)|Importing Data (general instructions)]] page.

FAOSTAT Food Security and Nutrition

2023-09-20T23:46:39Z

Sami.McKinsey:

=== Summary ===
FAOSTAT gives users access to food and agriculture data for over 245 countries and territories. Additionally, it covers all FAO regional groupings from 1961 to the most recent year available. The Suite of Food Security Indicators presents the core set of food security indicators. The choice of the indicators has been informed by expert judgment and the availability of data with sufficient coverage to enable comparisons across regions and over time. Many of these indicators are produced and published elsewhere by FAO and other international organizations. More indicators will be added to as more data will become available. Indicators are classified along the four dimensions of food security -- availability, access, utilization and stability.

The data team uses food security indicators for many series, including those related to hunger/malnutrition such as SeriesMalnDietEnSup, SeriesMalnPop%, SeriesMalnCaloricDistribution, SeriesMalnMinNutrRequired. To pull data, follow the instructions below.

=== FAO Food Security Pulling Steps ===
Step 1. Navigate to [https://www.fao.org/faostat/en/#data/FS FAOSTAT Food Security]site. This site will display a suite of food security indicators.

[[File:FAO IMG 1.jpg|center|thumb|696x696px|FAO Food Security Homepage]]

Step 2. The page displays four options that must be selected to download data: Countries/Regions/Special Groups, Elements, Years, and Items. For all these options, you can also type in a search box.
[[File:FAO IMG 2.jpg|center|thumb|754x754px|FAO Data Selections]]

Step 3. For '''“Countries”''' options, you will generally want to select '''“Select All”''' to capture all country data.
[[File:FAO IMG 3.jpg|center|thumb|688x688px|Country Selection]]

Step 4. For the '''“Elements”''' options you will most likely want to select '''“Value”''', depending on the desired dataset.
[[File:FAO IMG 4.jpg|center|thumb|684x684px|Elements Selection]]

Step 5. For '''“Years”''', you will generally want to select '''“Select All”''' to capture all years of data available.
[[File:FAO IMG 5.jpg|center|thumb|678x678px|Years Selection]]

Step 6. For '''“Items”''', select the desired indicator. You can also type in the search box for the indicator. This example will use '''“Prevalence of undernourishment (percent)”'''.
[[File:FAO IMG 6.jpg|center|thumb|684x684px|Items Selection]]

Step 7. After selecting for these for options, the page will display the completed selections.
[[File:FAO IMG 7.jpg|center|thumb|716x716px|Completed Selections ]]

Step 8. Scroll down below the options. You also have options to select the “Output Type”, “File Type”, and “Thousand Separator in ‘Show Data’”. It’s recommended to select '''“Pivot”''' under '''“Output Type”''' to have easier data cleaning.
[[File:FAO IMG 8.jpg|center|thumb|701x701px|Additional Data Options]]

Step 9. You may select '''“Show Data”''' to look at the data before downloading. This option will populate data at the bottom of the page.

Step 10. Click '''“Download Data”'''. Either an Excel or CSV sheet will download. Now you can format the data to upload it into IFs. To import data into IFs, please follow the instructions found in the [[Importing data (general instructions)|Importing Data (general instructions)]] page.
[[File:FAO IMG 9.jpg|center|thumb|718x718px|Show and Download Data]]

File:FAO IMG 9.jpg

2023-09-20T23:23:13Z

Sami.McKinsey:

Show and Download Data

File:FAO IMG 8.jpg

2023-09-20T23:22:05Z

Sami.McKinsey:

Additional Options

File:FAO IMG 7.jpg

2023-09-20T23:20:24Z

Sami.McKinsey:

Completed Selections

File:FAO IMG 6.jpg

2023-09-20T23:07:56Z

Sami.McKinsey:

FAO Items Selection

File:FAO IMG 5.jpg

2023-09-20T23:07:04Z

Sami.McKinsey:

FAO Years Selection

File:FAO IMG 4.jpg

2023-09-20T23:05:10Z

Sami.McKinsey:

Elements Selection

File:FAO IMG 3.jpg

2023-09-20T23:04:10Z

Sami.McKinsey:

Data Country Selection

File:FAO IMG 2.jpg

2023-09-20T23:02:59Z

Sami.McKinsey:

FAO Data Options

File:FAO IMG 1.jpg

2023-09-20T23:00:33Z

Sami.McKinsey:

FAOSTAT Food Security Homepage