Importing data (general instructions) - Revision history

Sami.McKinsey at 02:19, 25 September 2023

2023-09-25T02:19:00Z

← Older revision		Revision as of 02:19, 25 September 2023
Line 46:		Line 46:
	This feature can be accessed through IFs menu system:		This feature can be accessed through IFs menu system:

	IFs -> MainMenu->Extended Features ~~-> Manage Country Data->Historic Data File Processing~~-> Import Data ~~from XLS File~~.		IFs -> MainMenu->Extended Features -> Import Data (Single Series).
			[[File:Importing IMG 1.jpg\|center\|thumb\|872x872px\|IFs Main Menu]]
	[[File:~~ImportDataFromXLSFile~~.~~png~~\|~~RTENOTITLE~~]]

	This opens up a form shown below:		This opens up a form shown below:
			[[File:Importing IMG 3.jpg\|center\|thumb\|869x869px\|Single Import Page]]

	~~[[File:ImportDataWizard.png\|RTENOTITLE]]~~

	The form is self-explanatory for the most part. You have to start by opening the source Excel file (~~option on the top bar~~)~~ ~~and selecting the spreadsheet which contains source. You then need to choose one from the six options showing the format of the data organization in the source spreadsheet (see the right side of the import screen). The default of these six options (the one at the bottom) works in most cases, however you can compare each option to your data by selecting a data organization option and viewing the layout in the image directly below the final option.		The form is self-explanatory for the most part. You have to start by opening the source Excel file (select "Choose File") and selecting the spreadsheet which contains source. Then click "Refresh Sheets" and choose the desired Excel sheet. You then need to choose one from the six options showing the format of the data organization in the source spreadsheet (see the right side of the import screen). The default of these six options (the one at the bottom) works in most cases, however you can compare each option to your data by selecting a data organization option and viewing the layout in the image directly below the final option.

	The list of steps to follow when importing data can be found in the "Help" menu option, which opens up a dialog box with the basic list.		The list of steps to follow when importing data can be found in the "Help" menu option, which opens up a dialog box with the basic list.
Line 127:		Line 126:

	''Forecast'': click Yes if the series imported is a forecast and not a historical data series		''Forecast'': click Yes if the series imported is a forecast and not a historical data series
			[[File:Importing IMG 4.jpg\|center\|thumb\|912x912px\|Data Dictionary]]
	[[File:~~DataDictWizard~~.~~png~~\|~~RTENOTITLE~~]]



	'''13. Imported data will go to IFs\Data\IFsDataImport.mdb.'''		'''13. Imported data will go to IFs\Data\IFsDataImport.mdb.'''

MeredithMoon at 16:59, 25 September 2017

2017-09-25T16:59:06Z

← Older revision		Revision as of 16:59, 25 September 2017
Line 56:		Line 56:
	The form is self-explanatory for the most part. You have to start by opening the source Excel file (option on the top bar) and selecting the spreadsheet which contains source. You then need to choose one from the six options showing the format of the data organization in the source spreadsheet (see the right side of the import screen). The default of these six options (the one at the bottom) works in most cases, however you can compare each option to your data by selecting a data organization option and viewing the layout in the image directly below the final option.		The form is self-explanatory for the most part. You have to start by opening the source Excel file (option on the top bar) and selecting the spreadsheet which contains source. You then need to choose one from the six options showing the format of the data organization in the source spreadsheet (see the right side of the import screen). The default of these six options (the one at the bottom) works in most cases, however you can compare each option to your data by selecting a data organization option and viewing the layout in the image directly below the final option.

	The list of steps to follow when importing data can be found in the "Help" menu option, which opens up a dialog box with the basic list.		The list of steps to follow when importing data can be found in the "Help" menu option, which opens up a dialog box with the basic list.

	[[File:ImportDataInstructions.png\|RTENOTITLE]]		[[File:ImportDataInstructions.png\|RTENOTITLE]]
Line 114:		Line 114:
	'''12. After accepting imported data, you'll see a form to enter data dictionary information. '''		'''12. After accepting imported data, you'll see a form to enter data dictionary information. '''

	Here are some tips on what to enter in the different fields in the data dict:		Here are some tips on what to enter in the different fields in the data dict that do not auto-populate from the import:

			''Series ''& ''Cohort'': the entry for Series is Yes and for Cohort is No

	''Source Definition'': the exact definition from the data source		''Source Definition'': the exact definition from the data source
Line 121:		Line 123:

	''Name in Source'' & ''Code in Source'': only relevant for batch pulls since there are multiple series in the pull		''Name in Source'' & ''Code in Source'': only relevant for batch pulls since there are multiple series in the pull

			''Notes'': enter your initials as data puller/importer and the data vetter will then enter their initials

			''Forecast'': click Yes if the series imported is a forecast and not a historical data series

	[[File:DataDictWizard.png\|RTENOTITLE]]		[[File:DataDictWizard.png\|RTENOTITLE]]

MeredithMoon at 16:50, 25 September 2017

2017-09-25T16:50:19Z

Show changes

MeredithMoon at 23:08, 21 September 2017

2017-09-21T23:08:33Z

@@ Line 42: / Line 42: @@
 = Automated Import Using ImportXLS: Single Import =
-External data available in Excel format known to IFs can be extracted into Access tables using IFs application's automated ImportXLS (the process works only for Excel files and hence the name) feature that work by importing one time series at a time. If you have subnational models broken out in IFs on your computer, you need to switch back to the full 186 country model. Instructions on how to do this can be found in the&nbsp;[[SubRegionalization_Handbook|SubRegionalization Handbook]].&nbsp;
+External data available in Excel format known to IFs can be extracted into Access tables using IFs application's automated ImportXLS (the process works only for Excel files and hence the name) feature that work by importing one time series at a time. If you have subnational models broken out in IFs on your computer, you need to switch back to the full 186 country model. Instructions on how to do this can be found in the&nbsp;[[SubRegionalization_Handbook|SubRegionalization Handbook]]. This process will take around five minutes, so can be done as frequently as needed.&nbsp;
 This feature can be accessed through IFs menu system:
@@ Line 54: / Line 54: @@
 [[File:ImportDataWizard.png|RTENOTITLE]]
-The form is self-explanatory for the most part. You have to start by opening the source Excel file and selecting the spreadsheet which contains source. You then need to choose one from the six options showing the format of the data organization in the source spreadsheet (see the right side of the import screen). The default of these six options (the one at the bottom) works in most cases. The "Help" menu option in the form opens up a dialog box with a step by step guide as shown in the figure below:
+The form is self-explanatory for the most part. You have to start by opening the source Excel file (option on the top bar)&nbsp;and selecting the spreadsheet which contains source. You then need to choose one from the six options showing the format of the data organization in the source spreadsheet (see the right side of the import screen). The default of these six options (the one at the bottom) works in most cases, however you can compare each option to your data by selecting a data organization option and viewing the layout in the image directly below the final option.
 [[File:ImportDataInstructions.png|RTENOTITLE]]

MeredithMoon at 21:24, 21 September 2017

2017-09-21T21:24:45Z

← Older revision		Revision as of 21:24, 21 September 2017
Line 42:		Line 42:
	= Automated Import Using ImportXLS: Single Import =		= Automated Import Using ImportXLS: Single Import =

	External data available in Excel format known to IFs can be extracted into Access tables using IFs application's automated ImportXLS (the process works only for Excel files and hence the name) feature that work by importing one time series at a time.		External data available in Excel format known to IFs can be extracted into Access tables using IFs application's automated ImportXLS (the process works only for Excel files and hence the name) feature that work by importing one time series at a time. If you have subnational models broken out in IFs on your computer, you need to switch back to the full 186 country model. Instructions on how to do this can be found in the [[SubRegionalization_Handbook\|SubRegionalization Handbook]].

	This feature can be accessed through IFs menu system:		This feature can be accessed through IFs menu system:
Line 116:		Line 116:
	Extracted data will be saved in <u>Data\IFsDataImportBatch.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.		Extracted data will be saved in <u>Data\IFsDataImportBatch.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.
	<div><div id="_com_1"></div></div>		<div><div id="_com_1"></div></div>

	= Automated Import Using ImportXLS: Province Import =		= Automated Import Using ImportXLS: Province Import =

HarrySmythe at 23:59, 4 January 2017

2017-01-04T23:59:03Z

← Older revision		Revision as of 23:59, 4 January 2017
Line 90:		Line 90:
	Before using the batch import process, new variables must be added to the datadict and the source column of the datadict must be formatted for batch import by including one of the qualified series listed above (FAO, IMF, AQU, etc). Additionally, each data series in the external spreadsheet is matched with an IFs data table through a "series code" that is previously collected in the IFs "datadict" table. For WDI, these codes appear as combination of two to four letter code sections separated by periods (e.g., AG.PRD.FOOD.XD). For UNESCO education data, we take series names used by UIS, exactly as they type it, as series codes. For FAO data, Element code and Item code used by FAO are used by IFS (accordingly, there will be two series code  boxes in the batch update form when you are updating FAO data; identify Element code as the first one and Item code as the second one).		Before using the batch import process, new variables must be added to the datadict and the source column of the datadict must be formatted for batch import by including one of the qualified series listed above (FAO, IMF, AQU, etc). Additionally, each data series in the external spreadsheet is matched with an IFs data table through a "series code" that is previously collected in the IFs "datadict" table. For WDI, these codes appear as combination of two to four letter code sections separated by periods (e.g., AG.PRD.FOOD.XD). For UNESCO education data, we take series names used by UIS, exactly as they type it, as series codes. For FAO data, Element code and Item code used by FAO are used by IFS (accordingly, there will be two series code  boxes in the batch update form when you are updating FAO data; identify Element code as the first one and Item code as the second one).

	The decimal places box is used for specifying the decimal places that will be kept in extracted tables. The setting works for those tables which do not have such setting established already (as shown in the grid above).		The decimal places box is used for specifying the decimal places that will be kept in extracted tables. As a general rule, decimals should be truncated after the 4th decimal place. The setting works for those tables which do not have such setting established already (as shown in the grid above).

	[[File:BatchImportWizard.png\|RTENOTITLE]]		[[File:BatchImportWizard.png\|RTENOTITLE]]
Line 116:		Line 116:
	Extracted data will be saved in <u>Data\IFsDataImportBatch.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.		Extracted data will be saved in <u>Data\IFsDataImportBatch.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.
	<div><div id="_com_1"></div></div>		<div><div id="_com_1"></div></div>

	= Automated Import Using ImportXLS: Province Import =		= Automated Import Using ImportXLS: Province Import =

HarrySmythe at 23:58, 4 January 2017

2017-01-04T23:58:38Z

← Older revision		Revision as of 23:58, 4 January 2017
Line 58:		Line 58:
	[[File:ImportDataInstructions.png\|RTENOTITLE]]		[[File:ImportDataInstructions.png\|RTENOTITLE]]

	As mentioned above, Country Concordance lists are used in converting external data to IFs format. "Country Concordance" options are shown in a dropdown list at the bottom of the screen. The prepackaged lists that you see in the dropdown were built over the years from the major international databases like World Bank's WDI, UNESCO, IEA etc. If you do not see a list that you can use you need to either use a manual import described above in section 3.1.1 or leave ImportXLS for now, build and update a country list and then come back to import. Unless a data source has a large number of tables that we will be regularly updating we do not spend the time in building and updating a new country concordance table.		As mentioned above, Country Concordance lists are used in converting external data to IFs format. "Country Concordance" options are shown in a dropdown list at the bottom of the screen. The prepackaged lists that you see in the dropdown were built over the years from the major international databases like World Bank's WDI, UNESCO, IEA etc. If you do not see a list that you can use you need to either use a manual import described above in section 3.1.1 or leave ImportXLS for now, build and update a country list and then come back to import. Unless a data source has a large number of tables that we will be regularly updating we do not spend the time in building and updating a new country concordance table. As a general rule, decimals should be truncated after the 4th decimal place.

	If you would like to update an existing table rather than import a new one, you can select the name of the table from a dropdown box just above the variable name box. Selecting an existing table will automatically bring in dictionary information from the master copy of the data dictionary.		If you would like to update an existing table rather than import a new one, you can select the name of the table from a dropdown box just above the variable name box. Selecting an existing table will automatically bring in dictionary information from the master copy of the data dictionary.

Wikiadmin at 23:25, 28 March 2016

2016-03-28T23:25:32Z

Wikiadmin at 23:24, 28 March 2016

2016-03-28T23:24:42Z

← Older revision		Revision as of 23:24, 28 March 2016
Line 74:		Line 74:
	IFs data tables cover 183 countries now. But we plan to extend IFs to 186 countries by including Seychelles, Kosovo and South Sudan. Just below the “Import Data from XLS File" there is another menu option titled "Import Data from XLS File 186". The form that opens will follow the same procedure as 183 country import, but will create two tables one with 186 countries and one with 183, both in the same location.		IFs data tables cover 183 countries now. But we plan to extend IFs to 186 countries by including Seychelles, Kosovo and South Sudan. Just below the “Import Data from XLS File" there is another menu option titled "Import Data from XLS File 186". The form that opens will follow the same procedure as 183 country import, but will create two tables one with 186 countries and one with 183, both in the same location.

	== Batch Update Using ImportXLSBatch ==		= Batch Update Using ImportXLSBatch =

	Another form has been created that allows import from an access table. This needs to be described, and shown.		Another form has been created that allows import from an access table. This needs to be described, and shown.

Wikiadmin at 23:23, 28 March 2016

2016-03-28T23:23:51Z

← Older revision		Revision as of 23:23, 28 March 2016
Line 40:		Line 40:
	Extracted data, saved in <u>Data\IFsDataImport.Mdb, </u>need to be sent to the person who will do the vetting. Please also pass all source information (Excel spreadsheet data or pdf file/s or the website from which data is taken). Some initial scrutiny of the data before it is sent for vetting is recommended.		Extracted data, saved in <u>Data\IFsDataImport.Mdb, </u>need to be sent to the person who will do the vetting. Please also pass all source information (Excel spreadsheet data or pdf file/s or the website from which data is taken). Some initial scrutiny of the data before it is sent for vetting is recommended.

	== Automated Import Using ImportXLS: Single Import ==		= Automated Import Using ImportXLS: Single Import =

	External data available in Excel format known to IFs can be extracted into Access tables using IFs application's automated ImportXLS (the process works only for Excel files and hence the name) feature that work by importing one time series at a time.		External data available in Excel format known to IFs can be extracted into Access tables using IFs application's automated ImportXLS (the process works only for Excel files and hence the name) feature that work by importing one time series at a time.
Line 66:		Line 66:
	[[File:DataDictWizard.png\|RTENOTITLE]]		[[File:DataDictWizard.png\|RTENOTITLE]]

	=== Passing Extracted Data for Vetting ===		== Passing Extracted Data for Vetting ==

	Extracted data will be saved in <u>Data\IFsDataImport.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.		Extracted data will be saved in <u>Data\IFsDataImport.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.

← Older revision		Revision as of 23:25, 28 March 2016
Line 102:		Line 102:
	Imported tables can be subjected to automated blending and vetting using the new vetting/comparison interface described later in this document.		Imported tables can be subjected to automated blending and vetting using the new vetting/comparison interface described later in this document.

	=== Source Data for Batch Update ===		== Source Data for Batch Update ==

	World Development Indicator (WDI) data is released annually and is usually available as a single Excel file at their website:		World Development Indicator (WDI) data is released annually and is usually available as a single Excel file at their website:
Line 112:		Line 112:
	IMF data is available only by accessing our IMF e-Library account, registered to Mohammod Irfan.		IMF data is available only by accessing our IMF e-Library account, registered to Mohammod Irfan.

	=== Passing Extracted Batch Data for Vetting ===		== Passing Extracted Batch Data for Vetting ==

	Extracted data will be saved in <u>Data\IFsDataImportBatch.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.		Extracted data will be saved in <u>Data\IFsDataImportBatch.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.
	<div><div id="_com_1"></div></div>		<div><div id="_com_1"></div></div>
	== Automated Import Using ImportXLS: Province Import ==		= Automated Import Using ImportXLS: Province Import =

	Importing data for a provincial model can be done with the Province Data Import Feature. IFs does not need to be sub-regionalized to use this feature.		Importing data for a provincial model can be done with the Province Data Import Feature. IFs does not need to be sub-regionalized to use this feature.
Line 126:		Line 126:
	The form that opens up is shown below. The format reflects the single series import process already described. A recognized excel format must be selected from the list on the right. The location of Province names, Years, and Series from your chosen excel data sheet must be designated. The provincial country concordance table must also be selected from the drop down list. The form contains the option to view the excel sheet you are importing, and to view the country concordance tables. Lastly, the variable you are importing needs to be selected from the drop down list. Existing variables will populate the definition field. New variables will need to be named using the Ifs naming convention reviewed above, and defined.		The form that opens up is shown below. The format reflects the single series import process already described. A recognized excel format must be selected from the list on the right. The location of Province names, Years, and Series from your chosen excel data sheet must be designated. The provincial country concordance table must also be selected from the drop down list. The form contains the option to view the excel sheet you are importing, and to view the country concordance tables. Lastly, the variable you are importing needs to be selected from the drop down list. Existing variables will populate the definition field. New variables will need to be named using the Ifs naming convention reviewed above, and defined.

	=== Passing Extracted Data for Vetting ===		== Passing Extracted Data for Vetting ==

	Extracted data will be saved in <u>Data\IFsDataImport.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.		Extracted data will be saved in <u>Data\IFsDataImport.Mdb</u>. Dictionary info is saved in a table named "datadict" inside this same .mdb. You need to send this .mdb and all external source Excel to the person who will do the vetting. Some initial scrutiny of the data before it is sent for vetting is recommended. The new automated vetting feature of IFs (described in the vetting section below) can be useful.