Vetting data - Revision history

Sami.McKinsey at 19:11, 25 September 2023

2023-09-25T19:11:51Z

← Older revision		Revision as of 19:11, 25 September 2023
Line 51:		Line 51:
	[[File:Vetting IMG 1.png\|center\|thumb\|877x877px\|IFs Main Page]]		[[File:Vetting IMG 1.png\|center\|thumb\|877x877px\|IFs Main Page]]

	The Compare Imported Table With Existing Table screen is then opened. Currently, the .NET version of IFs only compares tables that have been imported previously, as opposed to being able to open~~/compared~~ Access files. Make sure you [[Importing data (general instructions)\|import data]] into IFs before vetting. Select ~~which~~ "New Imported Series" you would like to vet.		The Compare Imported Table With Existing Table screen is then opened. Currently, the .NET version of IFs only compares tables that have been imported previously, as opposed to being able to open Access files. Make sure you [[Importing data (general instructions)\|import data]] into IFs before vetting. Select a series from "New Imported Series" you would like to vet.
	[[File:Vetting IMG 2.jpg\|center\|thumb\|863x863px\|Compare Imported Table with Existing Table]]		[[File:Vetting IMG 2.jpg\|center\|thumb\|863x863px\|Compare Imported Table with Existing Table]]
	This will then populate the grids below with the new data (the data you are importing) and the old, historical data that is in IFs. The old data is in the first grid and the new data is in the second grid, as shown below.		This will then populate the grids below with the new data (the data you are importing) and the old, historical data that is in IFs. The old data is in the first grid and the new data is in the second grid, as shown below.

Sami.McKinsey: Updated pictures to .NET version

2023-09-25T19:06:09Z

Updated pictures to .NET version

← Older revision		Revision as of 19:06, 25 September 2023
Line 51:		Line 51:
	[[File:Vetting IMG 1.png\|center\|thumb\|877x877px\|IFs Main Page]]		[[File:Vetting IMG 1.png\|center\|thumb\|877x877px\|IFs Main Page]]

	The Compare Imported Table With Existing Table screen is then opened.		The Compare Imported Table With Existing Table screen is then opened. Currently, the .NET version of IFs only compares tables that have been imported previously, as opposed to being able to open/compared Access files. Make sure you [[Importing data (general instructions)\|import data]] into IFs before vetting. Select which "New Imported Series" you would like to vet.
			[[File:Vetting IMG 2.jpg\|center\|thumb\|863x863px\|Compare Imported Table with Existing Table]]
	[[~~File:2019-07-25~~ (2)~~.png~~\|~~900x500px~~]]		This will then populate the grids below with the new data (the data you are importing) and the old, historical data that is in IFs. The old data is in the first grid and the new data is in the second grid, as shown below.
			[[File:Vetting IMG 3.jpg\|center\|thumb\|861x861px\|Vetting Data Page]]
	~~Here, go to '''File>Open Database with~~ Imported ~~Tables, '''and open the Access file with the data series that~~ you ~~want~~ to ~~import into IFs. This Access file contains data series tables and a dictionary table with a list of new and/or updated data variables~~. ~~This will then populate the top grid with every series within that Access file.~~

	[[File:~~2019-07-25 (3)~~.~~png~~\|~~900x500px~~]]

	~~Next, double click on one of the series (rows) in the top grid to display the data.~~ This will then populate the new data (the data you are importing) and the old, historical data that is in IFs. The old data is in the first grid and the new data is in the second grid, as shown ~~above~~.

	Here you have a lot of different options to compare the new data with the old. The vetting tool will automatically mark any zeros in the table. You will want to double check to see if, in fact, these are suppose to be zeros or if they are just null values. If they are just null values, you can click the button, '''Delete Zeros.'''		Here you have a lot of different options to compare the new data with the old. The vetting tool will automatically mark any zeros in the table. You will want to double check to see if, in fact, these are suppose to be zeros or if they are just null values. If they are just null values, you can click the button, '''Delete Zeros.'''

	Next, you need to click the button, '''Mark Differences between New and Old Data for same country-year. '''This will allow you to easily observe any differences, as the vetting tool will highlight them. You can also change the threshold that it will mark. The default is that it will mark any more than 10% difference. As you can see in the screenshot below, the values for Afghanistan in 2014 and 2015 are marked. You will want to go through all of the countries in each series you are importing to look at these marked differences and write down any that are very large.		Next, you need to click the button, '''Mark Differences between New and Old Data for same country-year. '''This will allow you to easily observe any differences, as the vetting tool will highlight them. You can also change the threshold that it will mark. The default is that it will mark any more than 10% difference. As you can see in the screenshot below, the values for Afghanistan in 2014 and 2015 are marked. You will want to go through all of the countries in each series you are importing to look at these marked differences and write down any that are very large.

	~~[[File:2019-07-25 (4).png\|900x500px]]~~

	You can also click the button '''Mark''' '''Year to Year Jumps in New Data, '''which is helpful to observe any large changes year-to-year.		You can also click the button '''Mark''' '''Year to Year Jumps in New Data, '''which is helpful to observe any large changes year-to-year.
			[[File:Vetting IMG 4.jpg\|center\|thumb\|851x851px\|Vetting Methods]]

	'''Blending'''		'''Blending'''
Line 76:		Line 70:

	You will also want to do the same process for the '''Blend Columns (Years) '''button, if there are any missing years. This ensures that we retain the historical values.		You will also want to do the same process for the '''Blend Columns (Years) '''button, if there are any missing years. This ensures that we retain the historical values.
			[[File:Vetting IMG 5.jpg\|center\|thumb\|848x848px\|Blending Data in IFs]]
	[[File:~~2019-07-25 (~~5).~~png~~\|~~900x500px~~]]

	~~ ~~

Sami.McKinsey at 02:47, 25 September 2023

2023-09-25T02:47:00Z

← Older revision		Revision as of 02:47, 25 September 2023
Line 48:		Line 48:
	IFs has a feature to do some basic initial vetting and is a tool that should be used in every vetting process. The vetting tool can be found by the following:		IFs has a feature to do some basic initial vetting and is a tool that should be used in every vetting process. The vetting tool can be found by the following:

	'''IFs -> MainMenu->Extended Features ~~-> Manage Country Data->Historic Data File Processing~~-> Vet Imported Data'''.		'''IFs -> MainMenu->Extended Features -> Vet Imported Data'''.
			[[File:Vetting IMG 1.png\|center\|thumb\|877x877px\|IFs Main Page]]
	~~ ~~[[File:~~2019-07-25 (~~1).png\|~~900x500px~~]]



	The Compare Imported Table With Existing Table screen is then opened.		The Compare Imported Table With Existing Table screen is then opened.

JakeDubbert at 16:51, 25 July 2019

2019-07-25T16:51:33Z

← Older revision		Revision as of 16:51, 25 July 2019
Line 50:		Line 50:
	'''IFs -> MainMenu->Extended Features -> Manage Country Data->Historic Data File Processing-> Vet Imported Data'''.		'''IFs -> MainMenu->Extended Features -> Manage Country Data->Historic Data File Processing-> Vet Imported Data'''.

	[[File:2019-07-25 (1).png]]		[[File:2019-07-25 (1).png\|900x500px]]


Line 56:		Line 56:
	The Compare Imported Table With Existing Table screen is then opened.		The Compare Imported Table With Existing Table screen is then opened.

	[[File:\|~~624x351px\|A screenshot of a social media post~~		[[File:2019-07-25 (2).png\|900x500px]]

	~~Description automatically generated~~]]

	Here, go to '''File>Open Database with Imported Tables, '''and open the Access file with the data series that you want to import into IFs. This Access file contains data series tables and a dictionary table with a list of new and/or updated data variables. This will then populate the top grid with every series within that Access file.		Here, go to '''File>Open Database with Imported Tables, '''and open the Access file with the data series that you want to import into IFs. This Access file contains data series tables and a dictionary table with a list of new and/or updated data variables. This will then populate the top grid with every series within that Access file.

	[[File:\|~~624x351px\|A screenshot of a cell phone~~		[[File:2019-07-25 (3).png\|900x500px]]

	~~Description automatically generated~~]]

	Next, double click on one of the series (rows) in the top grid to display the data. This will then populate the new data (the data you are importing) and the old, historical data that is in IFs. The old data is in the first grid and the new data is in the second grid, as shown above.		Next, double click on one of the series (rows) in the top grid to display the data. This will then populate the new data (the data you are importing) and the old, historical data that is in IFs. The old data is in the first grid and the new data is in the second grid, as shown above.
Line 72:		Line 68:
	Next, you need to click the button, '''Mark Differences between New and Old Data for same country-year. '''This will allow you to easily observe any differences, as the vetting tool will highlight them. You can also change the threshold that it will mark. The default is that it will mark any more than 10% difference. As you can see in the screenshot below, the values for Afghanistan in 2014 and 2015 are marked. You will want to go through all of the countries in each series you are importing to look at these marked differences and write down any that are very large.		Next, you need to click the button, '''Mark Differences between New and Old Data for same country-year. '''This will allow you to easily observe any differences, as the vetting tool will highlight them. You can also change the threshold that it will mark. The default is that it will mark any more than 10% difference. As you can see in the screenshot below, the values for Afghanistan in 2014 and 2015 are marked. You will want to go through all of the countries in each series you are importing to look at these marked differences and write down any that are very large.

	[[File:\|~~624x351px\|A screenshot of a social media post~~		[[File:2019-07-25 (4).png\|900x500px]]

	~~Description automatically generated~~]]

	You can also click the button '''Mark''' '''Year to Year Jumps in New Data, '''which is helpful to observe any large changes year-to-year.		You can also click the button '''Mark''' '''Year to Year Jumps in New Data, '''which is helpful to observe any large changes year-to-year.
Line 86:		Line 80:
	You will also want to do the same process for the '''Blend Columns (Years) '''button, if there are any missing years. This ensures that we retain the historical values.		You will also want to do the same process for the '''Blend Columns (Years) '''button, if there are any missing years. This ensures that we retain the historical values.

	[[File:\|~~624x351px\|A screenshot of a social media post~~		[[File:2019-07-25 (5).png\|900x500px]]

	~~Description automatically generated~~]]

JakeDubbert at 16:46, 25 July 2019

2019-07-25T16:46:25Z

← Older revision		Revision as of 16:46, 25 July 2019
Line 50:		Line 50:
	'''IFs -> MainMenu->Extended Features -> Manage Country Data->Historic Data File Processing-> Vet Imported Data'''.		'''IFs -> MainMenu->Extended Features -> Manage Country Data->Historic Data File Processing-> Vet Imported Data'''.

			[[File:2019-07-25 (1).png]]

JakeDubbert at 16:44, 25 July 2019

2019-07-25T16:44:43Z

Show changes

EllieBremer at 16:55, 5 September 2017

2017-09-05T16:55:17Z

← Older revision		Revision as of 16:55, 5 September 2017
Line 1:		Line 1:
	Vetting is the process of checking that the data that we import to IFs are of good quality. Even when we import the data through an automated application there could be errors or missing data in the imported series.		Vetting is the process of checking that the data that we import to IFs are of good quality. Even when we import the data through an automated application there could be errors or missing data in the imported series.

	One of the major reason an entire country (or a number of countries) could be missed is the mismatch in country concordance table. Make sure the IFs country concordance list (i.e., the relevant column in the '''Country '''Translation table in IFs.mdb) is correct and up to date. Make sure none of the year columns are missed. Sometime the year column is missed because the data is formatted as text and not as numbers. One more general issue is around the units. Make sure you perform any unit conversion required before you import the data. ~~Cmparing~~ the new data with the current IFs series will give you an idea on any unit conversion that might be needed.		One of the major reason an entire country (or a number of countries) could be missed is the mismatch in country concordance table. Make sure the IFs country concordance list (i.e., the relevant column in the '''Country '''Translation table in IFs.mdb) is correct and up to date. Make sure none of the year columns are missed. Sometime the year column is missed because the data is formatted as text and not as numbers. One more general issue is around the units. Make sure you perform any unit conversion required before you import the data. Comparing the new data with the current IFs series will give you an idea on any unit conversion that might be needed.

	It is not recommended that you vet every single country. You can ensure data quality by following certain procedures:		It is not recommended that you vet every single country. You can ensure data quality by following certain procedures:

AlannaMarkle at 22:52, 7 August 2017

2017-08-07T22:52:15Z

← Older revision		Revision as of 22:52, 7 August 2017
Line 3:		Line 3:
	One of the major reason an entire country (or a number of countries) could be missed is the mismatch in country concordance table. Make sure the IFs country concordance list (i.e., the relevant column in the '''Country '''Translation table in IFs.mdb) is correct and up to date. Make sure none of the year columns are missed. Sometime the year column is missed because the data is formatted as text and not as numbers. One more general issue is around the units. Make sure you perform any unit conversion required before you import the data. Cmparing the new data with the current IFs series will give you an idea on any unit conversion that might be needed.		One of the major reason an entire country (or a number of countries) could be missed is the mismatch in country concordance table. Make sure the IFs country concordance list (i.e., the relevant column in the '''Country '''Translation table in IFs.mdb) is correct and up to date. Make sure none of the year columns are missed. Sometime the year column is missed because the data is formatted as text and not as numbers. One more general issue is around the units. Make sure you perform any unit conversion required before you import the data. Cmparing the new data with the current IFs series will give you an idea on any unit conversion that might be needed.

	It is not recommended that you vet every single country. can ensure data quality by following certain procedures:		It is not recommended that you vet every single country. You can ensure data quality by following certain procedures:

	#For an updated table, open the existing and updated table side by side. Most socio-economic data do not change very fast and we usually update data on a regular basis. If you see big difference on the most recent field you should be concerned. Sort the "MostRecent" column ascending and descending and compare the countries that are highest and lowest in the two tables.		#For an updated table, open the existing and updated table side by side. Most socio-economic data do not change very fast and we usually update data on a regular basis. If you see big difference on the most recent field you should be concerned. Sort the "MostRecent" column ascending and descending and compare the countries that are highest and lowest in the two tables.
Line 32:		Line 32:
	Once you double click on any of the rows in the top grid, data for the new and the existing table (for updates) will show up in the middle grids. Any zero or negative will be marked red. You can use buttons below the grids to clear out zeroes and/or negatives from the table containing extracted data.		Once you double click on any of the rows in the top grid, data for the new and the existing table (for updates) will show up in the middle grids. Any zero or negative will be marked red. You can use buttons below the grids to clear out zeroes and/or negatives from the table containing extracted data.

	[[File:VetDataCompareWizard.png]]		[[File:VetDataCompareWizard.png\|RTENOTITLE]]

Wikiadmin at 21:46, 31 March 2016

2016-03-31T21:46:31Z

← Older revision		Revision as of 21:46, 31 March 2016
Line 31:		Line 31:

	Once you double click on any of the rows in the top grid, data for the new and the existing table (for updates) will show up in the middle grids. Any zero or negative will be marked red. You can use buttons below the grids to clear out zeroes and/or negatives from the table containing extracted data.		Once you double click on any of the rows in the top grid, data for the new and the existing table (for updates) will show up in the middle grids. Any zero or negative will be marked red. You can use buttons below the grids to clear out zeroes and/or negatives from the table containing extracted data.

			[[File:VetDataCompareWizard.png]]

Wikiadmin at 21:45, 31 March 2016

2016-03-31T21:45:23Z

← Older revision		Revision as of 21:45, 31 March 2016
Line 32:		Line 32:
	Once you double click on any of the rows in the top grid, data for the new and the existing table (for updates) will show up in the middle grids. Any zero or negative will be marked red. You can use buttons below the grids to clear out zeroes and/or negatives from the table containing extracted data.		Once you double click on any of the rows in the top grid, data for the new and the existing table (for updates) will show up in the middle grids. Any zero or negative will be marked red. You can use buttons below the grids to clear out zeroes and/or negatives from the table containing extracted data.

	~~[[File:\|624x474px]]~~

	The button titled Mark Transients marks any spike found across columns (for the same country in the same table). The marking algorithm uses a default threshold of 10% (up or down) that can be changed by using the Marking Threshold box.		The button titled Mark Transients marks any spike found across columns (for the same country in the same table). The marking algorithm uses a default threshold of 10% (up or down) that can be changed by using the Marking Threshold box.