Country Concordances - Revision history

Norah.Shamin: /* Special Cases */

2025-10-03T18:01:54Z

Special Cases

← Older revision		Revision as of 18:01, 3 October 2025
Line 45:		Line 45:

	== Special Cases ==		== Special Cases ==
	Each organization might have special cases due to the nature of their country list. Therefore in the "Notes" section of each data source page (example: [[FAOSTAT Land Use#Notes]]) contains information necessary to complete the data pulling tasks (example: FAOSTAT Land Use needs disaggregation, notes for Ethiopia, and notes for Kosovo and Serbia).		Each organization might have special cases due to the nature of their country list. Therefore in the "Notes"/"Data Notes" section of each data source page (example: [[FAOSTAT Land Use#Notes]]) contains information necessary to complete the data pulling tasks (example: FAOSTAT Land Use needs disaggregation, notes for Ethiopia, and notes for Kosovo and Serbia).

	=== Proxies ===		=== Proxies ===

Norah.Shamin at 18:01, 3 October 2025

2025-10-03T18:01:15Z

← Older revision		Revision as of 18:01, 3 October 2025
Line 1:		Line 1:
	== Summary ==		== Summary ==
	[https://github.com/n1shamin/Country-Concordance Country concordance] is an important aspect of a data technician's job to maintain the International Futures database. Country concordance refers to the differences between the IFs Country list and another organization's country list and merging them to ensure the IFs country list in IFs system. For example, in some organization "Türkiye" is the name displayed, but for IFs we use the name "Turkey". Therefore IFs data technicians need to change the name to "Turkey" for the system to process it.		[https://github.com/n1shamin/Country-Concordance Country concordance] is an important aspect of a data technician's job to maintain the International Futures database. Country concordance refers to the differences between the IFs Country list and another organization's country list and merging them to ensure the IFs country list in IFs system. For example, in some organization "Türkiye" is the name displayed, but for IFs we use the name "Turkey". Therefore IFs data technicians need to change the name to "Turkey" for the system to process it. To do this there are multiple steps that can be taken and they are shown below:

	== GitHub Repositories ==		== GitHub Repositories ==
Line 26:		Line 26:
	=== Process to Create Country Concordance ===		=== Process to Create Country Concordance ===
	Use the app to import your dataset. Once you choose a file, and click upload and select the worksheet if needed:		Use the app to import your dataset. Once you choose a file, and click upload and select the worksheet if needed:
	[[File:Step 1- Upload Excel-CSV File - sheet.png\|none\|thumb\|600x600px]]The page will go to Data Preview. Select the row or column that contains the country names. For this example it is the column "country_name"[[File:Step 2- Data Preview.png\|none\|thumb\|600x600px]]Select Fuzzy Match and at least 80%. [[File:Step 3- Name-matching.png\|none\|thumb\|600x600px]]Once you click match names you will be led to this page: [[File:Review Name Matches (Page 1 of 1).png\|none\|thumb\|600x600px]]In the bottom of the page there are two important options; "Save Mapping" and "Submit & Download File."		[[File:Step 1- Upload Excel-CSV File - sheet.png\|none\|thumb\|600x600px]]The page will go to Data Preview. Select the row or column that contains the country names. For this example it is the column "country_name"[[File:Step 2- Data Preview.png\|none\|thumb\|600x600px]]Select Fuzzy Match and at least 80%. You can go higher but high percentages mean that names can be missed if they fall short. [[File:Step 3- Name-matching.png\|none\|thumb\|600x600px]]Once you click match names you will be led to this page: [[File:Review Name Matches (Page 1 of 1).png\|none\|thumb\|600x600px]]In the bottom of the page there are two important options; "Save Mapping" and "Submit & Download File."

	"Submit & Download File" changes the name in the original file to the IFs country names so the data technician can use the IFs interface to import into a db file.		"Submit & Download File" changes the name in the original file to the IFs country names so the data technician can use the IFs interface to import into a db file. '''Before doing this make sure that all the country names have been changed correctly'''. Sometimes Turkey can be missed due to a less percentage in FuzzyMatch.

	"Save Mapping" will download a file that shows the organization's country list compared to IFs country list.		"Save Mapping" will download a file that shows the organization's country list compared to IFs country list.

Norah.Shamin: /* Proxies */

2025-10-01T17:41:48Z

Proxies

← Older revision		Revision as of 17:41, 1 October 2025
Line 23:		Line 23:
	\| colspan="2" \|and so on...		\| colspan="2" \|and so on...
	\|}		\|}
	==== Process to Create Country Concordance ====
			=== Process to Create Country Concordance ===
	Use the app to import your dataset. Once you choose a file, and click upload and select the worksheet if needed:		Use the app to import your dataset. Once you choose a file, and click upload and select the worksheet if needed:
	[[File:Step 1- Upload Excel-CSV File - sheet.png\|none\|thumb\|600x600px]]The page will go to Data Preview. Select the row or column that contains the country names. For this example it is the column "country_name"[[File:Step 2- Data Preview.png\|none\|thumb\|600x600px]]Select Fuzzy Match and at least 80%. [[File:Step 3- Name-matching.png\|none\|thumb\|600x600px]]Once you click match names you will be led to this page: [[File:Review Name Matches (Page 1 of 1).png\|none\|thumb\|600x600px]]In the bottom of the page there are two important options; "Save Mapping" and "Submit & Download File."		[[File:Step 1- Upload Excel-CSV File - sheet.png\|none\|thumb\|600x600px]]The page will go to Data Preview. Select the row or column that contains the country names. For this example it is the column "country_name"[[File:Step 2- Data Preview.png\|none\|thumb\|600x600px]]Select Fuzzy Match and at least 80%. [[File:Step 3- Name-matching.png\|none\|thumb\|600x600px]]Once you click match names you will be led to this page: [[File:Review Name Matches (Page 1 of 1).png\|none\|thumb\|600x600px]]In the bottom of the page there are two important options; "Save Mapping" and "Submit & Download File."
Line 43:		Line 44:
	dt = dt.rename(columns={'IFs Country':'Country'}) #this must be changed before importing as the column name in IFs is Country		dt = dt.rename(columns={'IFs Country':'Country'}) #this must be changed before importing as the column name in IFs is Country

	== Proxies ==		== Special Cases ==
			Each organization might have special cases due to the nature of their country list. Therefore in the "Notes" section of each data source page (example: [[FAOSTAT Land Use#Notes]]) contains information necessary to complete the data pulling tasks (example: FAOSTAT Land Use needs disaggregation, notes for Ethiopia, and notes for Kosovo and Serbia).

			=== Proxies ===
	Proxies are used when a certain country does not have data points and estimates are needed. Proxies should be used on a case by case basis and for certain series. Proxies essentially use a similar country to the country being approximated (in terms of population, GDP, etc.) and calculates the scale of what the approximated countries value should be.		Proxies are used when a certain country does not have data points and estimates are needed. Proxies should be used on a case by case basis and for certain series. Proxies essentially use a similar country to the country being approximated (in terms of population, GDP, etc.) and calculates the scale of what the approximated countries value should be.

Line 50:		Line 54:
	* Kosovo’s number of deaths = Albania’s number of deaths * (Kosovo’s Population / Albania’s Population)		* Kosovo’s number of deaths = Albania’s number of deaths * (Kosovo’s Population / Albania’s Population)

	== Disaggregation ==		=== Disaggregation ===
	A lot of organizations have values for dissolved states such as USSR, Yugoslavia, Czechoslovakia, and more. This then leads to gaps in values for the newly formed such as Serbia, Slovakia, Czechia, etc. Therefore, disaggregation is a great tool to fill in these gaps. Below is the disaggregation steps for common groups:		A lot of organizations have values for dissolved states such as USSR, Yugoslavia, Czechoslovakia, and more. This then leads to gaps in values for the newly formed such as Serbia, Slovakia, Czechia, etc. Therefore, disaggregation is a great tool to fill in these gaps. Below is the disaggregation steps for common groups:

Norah.Shamin at 17:31, 1 October 2025

2025-10-01T17:31:28Z

← Older revision		Revision as of 17:31, 1 October 2025
Line 1:		Line 1:
	=== Summary ===		== Summary ==
	[https://github.com/n1shamin/Country-Concordance Country concordance] is an important aspect of a data technician's job to maintain the International Futures database. Country concordance refers to the differences between the IFs Country list and another organization's country list and merging them to ensure the IFs country list in IFs system. For example, in some organization "Türkiye" is the name displayed, but for IFs we use the name "Turkey". Therefore IFs data technicians need to change the name to "Turkey" for the system to process it.		[https://github.com/n1shamin/Country-Concordance Country concordance] is an important aspect of a data technician's job to maintain the International Futures database. Country concordance refers to the differences between the IFs Country list and another organization's country list and merging them to ensure the IFs country list in IFs system. For example, in some organization "Türkiye" is the name displayed, but for IFs we use the name "Turkey". Therefore IFs data technicians need to change the name to "Turkey" for the system to process it.

	~~====~~== GitHub Repositories ~~====~~==		== GitHub Repositories ==
	There are two repositories essential to this process; [https://github.com/quciet/datagator_flask_mongo a mini app that can handle concordance for files less than 15mb] and [https://github.com/n1shamin/Country-Concordance a centralized GitHub repository for all country concordance tables].		There are two repositories essential to this process; [https://github.com/quciet/datagator_flask_mongo a mini app that can handle concordance for files less than 15mb] and [https://github.com/n1shamin/Country-Concordance a centralized GitHub repository for all country concordance tables].

Line 42:		Line 42:

	dt = dt.rename(columns={'IFs Country':'Country'}) #this must be changed before importing as the column name in IFs is Country		dt = dt.rename(columns={'IFs Country':'Country'}) #this must be changed before importing as the column name in IFs is Country
	==== Proxies ====
			== Proxies ==
	Proxies are used when a certain country does not have data points and estimates are needed. Proxies should be used on a case by case basis and for certain series. Proxies essentially use a similar country to the country being approximated (in terms of population, GDP, etc.) and calculates the scale of what the approximated countries value should be.		Proxies are used when a certain country does not have data points and estimates are needed. Proxies should be used on a case by case basis and for certain series. Proxies essentially use a similar country to the country being approximated (in terms of population, GDP, etc.) and calculates the scale of what the approximated countries value should be.

Line 49:		Line 50:
	* Kosovo’s number of deaths = Albania’s number of deaths * (Kosovo’s Population / Albania’s Population)		* Kosovo’s number of deaths = Albania’s number of deaths * (Kosovo’s Population / Albania’s Population)

	==== Disaggregation ====		== Disaggregation ==
	A lot of organizations have values for dissolved states such as USSR, Yugoslavia, Czechoslovakia, and more. This then leads to gaps in values for the newly formed such as Serbia, Slovakia, Czechia, etc. Therefore, disaggregation is a great tool to fill in these gaps. Below is the disaggregation steps for common groups:		A lot of organizations have values for dissolved states such as USSR, Yugoslavia, Czechoslovakia, and more. This then leads to gaps in values for the newly formed such as Serbia, Slovakia, Czechia, etc. Therefore, disaggregation is a great tool to fill in these gaps. Below is the disaggregation steps for common groups:

Norah.Shamin at 17:28, 1 October 2025

2025-10-01T17:28:24Z

← Older revision		Revision as of 17:28, 1 October 2025
Line 24:		Line 24:
	\|}		\|}
	==== Process to Create Country Concordance ====		==== Process to Create Country Concordance ====
	Use the app to import your dataset. Once you choose a file, and click upload		Use the app to import your dataset. Once you choose a file, and click upload and select the worksheet if needed:
	[[File:Step 1- Upload Excel-CSV File - sheet.png\|none\|thumb\|600x600px]]		[[File:Step 1- Upload Excel-CSV File - sheet.png\|none\|thumb\|600x600px]]The page will go to Data Preview. Select the row or column that contains the country names. For this example it is the column "country_name"[[File:Step 2- Data Preview.png\|none\|thumb\|600x600px]]Select Fuzzy Match and at least 80%. [[File:Step 3- Name-matching.png\|none\|thumb\|600x600px]]Once you click match names you will be led to this page: [[File:Review Name Matches (Page 1 of 1).png\|none\|thumb\|600x600px]]In the bottom of the page there are two important options; "Save Mapping" and "Submit & Download File."
	[[File:Step 2- Data Preview.png\|none\|thumb\|600x600px]]
	[[File:Step 3- Name-matching.png\|none\|thumb\|600x600px]]Once you click match names you will be led to this page: [[File:Review Name Matches (Page 1 of 1).png\|none\|thumb\|600x600px]]In the bottom of the page there are two important options; "Save Mapping" and "Submit & Download File."

	"Submit & Download File" changes the name in the original file to the IFs country names so the data technician can use the IFs interface to import into a db file.		"Submit & Download File" changes the name in the original file to the IFs country names so the data technician can use the IFs interface to import into a db file.

Norah.Shamin at 17:23, 1 October 2025

2025-10-01T17:23:41Z

@@ Line 23: / Line 23: @@
 | colspan="2" |and so on...
 |}
-Using pandas and Python users can use the following snippet of code to merge the two before inputting into a db file:
+==== Process to Create Country Concordance ====
 country = pd.read_excel('ITU Country.xlsx') #get this excel from the repo listed above
@@ Line 32: / Line 44: @@
 dt = dt.rename(columns={'IFs Country':'Country'}) #this must be changed before importing as the column name in IFs is Country
 ==== Proxies ====
 Proxies are used when a certain country does not have data points and estimates are needed. Proxies should be used on a case by case basis and for certain series. Proxies essentially use a similar country to the country being approximated (in terms of population, GDP, etc.) and calculates the scale of what the approximated countries value should be.

Norah.Shamin at 16:06, 1 October 2025

2025-10-01T16:06:24Z

← Older revision		Revision as of 16:06, 1 October 2025
Line 34:		Line 34:

	==== Process to Create Country Concordance ====		==== Process to Create Country Concordance ====
	Use app to import your dataset,		Use the app to import your dataset:
			[[File:Step 1- Upload Excel-CSV File - sheet.png\|none\|thumb\|600x600px]]
			[[File:Step 2- Data Preview.png\|none\|thumb\|600x600px]]
			[[File:Step 3- Name-matching.png\|none\|thumb\|600x600px]]
			[[File:Review Name Matches (Page 1 of 1).png\|none\|thumb\|600x600px]]

	==== Proxies ====		==== Proxies ====

Norah.Shamin at 01:08, 28 September 2025

2025-09-28T01:08:13Z

← Older revision		Revision as of 01:08, 28 September 2025
Line 32:		Line 32:

	dt = dt.rename(columns={'IFs Country':'Country'}) #this must be changed before importing as the column name in IFs is Country		dt = dt.rename(columns={'IFs Country':'Country'}) #this must be changed before importing as the column name in IFs is Country

			==== Process to Create Country Concordance ====
			Use app to import your dataset,

	==== Proxies ====		==== Proxies ====

Norah.Shamin at 00:30, 28 September 2025

2025-09-28T00:30:13Z

← Older revision		Revision as of 00:30, 28 September 2025
Line 57:		Line 57:
	# Serbia and Montenegro add up the values for 2006 (year it disbanded); have totals for 1992-2005; have the values for 2 states for 2006; extrapolate data for 2 states 1992-2005		# Serbia and Montenegro add up the values for 2006 (year it disbanded); have totals for 1992-2005; have the values for 2 states for 2006; extrapolate data for 2 states 1992-2005
	An example of this disaggregation can be found in [[FAOSTAT Land Use]].		An example of this disaggregation can be found in [[FAOSTAT Land Use]].

	~~== Organizations ==~~

	~~=== [https://github.com/shangkexin/The_Federal_Institute_for_Geosciences_and_Natural_Resources_BGR/blob/main/Country.xlsx BGR] ===~~
	~~In BGR there are German and English versions and usually the latest are in German. Below are the differing versions. As usual the link leads to a full country list.~~
	~~=== [https://github.com/shangkexin/CIA_World_Factbook_Ethnic_groups/blob/main/Country.xlsx CIA] ===~~
	~~'''This country concordance sheet is for the ethnic group page.''' The following shows what countries do not have values listed in the page and the differing country names.~~
	~~=== [https://github.com/n1shamin/Country-Concordance/blob/main/FAO%20Country.xlsx Food Agricultural Organization (FAO)] ===~~
	~~FAO does not have values for Kosovo.~~

	~~=== [https://github.com/n1shamin/Country-Concordance/blob/main/GCP%20Country.xlsx Global Carbon Project GCP] ===~~
	~~Puerto Rico and Sahrawi Arab Democratic Republic are not present in dataset.~~

	~~=== [https://github.com/n1shamin/Country-Concordance/blob/main/GWNO%20Country.xlsx GNWO Codes]: Used in [[Historical GDP/GDPPC/Population\|Historical GDP]] ===~~
	~~Numerical values are used instead of IDs and or names. Click the links to head to the GitHub repo or wiki.~~

	~~=== [https://github.com/n1shamin/Country-Concordance/blob/main/CountryIHME.xlsx Institute for Health Metrics and Evaluation (IHME)] ===~~
	~~There are different versions of country lists used by the IHME. For example interpersonal violence has a different country list than others.~~
	~~=== [https://github.com/n1shamin/Country-Concordance/blob/main/ILO%20Country.xlsx International Labor Organization] ===~~
	~~Grenada, Kiribati, Kosovo, Micronesia, and Seychelles are not present in ILO.~~

	~~=== International Monetary Funds (IMF) ===~~

	~~====== IMF World Economics Outlook ======~~
	~~Cuba, Korea and Dem. People's Republic do not exist in this database.~~

	~~====== [https://github.com/n1shamin/Country-Concordance/blob/main/IMFGFS%20Country.xlsx IMF Government Financial Statistics] ======~~
	~~GFS is limited and many countries are missing. View the GitHub repo.~~

	~~=== [https://github.com/n1shamin/Country-Concordance/blob/main/ITU%20Country.xlsx International Telecommunication Union] ===~~
	~~Korea, Dem. People's Republic, Kosovo, and Sahrawi Arab Democratic Republic are not present in the database.~~

	~~=== [https://github.com/n1shamin/Country-Concordance/blob/main/JMP%20WHO%20UNICEF%20Country.xlsx Joint Monitoring Programme WHO UNICEF] ===~~
	~~Kosovo, Sahrawi Arab Democratic Republic, and Taiwan are not present in dataset.~~

	~~=== [https://github.com/shangkexin/The_Organisation_for_Economic_Cooperation_and_Development_OECD/blob/main/Country.xlsx OECD] ===~~
	In sheet in "OECD" in excel. There are many missing countries; there are also sheets specific to the table (ie Donor) that might have more countries not included in the base country tab. '''Make sure to check before pulling'''.
	~~=== [https://github.com/n1shamin/Country-Concordance/blob/main/Transparency%20International%20Country.xlsx Transparency International] ===~~
	~~Belize, Kiribati, Micronesia, Palestine, Sahrawi Arab Democratic Republic, Samoa, and Tonga are not present in the dataset.~~

	~~=== [https://github.com/n1shamin/Country-Concordance/blob/main/UIS%20UNESCO%20Country.xlsx UIS UNESCO] ===~~
	~~Kosovo and Taiwan are not present in this dataset. This dataset is also present in IDs and not names.~~

	~~=== [https://github.com/n1shamin/Country-Concordance/blob/main/UNDP%20Country.xlsx United Nation Development Programme (UNDP)] ===~~
	~~=== World Bank ===~~

	~~====== [https://github.com/shangkexin/World_Development_Indicators_WDI/blob/main/WDI_Country.xlsx World Development Indicators] ======~~
	~~Sahrawi Arab Democratic Republic is not present in this dataset.~~

	~~For some World Bank indicators, Taiwan, China is present but in WDI is not.~~

	~~=== World Health Organization (WHO) ===~~

	~~====== [https://github.com/n1shamin/Country-Concordance/blob/main/Global%20Health%20Observatory%20(GHO)%2C%20WHO.csv Global Health Observatory (GHO)] ======~~
	~~Hong Kong, Kosovo, Palestine, Sahrawi Arab Democratic Republic, and Taiwan are not present in database.~~

Norah.Shamin at 00:24, 28 September 2025

2025-09-28T00:24:10Z

@@ Line 1: / Line 1: @@
 === Summary ===
-[https://github.com/n1shamin/Country-Concordance Country concordance] is an important aspect of a data technician's job to maintain the International Futures database. Country concordance refers to the differences between the IFs Country list and another organization's country list and merging them to ensure the IFs country list in IFs system. For example, in some organization "Türkiye" is the name displayed, but for IFs we use the name "Turkey". Therefore IFs data technicians need to change the name to "Turkey" for the system to process it. '''Below lists the organizations IFs draw their data from and the common changes needed as well as a link to a GitHub repository that contains the corresponding country list.'''
+[https://github.com/n1shamin/Country-Concordance Country concordance] is an important aspect of a data technician's job to maintain the International Futures database. Country concordance refers to the differences between the IFs Country list and another organization's country list and merging them to ensure the IFs country list in IFs system. For example, in some organization "Türkiye" is the name displayed, but for IFs we use the name "Turkey". Therefore IFs data technicians need to change the name to "Turkey" for the system to process it.
 ==== Proxies ====