August 2016 model validation and verification (version 7.23): Difference between revisions

From Pardee Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
This page contains an overview of the model validating and verification of IFs version 7.23 in August 2016. For more detailed analysis of the results, please refer to the excel spreadsheet available here [[Media:ModelVettingNotesV723.xlsx|here]].
= Population module =
= Population module =


Line 11: Line 13:
'''Comparison to other models'''
'''Comparison to other models'''


IFs forecasts global population in 2050 to be 9,542,324,000. The World Bank’s Population Estimates and Projections forecasts global population to be 9,714,894,000 in 2050.[[#_ftn1|[1]]] The United Nations Population Division (UNPD) forecasts global population to be 9,725,148,000 in 2050.[[#_ftn2|[2]]]
IFs forecasts global population in 2050 to be 9,542,324,000. The World Bank’s Population Estimates and Projections forecasts global population to be 9,714,894,000 in 2050.[[#_ftn1|[1]]] The United Nations Population Division (UNPD) forecasts global population to be 9,725,148,000 in 2050.<sup>[2]</sup>


'''Degenerative test'''
'''Degenerative test'''
Line 29: Line 31:
'''Comparison to other models'''
'''Comparison to other models'''


IFs forecasts the global CBR in 2050 to be 13.89. The World Bank forecasts the CBR in 2050 to be 14.6.[[#_ftn3|[3]]] The UNPD forecasts the CBR in 2050 to be 14.4.[[#_ftn4|[4]]] While these vary by country, overall IFs is lower (see spreadsheet for country specific details).
IFs forecasts the global CBR in 2050 to be 13.89. The World Bank forecasts the CBR in 2050 to be 14.6.<sup>[3] </sup>The UNPD forecasts the CBR in 2050 to be 14.4.<sup>[4]</sup> While these vary by country, overall IFs is lower (see spreadsheet for country specific details).


'''Degenerative test'''
'''Degenerative test'''


Births, TFR, and population are all compatible.[[#_ftn5|[5]]]
Births, TFR, and population are all compatible.<sup>[5]</sup>






== [[Crude_death_rate_(CDR)_(version_7.23)|Crude death rate (CDR)]] ==
== Crude death rate (CDR) ==


=== [[Face_validity_(version_7.23)|Face validity]] ===
'''Face validity'''


Kosovo, Uzbekistan, Azerbaijan, Taiwan have large jumps&nbsp;in 2014.
Kosovo, Uzbekistan, Azerbaijan, Taiwan have large jumps&nbsp;in 2014.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


There is no historical data for Taiwan but Kosovo, Uzbekistan and Azerbaijan are all initialized with values much different from their historical data. We prioritize mortality tables over CDRs so it is not necessarily an error.
There is no historical data for Taiwan but Kosovo, Uzbekistan and Azerbaijan are all initialized with values much different from their historical data. We prioritize mortality tables over CDRs so it is not necessarily an error.


=== [[Comparison_to_other_models_(version_7.23)|Comparison to other models]] ===
'''Comparison to other models'''


Ifs forecasts global CDR in 2050 to be 9.76. The World Bank forecasts global CDR in 2050 to be 9.3. The UNPD forecasts global CDR in 2050 to be 9.4. For country specific comparisons see spreadsheet.
Ifs forecasts global CDR in 2050 to be 9.76. The World Bank forecasts global CDR in 2050 to be 9.3. The UNPD forecasts global CDR in 2050 to be 9.4. For country specific comparisons see spreadsheet.


=== [[Degenerative_test_(version_7.23)|Degenerative test]] ===
'''Degenerative test'''


Deaths, population, CBR, and life expectancy make sense.[[#_ftn6|[6]]]
Deaths, population, CBR, and life expectancy make sense.<sup>[6]</sup>


&nbsp;
== Migration ==


== [[Migration_(version_7.23)|Migration]] ==
'''Face validity'''
 
=== [[Face_validity_(version_7.23)|Face validity]] ===


Bahrain has no migration in 2014. Turkey, Sudan and Eritrea have jumps in&nbsp;% change.
Bahrain has no migration in 2014. Turkey, Sudan and Eritrea have jumps in&nbsp;% change.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


Data for Bahrain is “0” – comes from SeriesForecastNetMigrationUNPD.
Data for Bahrain is “0” – comes from SeriesForecastNetMigrationUNPD.


=== [[Comparison_to_other_models_(version_7.23)|Comparison to other models]] ===
'''Comparison to other models'''


Both UNPD and World Bank use UNPD migration forecast data, the same we use.
Both UNPD and World Bank use UNPD migration forecast data, the same we use.


&nbsp;
'''Degenerative test'''


=== [[Degenerative_test_(version_7.23)|Degenerative test]] ===
CDR, CBR, migration, and population make sense.<sup>[7]</sup>


CDR, CBR, migration, and population make sense.[[#_ftn7|[7]]]
== Total Fertility Rate (TFR) ==


&nbsp;
'''Face validity'''
 
== [[Total_Fertility_Rate_(TFR)_(version_7.23)|Total Fertility Rate (TFR)]] ==
 
=== [[Face_validity_(version_7.23)|Face validity]] ===


Looks good.
Looks good.


=== [[Comparison_to_other_models_(version_7.23)|Comparison to other models]] ===
'''Comparison to other models'''


IFs forecasts global TFR in 2050 to be 2.0851. The World Bank forecasts global TFR in 2050 to be 2.2. The UNPD forecasts global TFR in 2050 to be 2.2. See spreadsheet for country specific comparisons.
IFs forecasts global TFR in 2050 to be 2.0851. The World Bank forecasts global TFR in 2050 to be 2.2. The UNPD forecasts global TFR in 2050 to be 2.2. See spreadsheet for country specific comparisons.


=== [[Degenerative_test_(version_7.23)|Degenerative test]] ===
'''Degenerative test'''


Population, infant mortality, and contraception use all make sense.[[#_ftn8|[8]]]
Population, infant mortality, and contraception use all make sense.<sup>[8]</sup>


= [[Economic_module_(version_7.23)|Economic module]] =
= Economic module =


The key variables we looked at in the economic module are: GDP (MER), GDP per capita (PPP), and value added (for all sectors).
The key variables we looked at in the economic module are: GDP (MER), GDP per capita (PPP), and value added (for all sectors).


&nbsp;
== GDP (MER) ==
 
== [[GDP_(MER)_(version_7.23)|GDP (MER)]] ==


=== [[Face_validity_(version_7.23)|Face validity]] ===
'''Face validity'''


Big drop for Libya in 2011 and then jumps around a bit. Another drop in Libya 2022. Yemen and South Sudan also have pretty drastic variations over recent history due to conflict.<br/>Traces
Big drop for Libya in 2011 and then jumps around a bit. Another drop in Libya 2022. Yemen and South Sudan also have pretty drastic variations over recent history due to conflict.<br/>Traces
Line 107: Line 101:
All of the issues raised in face validity tests (both history and forecast) were reflective of the data – we take IMF GDP until 2021. The drop in Libya in 2022 is a potential issue.
All of the issues raised in face validity tests (both history and forecast) were reflective of the data – we take IMF GDP until 2021. The drop in Libya in 2022 is a potential issue.


&nbsp;
== GDP per capita (PPP) ==
 
== [[GDP_per_capita_(PPP)_(version_7.23)|GDP per capita (PPP)]] ==


=== [[Face_validity_(version_7.23)|Face validity]] ===
'''Face validity'''


GDP per capita for Equatorial Guinea is over 192 thousand USD by 2100, the highest in the world.<br/>Traces
GDP per capita for Equatorial Guinea is over 192 thousand USD by 2100, the highest in the world.<br/>Traces
Line 117: Line 109:
Equatorial Guinea is initialized correctly.
Equatorial Guinea is initialized correctly.


&nbsp;
== Value Added ==


== [[Value_Added_(version_7.23)|Value Added]]<br/>Face validity ==
'''Face validity'''


Because VADD is forced to reconcile with GDP, the historical values are often quite different from the value in the initial year.
Because VADD is forced to reconcile with GDP, the historical values are often quite different from the value in the initial year.


==== Agriculture ====
Agriculture


The biggest changes (in percentage points) in the initial year (2014) for VADD(agriculture) are: Qatar, Kuwait, Bahrain, Trinidad, and UAE.
The biggest changes (in percentage points) in the initial year (2014) for VADD(agriculture) are: Qatar, Kuwait, Bahrain, Trinidad, and UAE.


==== Manufactures ====
Manufactures


The biggest changes (in percentage points) in the initial year (2014) for VADD(manufactures) are: Micronesia, Timor-Leste, Gabon, Kuwait, and Hong Kong.
The biggest changes (in percentage points) in the initial year (2014) for VADD(manufactures) are: Micronesia, Timor-Leste, Gabon, Kuwait, and Hong Kong.


==== Services ====
Services


The biggest changes (in percentage points) in the initial year (2014) for VADD(services) are: Puerto Rico, Republic of Congo, Chad, Gabon, and Azerbaijan.
The biggest changes (in percentage points) in the initial year (2014) for VADD(services) are: Puerto Rico, Republic of Congo, Chad, Gabon, and Azerbaijan.
&nbsp;


The data for ICT value added is scarce and VADD(energy) and VADD(materials) have no historical analogs.
The data for ICT value added is scarce and VADD(energy) and VADD(materials) have no historical analogs.


= [[Health_Module_(version_7.23)|Health Module]] =
= Health Module =


We looked at all types of death and life expectancy as the key variables.
We looked at all types of death and life expectancy as the key variables.


&nbsp;
== Face validity ==


== [[Face_validity_(version_7.23)|Face validity]] ==
'''Traffic'''
 
=== [[Traffic_(version_7.23)|Traffic]] ===


Looks good
Looks good


=== [[Communicable_disease_(other)_(version_7.23)|Communicable disease (other)]] ===
'''Communicable disease (other)'''


Small spike in South Sudan in 2016. Spike in Sierra Leone in 2015. Increase in Equatorial Guinea in 2021.
Small spike in South Sudan in 2016. Spike in Sierra Leone in 2015. Increase in Equatorial Guinea in 2021.


=== [[Non-Communicable_Disease_(NCD)_(version_7.23)|Non-Communicable Disease (NCD)]] ===
'''Non-Communicable Disease (NCD)'''


Steep incline in Hong Kong and Taiwan.
Steep incline in Hong Kong and Taiwan.


=== [[Diabetes_(version_7.23)|Diabetes]] ===
'''Diabetes'''


Kosovo jumps around especially after 2090. Mauritius much higher than other island nations.
Kosovo jumps around especially after 2090. Mauritius much higher than other island nations.


=== [[Malignant_neoplasm_(version_7.23)|Malignant neoplasm]] ===
'''Malignant neoplasm'''


Large increase in Zimbabwe. Large increase in Qatar and UAE for men.
Large increase in Zimbabwe. Large increase in Qatar and UAE for men.


=== [[Cardiovascular_(version_7.23)|Cardiovascular]] ===
'''Cardiovascular'''


Spike in 2015 in Ukraine. Spike in 2015 in Belarus. Large increase then decrease for men in UAE.
Spike in 2015 in Ukraine. Spike in 2015 in Belarus. Large increase then decrease for men in UAE.


=== [[Digestive_(version_7.23)|Digestive]] ===
'''Digestive'''


Spike in 2015 for men in Sierra Leone. Moldova very high and peak in 2051.
Spike in 2015 for men in Sierra Leone. Moldova very high and peak in 2051.


=== [[Respiratory_(version_7.23)|Respiratory]] ===
'''Respiratory'''


Spike in 2015 in Hong Kong and Taiwan for females. Kosovo much higher than other countries with a lot of jumping around.
Spike in 2015 in Hong Kong and Taiwan for females. Kosovo much higher than other countries with a lot of jumping around.


=== [[Diarrhea_(version_7.23)|Diarrhea]] ===
'''Diarrhea'''


Spike in 2015 in Yemen and Sierra Leone. Jump in 2015 and 2016 for Equatorial Guinea.
Spike in 2015 in Yemen and Sierra Leone. Jump in 2015 and 2016 for Equatorial Guinea.


=== [[Malaria_(version_7.23)|Malaria]] ===
'''Malaria'''


Spike in 2016 in South Sudan. Spike in 2015 in Sierra Leone. Spike in 2017 and then decrease in Chad. Increase until 2021 in Equatorial Guinea – almost all other countries decrease.
Spike in 2016 in South Sudan. Spike in 2015 in Sierra Leone. Spike in 2017 and then decrease in Chad. Increase until 2021 in Equatorial Guinea – almost all other countries decrease.


=== [[Respiratory_infections_(version_7.23)|Respiratory infections]] ===
'''Respiratory infections'''


Spike in 2015 in Yemen. Spike in 2016 in South Sudan. Spike in 2015 in Sierra Leone. Large increase then steady decrease in Japan, Singapore, and Thailand. Spike in 2017 in Chad. Increase until 2021 in Equatorial Guinea. Decrease then an increase until 2053 in Seychelles.
Spike in 2015 in Yemen. Spike in 2016 in South Sudan. Spike in 2015 in Sierra Leone. Large increase then steady decrease in Japan, Singapore, and Thailand. Spike in 2017 in Chad. Increase until 2021 in Equatorial Guinea. Decrease then an increase until 2053 in Seychelles.


=== [[Intended_injuries_(version_7.23)|Intended injuries]] ===
'''Intended injuries'''


Countries with high levels of intended injuries (Syria, El Salvador, and Guatemala) maintain these high levels or even increase until 2100.
Countries with high levels of intended injuries (Syria, El Salvador, and Guatemala) maintain these high levels or even increase until 2100.


=== [[HIV/AIDS_(version_7.23)|HIV/AIDS]] ===
'''HIV/AIDS'''


Some countries increase – this “double bubble” is something we have been meaning to look into.
Some countries increase – this “double bubble” is something we have been meaning to look into.


&nbsp;
'''Life expectancy'''
 
=== [[Life_expectancy_(version_7.23)|Life expectancy]] ===


Drop in 2015 in Yemen, Sierra Leone, Hong Kong, Taiwan, Equatorial Guinea.
Drop in 2015 in Yemen, Sierra Leone, Hong Kong, Taiwan, Equatorial Guinea.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


Life expectancy for Malawi, Sierra Leone, Equatorial Guinea, Hong Kong, and Yemen is initialized higher than both the 2015 value (data) and the 2015 forecasted value. This is why it looks like there is a drop in 2015 for these countries.
Life expectancy for Malawi, Sierra Leone, Equatorial Guinea, Hong Kong, and Yemen is initialized higher than both the 2015 value (data) and the 2015 forecasted value. This is why it looks like there is a drop in 2015 for these countries.


&nbsp;
== Comparison to other models ==
 
== [[Comparison_to_other_models_(version_7.23)|Comparison to other models]] ==


[See spreadsheet attached]
[See spreadsheet attached]
Line 217: Line 201:
Overall, our initial death rates and forecasts are often very different from the data and forecast from the WHO Global Burden of Disease forecast.[[#_ftn9|[9]]]
Overall, our initial death rates and forecasts are often very different from the data and forecast from the WHO Global Burden of Disease forecast.[[#_ftn9|[9]]]


&nbsp;
== Extreme conditions test ==
 
== [[Extreme_conditions_test_(version_7.23)|Extreme conditions test]] ==


Tests were run by increasing mortality rates for all diseases (hlmortm). The model behaves as expected – deaths increase dramatically and then decrease as the stock of people is diminished.
Tests were run by increasing mortality rates for all diseases (hlmortm). The model behaves as expected – deaths increase dramatically and then decrease as the stock of people is diminished.


&nbsp;
== Sensitivity analysis ==
 
== [[Sensitivity_analysis_(version_7.23)|Sensitivity analysis]] ==


Sensitivity analysis was conducted on all types of disease and the model behaved as expected i.e. no large increases or decreases (see spreadsheet attached).
Sensitivity analysis was conducted on all types of disease and the model behaved as expected i.e. no large increases or decreases (see spreadsheet attached).


&nbsp;
= Government consumption =
 
= [[Government_consumption_(version_7.23)|Government consumption]] =


We looked at government spending by destination (GDS), government consumption (GOVCON), government expenditures (GOVEXP), government revenues (GOVREV), and government to household transfers (GOVHHTRN).
We looked at government spending by destination (GDS), government consumption (GOVCON), government expenditures (GOVEXP), government revenues (GOVREV), and government to household transfers (GOVHHTRN).


&nbsp;
== Government spending by destination (GDS) ==


== [[Government_spending_by_destination_(GDS)_(version_7.23)|Government spending by destination (GDS)]] ==
'''Face validity'''
 
=== [[Face_validity_(version_7.23)|Face validity]] ===


Bangladesh military spending in the first year. Micronesian health spending looks weird. Most countries have no R&D spending data. Bhutan has a large jump in infrastructure forecast. Many countries do not have infrastructure spending data (same with ‘other’ spending).
Bangladesh military spending in the first year. Micronesian health spending looks weird. Most countries have no R&D spending data. Bhutan has a large jump in infrastructure forecast. Many countries do not have infrastructure spending data (same with ‘other’ spending).


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


Bangladesh military spending data (GovtMil%GDPWDI) has 0.001 for both 2015 and 2014 but we initialize at 1.29.
Bangladesh military spending data (GovtMil%GDPWDI) has 0.001 for both 2015 and 2014 but we initialize at 1.29.


&nbsp;
== Government consumption (GOVCON) ==


== [[Government_consumption_(GOVCON)_(version_7.23)|Government consumption (GOVCON)]] ==
'''Face validity'''
 
=== [[Face_validity_(version_7.23)|Face validity]] ===


Timor-Leste has large drop in 2014. Countries with big spike in initial year: Puerto Rico, Tajikistan, CAR, Panama, Jamaica, Cyprus, Iran, Guinea Bissau, El Salvador, Vietnam, Chad, Somalia, Afghanistan, Cambodia.
Timor-Leste has large drop in 2014. Countries with big spike in initial year: Puerto Rico, Tajikistan, CAR, Panama, Jamaica, Cyprus, Iran, Guinea Bissau, El Salvador, Vietnam, Chad, Somalia, Afghanistan, Cambodia.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


The following countries seem to have problems with the initialization: Timor-Leste, Puerto Rico, Tajikistan, CAR, Panama, Jamaica, Cyprus, Iran, Guinea Bissau, El Salvador, Vietnam, Chad, Somalia, Afghanistan, and Cambodia. See spreadsheet for details.
The following countries seem to have problems with the initialization: Timor-Leste, Puerto Rico, Tajikistan, CAR, Panama, Jamaica, Cyprus, Iran, Guinea Bissau, El Salvador, Vietnam, Chad, Somalia, Afghanistan, and Cambodia. See spreadsheet for details.


&nbsp;
== Government expenditures (GovExp) ==


== [[Government_expenditures_(GovExp)_(version_7.23)|Government expenditures (GovExp)]] ==
'''Face validity'''
 
=== [[Face_validity_(version_7.23)|Face validity]] ===


Jumps in the initial year in Estonia, Japan, Lithuania, and Canada. Liberia looks strange in 2020.
Jumps in the initial year in Estonia, Japan, Lithuania, and Canada. Liberia looks strange in 2020.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


The following countries have issues with the initialization: Estonia, Canada, Japan, Lithuania, and Liberia.
The following countries have issues with the initialization: Estonia, Canada, Japan, Lithuania, and Liberia.


&nbsp;
'''Government revenue (GovRev)'''
 
== [[Government_revenue_(GovRev)_(version_7.23)|Government revenue (GovRev)]] ==


Missing data for many countries. Yemen initialized at 0 in 2014. Liberia and Micronesia behave strangely.
Missing data for many countries. Yemen initialized at 0 in 2014. Liberia and Micronesia behave strangely.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


The following countries seem to have issues with the initialization: Congo, Slovenia, Yemen, Liberia, and Micronesia.
The following countries seem to have issues with the initialization: Congo, Slovenia, Yemen, Liberia, and Micronesia.


&nbsp;
== Government to household transfers (GOVHHTRN) ==
 
== [[Government_to_household_transfers_(GOVHHTRN)_(version_7.23)|Government to household transfers (GOVHHTRN)]] ==


Yemen, Equatorial Guinea, Sierra Leone, and Spain initial year looks like a jump. Singapore has a jump in 2026.
Yemen, Equatorial Guinea, Sierra Leone, and Spain initial year looks like a jump. Singapore has a jump in 2026.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


Post 2014 there are jumps in Singapore, Yemen, Equatorial Guinea, Sierra Leone, and Spain.
Post 2014 there are jumps in Singapore, Yemen, Equatorial Guinea, Sierra Leone, and Spain.


= [[Education_module_(version_7.23)|Education module]] =
= Education module =


The variables we looked at were mean years of education for 15+ age group (EDYRSAG15), gender parity (mean education years 25+), educational enrollment (all levels), and education cost (total, as a percent of GDP).
The variables we looked at were mean years of education for 15+ age group (EDYRSAG15), gender parity (mean education years 25+), educational enrollment (all levels), and education cost (total, as a percent of GDP).


&nbsp;
== Years of education for 15+ ==
 
== [[Years_of_education_for_15+_(version_7.23)|Years of education for 15+]] ==


=== [[Face_validity_(version_7.23)|Face validity]] ===
'''Face validity'''


Looks good.
Looks good.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


Mozambique is initialized high and increases quite dramatically until 2100.
Mozambique is initialized high and increases quite dramatically until 2100.


&nbsp;
== Educational enrolment (all levels) ==


== [[Educational_enrolment_(all_levels)_(version_7.23)|Educational enrolment (all levels)]] ==
'''Face validity'''
 
=== [[Face_validity_(version_7.23)|Face validity]] ===


Secondary enrolment for Australia, Belgium, and Finland looks strange. Tertiary for Australia looks strange.
Secondary enrolment for Australia, Belgium, and Finland looks strange. Tertiary for Australia looks strange.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


Secondary enrolment initialization issues all seem to be that they are initialized lower than the latest year of data (2013). Tertiary is also initialized low.
Secondary enrolment initialization issues all seem to be that they are initialized lower than the latest year of data (2013). Tertiary is also initialized low.


&nbsp;
== Expected years of education (EDYRSSLE) ==


== [[Expected_years_of_education_(EDYRSSLE)_(version_7.23)|Expected years of education (EDYRSSLE)]] ==
'''Face validity'''
 
=== [[Face_validity_(version_7.23)|Face validity]] ===


Looks strange for Australia, Belgium, China, Finalnd, New Zealand, Malaysia, Ireland, Syria, Spain, Seychelles, South Africa, Thailand, UK, and the world.
Looks strange for Australia, Belgium, China, Finalnd, New Zealand, Malaysia, Ireland, Syria, Spain, Seychelles, South Africa, Thailand, UK, and the world.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


Most seem to be initialized lower than the historical value (EdExpectedYearsofSchooling) but some are high. See spreadsheet for details.
Most seem to be initialized lower than the historical value (EdExpectedYearsofSchooling) but some are high. See spreadsheet for details.


&nbsp;
== Education cost ==
 
== [[Education_cost_(version_7.23)|Education cost]] ==


=== [[Face_validity_(version_7.23)|Face validity]] ===
'''Face validity'''


Solomon Islands, Bhutan, Congo, Equatorial Guinea all look weird.
Solomon Islands, Bhutan, Congo, Equatorial Guinea all look weird.


&nbsp;
== Extreme conditions test ==
 
== [[Extreme_conditions_test_(version_7.23)|Extreme conditions test]] ==


Were performed and model behaved well – see spreadsheet for details.
Were performed and model behaved well – see spreadsheet for details.


== [[Sensitivity_analysis_(version_7.23)|Sensitivity analysis]] ==
== Sensitivity analysis ==


Were performed and model behaved well – see spreadsheet for details.
Were performed and model behaved well – see spreadsheet for details.


&nbsp;
= Agriculture =
 
= [[Agriculture_(version_7.23)|Agriculture]] =


The variables we looked at were agricultural production (AGP), agricultural demand (AGDEM), land (LD), yield (YL), ag exports (AGX), ag imports (AGM), and calories per capita (CLPC).
The variables we looked at were agricultural production (AGP), agricultural demand (AGDEM), land (LD), yield (YL), ag exports (AGX), ag imports (AGM), and calories per capita (CLPC).


&nbsp;
== Agricultural production ==


== [[Agricultural_production_(version_7.23)|Agricultural production]] ==
'''Face validity'''
 
=== [[Face_validity_(version_7.23)|Face validity]] ===


The following countries have discrepancies in the first year: Russia, Kazakhstan, UAE (Crop) Montenegro, Palestine, Ethiopia (meat), China (Fish) Many jumps in total category: Kazakhstan, Russia, Cambodia, Ukraine, Gambia, UAE, Timor-Leste, Senegal, Maldives, Singapore, Qatar, Bahrain
The following countries have discrepancies in the first year: Russia, Kazakhstan, UAE (Crop) Montenegro, Palestine, Ethiopia (meat), China (Fish) Many jumps in total category: Kazakhstan, Russia, Cambodia, Ukraine, Gambia, UAE, Timor-Leste, Senegal, Maldives, Singapore, Qatar, Bahrain
Line 365: Line 319:
Also, AGP (fish) does not have a historical analog attached to it so it gives an error when clicked.
Also, AGP (fish) does not have a historical analog attached to it so it gives an error when clicked.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


See spreadsheet for details.
See spreadsheet for details.


&nbsp;
== Agricultural demand (AGDEM) ==


[[Agricultural_demand_(AGDEM)_(version_7.23)|Agricultural demand (AGDEM)]]<br/>Face validity
'''Face validity'''


Austria, Oman, and South Sudan have spikes.
Austria, Oman, and South Sudan have spikes.


&nbsp;
== Land (LD) ==


== [[Land_(LD)_(version_7.23)|Land (LD)]] ==
'''Face validity'''
 
=== [[Face_validity_(version_7.23)|Face validity]] ===


Crop land looks strange for: Lebanon, Sudan, New Zealand, Jordan, Tanzania, Botswana, Lesotho, and Cuba. Forest land looks weird for: Tonga and Comoros. ‘Other’ land looks weird for: China. Total land looks weird for: Burundi, Nigeria, Rwanda, Sierra Leone, Uruguay, and Austria.
Crop land looks strange for: Lebanon, Sudan, New Zealand, Jordan, Tanzania, Botswana, Lesotho, and Cuba. Forest land looks weird for: Tonga and Comoros. ‘Other’ land looks weird for: China. Total land looks weird for: Burundi, Nigeria, Rwanda, Sierra Leone, Uruguay, and Austria.


=== [[Traces_(version_7.23)|Traces]] ===
'''Traces'''


See spreadsheet
See spreadsheet


&nbsp;
== Yield (YL) ==
 
== [[Yield_(YL)_(version_7.23)|Yield (YL)]] ==


Looks strange for Maldives, Russia, Kazakstan, UAE, and Djibouti.
Looks strange for Maldives, Russia, Kazakstan, UAE, and Djibouti.


&nbsp;
== Ag exports (AGX) ==
 
== [[Ag_exports_(AGX)_(version_7.23)|Ag exports (AGX)]] ==


Brazilian crop exports have some spikes. Indian meat looks weird and Chinese fish looks weird.
Brazilian crop exports have some spikes. Indian meat looks weird and Chinese fish looks weird.


&nbsp;
== Ag imports (AGM) ==
 
== [[Ag_imports_(AGM)_(version_7.23)|Ag imports (AGM)]] ==


Crop looks strange in China, Nigeria, Tonga, Kosovo, Malawi, Zambia,&nbsp;Comoros, Equatorial Guinea, and Brazil. Fish looks strange in: Nigeria and China. Total looks strange in China and Nigeria as well.
Crop looks strange in China, Nigeria, Tonga, Kosovo, Malawi, Zambia,&nbsp;Comoros, Equatorial Guinea, and Brazil. Fish looks strange in: Nigeria and China. Total looks strange in China and Nigeria as well.


&nbsp;
== Calories per capita (CLPC) ==
 
== [[Calories_per_capita_(CLPC)_(version_7.23)|Calories per capita (CLPC)]] ==


Sierra Leone, Somalia, and Chad behave very strangely. Angola has some initial year discrepancies.
Sierra Leone, Somalia, and Chad behave very strangely. Angola has some initial year discrepancies.


= [[Next_steps_(version_7.23)|Next steps]] =
= Next steps =


From the operational validation exercises, issues have been identified and we have begun to diagnose these issues as either conceptual model issues or computerized model issues. This was not a fully comprehensive validation – there are still modules that need to be validated: infrastructure, socio-political, and energy. We also need to identify additional models to compare IFs to.
From the operational validation exercises, issues have been identified and we have begun to diagnose these issues as either conceptual model issues or computerized model issues. This was not a fully comprehensive validation – there are still modules that need to be validated: infrastructure, socio-political, and energy. We also need to identify additional models to compare IFs to.
Line 417: Line 361:
&nbsp;
&nbsp;


== [[Conceptual_model_issues_(version_7.23)|Conceptual model issues]] ==
== Conceptual model issues ==


#We preference mortality tables over CDRs
#We preference mortality tables over CDRs
Line 424: Line 368:
#Equatorial Guinea GDP per capita gets very high by 2100
#Equatorial Guinea GDP per capita gets very high by 2100


== [[Computerized_model_issues_(version_7.23)|Computerized model issues]] ==
== Computerized model issues ==


#VarLinks in IFsVar needs to be updated – the “drivers” display contains parameters that no longer exist
#VarLinks in IFsVar needs to be updated – the “drivers” display contains parameters that no longer exist
Line 430: Line 374:
#Some GDS and VADD dimensions do not have historical analogs – they give errors when clicked
#Some GDS and VADD dimensions do not have historical analogs – they give errors when clicked


== [[Issues_yet_to_be_diagnosed_(version_7.23)|Issues yet to be diagnosed]] ==
== Issues yet to be diagnosed ==


#All of the transients in the initial year identified throughout this document<ol style="list-style-type:lower-alpha;">
#All of the transients in the initial year identified throughout this document<ol style="list-style-type:lower-alpha;">
</ol></ol>


*There are transients in almost every module but VADD sticks out because there are likely conceptual model issues that need to be addressed i.e. that energy value added is manufacturing less materials
*There are transients in almost every module but VADD sticks out because there are likely conceptual model issues that need to be addressed i.e. that energy value added is manufacturing less materials
*All “strange’ behaviour in later years in the forecasts (see above)
*All “strange’ behaviour in later years in the forecasts (see above)
&lt;/ol&gt;
&nbsp;
<div><br/>
<div><br/>
----
----

Latest revision as of 09:41, 16 September 2016

This page contains an overview of the model validating and verification of IFs version 7.23 in August 2016. For more detailed analysis of the results, please refer to the excel spreadsheet available here here.

Population module

The key variables in the population module, upon which the validation techniques were conducted, were: population, crude death rate, crude birth rate, migration, and total fertility rate.

Population

Face validity

Initial year looks fine – there are no large initial year discrepancies. There is a large increase in population in Syria starting in 2016. The data indicate that population has decreased dramatically in Syria starting in 2011 (due to conflict) and the model forecasts that the population will bounce back over the short term. Likewise, the population of Lebanon has increased dramatically since 2011 (likely due to Syrian refugees), and the model forecasts population to decrease over the short term (2016 – 2020).

Comparison to other models

IFs forecasts global population in 2050 to be 9,542,324,000. The World Bank’s Population Estimates and Projections forecasts global population to be 9,714,894,000 in 2050.[1] The United Nations Population Division (UNPD) forecasts global population to be 9,725,148,000 in 2050.[2]

Degenerative test

Crude birth rate (CBR), crude death rate (CDR), and migration make sense for 4 most populous countries in 2050: US, Nigeria, China, and India.

Crude birth rate (CBR)

Face validity

Hong Kong, Jamaica, Puerto Rico, and Nigeria have discrepancies in the first year (2014).

Traces

There is no historical data for Hong Kong. Jamaica, Puerto Rico, and Nigeria are initialized at different values than the data (see spreadsheet).

Comparison to other models

IFs forecasts the global CBR in 2050 to be 13.89. The World Bank forecasts the CBR in 2050 to be 14.6.[3] The UNPD forecasts the CBR in 2050 to be 14.4.[4] While these vary by country, overall IFs is lower (see spreadsheet for country specific details).

Degenerative test

Births, TFR, and population are all compatible.[5]


Crude death rate (CDR)

Face validity

Kosovo, Uzbekistan, Azerbaijan, Taiwan have large jumps in 2014.

Traces

There is no historical data for Taiwan but Kosovo, Uzbekistan and Azerbaijan are all initialized with values much different from their historical data. We prioritize mortality tables over CDRs so it is not necessarily an error.

Comparison to other models

Ifs forecasts global CDR in 2050 to be 9.76. The World Bank forecasts global CDR in 2050 to be 9.3. The UNPD forecasts global CDR in 2050 to be 9.4. For country specific comparisons see spreadsheet.

Degenerative test

Deaths, population, CBR, and life expectancy make sense.[6]

Migration

Face validity

Bahrain has no migration in 2014. Turkey, Sudan and Eritrea have jumps in % change.

Traces

Data for Bahrain is “0” – comes from SeriesForecastNetMigrationUNPD.

Comparison to other models

Both UNPD and World Bank use UNPD migration forecast data, the same we use.

Degenerative test

CDR, CBR, migration, and population make sense.[7]

Total Fertility Rate (TFR)

Face validity

Looks good.

Comparison to other models

IFs forecasts global TFR in 2050 to be 2.0851. The World Bank forecasts global TFR in 2050 to be 2.2. The UNPD forecasts global TFR in 2050 to be 2.2. See spreadsheet for country specific comparisons.

Degenerative test

Population, infant mortality, and contraception use all make sense.[8]

Economic module

The key variables we looked at in the economic module are: GDP (MER), GDP per capita (PPP), and value added (for all sectors).

GDP (MER)

Face validity

Big drop for Libya in 2011 and then jumps around a bit. Another drop in Libya 2022. Yemen and South Sudan also have pretty drastic variations over recent history due to conflict.
Traces

All of the issues raised in face validity tests (both history and forecast) were reflective of the data – we take IMF GDP until 2021. The drop in Libya in 2022 is a potential issue.

GDP per capita (PPP)

Face validity

GDP per capita for Equatorial Guinea is over 192 thousand USD by 2100, the highest in the world.
Traces

Equatorial Guinea is initialized correctly.

Value Added

Face validity

Because VADD is forced to reconcile with GDP, the historical values are often quite different from the value in the initial year.

Agriculture

The biggest changes (in percentage points) in the initial year (2014) for VADD(agriculture) are: Qatar, Kuwait, Bahrain, Trinidad, and UAE.

Manufactures

The biggest changes (in percentage points) in the initial year (2014) for VADD(manufactures) are: Micronesia, Timor-Leste, Gabon, Kuwait, and Hong Kong.

Services

The biggest changes (in percentage points) in the initial year (2014) for VADD(services) are: Puerto Rico, Republic of Congo, Chad, Gabon, and Azerbaijan.

The data for ICT value added is scarce and VADD(energy) and VADD(materials) have no historical analogs.

Health Module

We looked at all types of death and life expectancy as the key variables.

Face validity

Traffic

Looks good

Communicable disease (other)

Small spike in South Sudan in 2016. Spike in Sierra Leone in 2015. Increase in Equatorial Guinea in 2021.

Non-Communicable Disease (NCD)

Steep incline in Hong Kong and Taiwan.

Diabetes

Kosovo jumps around especially after 2090. Mauritius much higher than other island nations.

Malignant neoplasm

Large increase in Zimbabwe. Large increase in Qatar and UAE for men.

Cardiovascular

Spike in 2015 in Ukraine. Spike in 2015 in Belarus. Large increase then decrease for men in UAE.

Digestive

Spike in 2015 for men in Sierra Leone. Moldova very high and peak in 2051.

Respiratory

Spike in 2015 in Hong Kong and Taiwan for females. Kosovo much higher than other countries with a lot of jumping around.

Diarrhea

Spike in 2015 in Yemen and Sierra Leone. Jump in 2015 and 2016 for Equatorial Guinea.

Malaria

Spike in 2016 in South Sudan. Spike in 2015 in Sierra Leone. Spike in 2017 and then decrease in Chad. Increase until 2021 in Equatorial Guinea – almost all other countries decrease.

Respiratory infections

Spike in 2015 in Yemen. Spike in 2016 in South Sudan. Spike in 2015 in Sierra Leone. Large increase then steady decrease in Japan, Singapore, and Thailand. Spike in 2017 in Chad. Increase until 2021 in Equatorial Guinea. Decrease then an increase until 2053 in Seychelles.

Intended injuries

Countries with high levels of intended injuries (Syria, El Salvador, and Guatemala) maintain these high levels or even increase until 2100.

HIV/AIDS

Some countries increase – this “double bubble” is something we have been meaning to look into.

Life expectancy

Drop in 2015 in Yemen, Sierra Leone, Hong Kong, Taiwan, Equatorial Guinea.

Traces

Life expectancy for Malawi, Sierra Leone, Equatorial Guinea, Hong Kong, and Yemen is initialized higher than both the 2015 value (data) and the 2015 forecasted value. This is why it looks like there is a drop in 2015 for these countries.

Comparison to other models

[See spreadsheet attached]

Overall, our initial death rates and forecasts are often very different from the data and forecast from the WHO Global Burden of Disease forecast.[9]

Extreme conditions test

Tests were run by increasing mortality rates for all diseases (hlmortm). The model behaves as expected – deaths increase dramatically and then decrease as the stock of people is diminished.

Sensitivity analysis

Sensitivity analysis was conducted on all types of disease and the model behaved as expected i.e. no large increases or decreases (see spreadsheet attached).

Government consumption

We looked at government spending by destination (GDS), government consumption (GOVCON), government expenditures (GOVEXP), government revenues (GOVREV), and government to household transfers (GOVHHTRN).

Government spending by destination (GDS)

Face validity

Bangladesh military spending in the first year. Micronesian health spending looks weird. Most countries have no R&D spending data. Bhutan has a large jump in infrastructure forecast. Many countries do not have infrastructure spending data (same with ‘other’ spending).

Traces

Bangladesh military spending data (GovtMil%GDPWDI) has 0.001 for both 2015 and 2014 but we initialize at 1.29.

Government consumption (GOVCON)

Face validity

Timor-Leste has large drop in 2014. Countries with big spike in initial year: Puerto Rico, Tajikistan, CAR, Panama, Jamaica, Cyprus, Iran, Guinea Bissau, El Salvador, Vietnam, Chad, Somalia, Afghanistan, Cambodia.

Traces

The following countries seem to have problems with the initialization: Timor-Leste, Puerto Rico, Tajikistan, CAR, Panama, Jamaica, Cyprus, Iran, Guinea Bissau, El Salvador, Vietnam, Chad, Somalia, Afghanistan, and Cambodia. See spreadsheet for details.

Government expenditures (GovExp)

Face validity

Jumps in the initial year in Estonia, Japan, Lithuania, and Canada. Liberia looks strange in 2020.

Traces

The following countries have issues with the initialization: Estonia, Canada, Japan, Lithuania, and Liberia.

Government revenue (GovRev)

Missing data for many countries. Yemen initialized at 0 in 2014. Liberia and Micronesia behave strangely.

Traces

The following countries seem to have issues with the initialization: Congo, Slovenia, Yemen, Liberia, and Micronesia.

Government to household transfers (GOVHHTRN)

Yemen, Equatorial Guinea, Sierra Leone, and Spain initial year looks like a jump. Singapore has a jump in 2026.

Traces

Post 2014 there are jumps in Singapore, Yemen, Equatorial Guinea, Sierra Leone, and Spain.

Education module

The variables we looked at were mean years of education for 15+ age group (EDYRSAG15), gender parity (mean education years 25+), educational enrollment (all levels), and education cost (total, as a percent of GDP).

Years of education for 15+

Face validity

Looks good.

Traces

Mozambique is initialized high and increases quite dramatically until 2100.

Educational enrolment (all levels)

Face validity

Secondary enrolment for Australia, Belgium, and Finland looks strange. Tertiary for Australia looks strange.

Traces

Secondary enrolment initialization issues all seem to be that they are initialized lower than the latest year of data (2013). Tertiary is also initialized low.

Expected years of education (EDYRSSLE)

Face validity

Looks strange for Australia, Belgium, China, Finalnd, New Zealand, Malaysia, Ireland, Syria, Spain, Seychelles, South Africa, Thailand, UK, and the world.

Traces

Most seem to be initialized lower than the historical value (EdExpectedYearsofSchooling) but some are high. See spreadsheet for details.

Education cost

Face validity

Solomon Islands, Bhutan, Congo, Equatorial Guinea all look weird.

Extreme conditions test

Were performed and model behaved well – see spreadsheet for details.

Sensitivity analysis

Were performed and model behaved well – see spreadsheet for details.

Agriculture

The variables we looked at were agricultural production (AGP), agricultural demand (AGDEM), land (LD), yield (YL), ag exports (AGX), ag imports (AGM), and calories per capita (CLPC).

Agricultural production

Face validity

The following countries have discrepancies in the first year: Russia, Kazakhstan, UAE (Crop) Montenegro, Palestine, Ethiopia (meat), China (Fish) Many jumps in total category: Kazakhstan, Russia, Cambodia, Ukraine, Gambia, UAE, Timor-Leste, Senegal, Maldives, Singapore, Qatar, Bahrain

Also, AGP (fish) does not have a historical analog attached to it so it gives an error when clicked.

Traces

See spreadsheet for details.

Agricultural demand (AGDEM)

Face validity

Austria, Oman, and South Sudan have spikes.

Land (LD)

Face validity

Crop land looks strange for: Lebanon, Sudan, New Zealand, Jordan, Tanzania, Botswana, Lesotho, and Cuba. Forest land looks weird for: Tonga and Comoros. ‘Other’ land looks weird for: China. Total land looks weird for: Burundi, Nigeria, Rwanda, Sierra Leone, Uruguay, and Austria.

Traces

See spreadsheet

Yield (YL)

Looks strange for Maldives, Russia, Kazakstan, UAE, and Djibouti.

Ag exports (AGX)

Brazilian crop exports have some spikes. Indian meat looks weird and Chinese fish looks weird.

Ag imports (AGM)

Crop looks strange in China, Nigeria, Tonga, Kosovo, Malawi, Zambia, Comoros, Equatorial Guinea, and Brazil. Fish looks strange in: Nigeria and China. Total looks strange in China and Nigeria as well.

Calories per capita (CLPC)

Sierra Leone, Somalia, and Chad behave very strangely. Angola has some initial year discrepancies.

Next steps

From the operational validation exercises, issues have been identified and we have begun to diagnose these issues as either conceptual model issues or computerized model issues. This was not a fully comprehensive validation – there are still modules that need to be validated: infrastructure, socio-political, and energy. We also need to identify additional models to compare IFs to.

 

Conceptual model issues

  1. We preference mortality tables over CDRs
  2. Death rate forecasts are very different from GBD
  3. “Double bubble” of AIDs deaths
  4. Equatorial Guinea GDP per capita gets very high by 2100

Computerized model issues

  1. VarLinks in IFsVar needs to be updated – the “drivers” display contains parameters that no longer exist
  2. AGP (fish) does not have a historical analog attached to it so it gives an error when clicked.
  3. Some GDS and VADD dimensions do not have historical analogs – they give errors when clicked

Issues yet to be diagnosed

  1. All of the transients in the initial year identified throughout this document
      • There are transients in almost every module but VADD sticks out because there are likely conceptual model issues that need to be addressed i.e. that energy value added is manufacturing less materials
      • All “strange’ behaviour in later years in the forecasts (see above)

      [1] World Bank Population Estimates and Projections http://data.worldbank.org/data-catalog/population-projection-tables Accessed September 1st, 2016.

      [2] UNPD World Population Prospects, 2015 revision https://esa.un.org/unpd/wpp/DataQuery/ Accessed September 1st, 2016.

      [3] World Bank Population Estimates and Projections http://data.worldbank.org/data-catalog/population-projection-tables Accessed September 1st, 2016.

      [4] UNPD World Population Prospects, 2015 revision https://esa.un.org/unpd/wpp/DataQuery/ Accessed September 1st, 2016.

      [5] CBR degenerative test method: Picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data.

      [6] CDR degenerative test method: Picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data

      [7] Migration degenerative test method: Picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data

      [8] TFR degenerative test method: picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data.

      [9] WHO Projections of mortality and burden of disease < http://www.who.int/healthinfo/global_burden_disease/projections2004/en/> accessed September 1st, 2016.

    Retrieved from "https://pardeewiki.du.edu//index.php?title=August_2016_model_validation_and_verification_(version_7.23)&oldid=1253"