August 2016 model validation and verification (version 7.23)
This page contains an overview of the model validating and verification of IFs version 7.23 in August 2016. For more detailed analysis of the results, please refer to the excel spreadsheet attached.
Population module
The key variables in the population module, upon which the validation techniques were conducted, were: population, crude death rate, crude birth rate, migration, and total fertility rate.
Population
Face validity
Initial year looks fine – there are no large initial year discrepancies. There is a large increase in population in Syria starting in 2016. The data indicate that population has decreased dramatically in Syria starting in 2011 (due to conflict) and the model forecasts that the population will bounce back over the short term. Likewise, the population of Lebanon has increased dramatically since 2011 (likely due to Syrian refugees), and the model forecasts population to decrease over the short term (2016 – 2020).
Comparison to other models
IFs forecasts global population in 2050 to be 9,542,324,000. The World Bank’s Population Estimates and Projections forecasts global population to be 9,714,894,000 in 2050.[1] The United Nations Population Division (UNPD) forecasts global population to be 9,725,148,000 in 2050.[2]
Degenerative test
Crude birth rate (CBR), crude death rate (CDR), and migration make sense for 4 most populous countries in 2050: US, Nigeria, China, and India.
Crude birth rate (CBR)
Face validity
Hong Kong, Jamaica, Puerto Rico, and Nigeria have discrepancies in the first year (2014).
Traces
There is no historical data for Hong Kong. Jamaica, Puerto Rico, and Nigeria are initialized at different values than the data (see spreadsheet).
Comparison to other models
IFs forecasts the global CBR in 2050 to be 13.89. The World Bank forecasts the CBR in 2050 to be 14.6.[3] The UNPD forecasts the CBR in 2050 to be 14.4.[4] While these vary by country, overall IFs is lower (see spreadsheet for country specific details).
Degenerative test
Births, TFR, and population are all compatible.[5]
Crude death rate (CDR)
Face validity
Kosovo, Uzbekistan, Azerbaijan, Taiwan have large jumps in 2014.
Traces
There is no historical data for Taiwan but Kosovo, Uzbekistan and Azerbaijan are all initialized with values much different from their historical data. We prioritize mortality tables over CDRs so it is not necessarily an error.
Comparison to other models
Ifs forecasts global CDR in 2050 to be 9.76. The World Bank forecasts global CDR in 2050 to be 9.3. The UNPD forecasts global CDR in 2050 to be 9.4. For country specific comparisons see spreadsheet.
Degenerative test
Deaths, population, CBR, and life expectancy make sense.[6]
Migration
Face validity
Bahrain has no migration in 2014. Turkey, Sudan and Eritrea have jumps in % change.
Traces
Data for Bahrain is “0” – comes from SeriesForecastNetMigrationUNPD.
Comparison to other models
Both UNPD and World Bank use UNPD migration forecast data, the same we use.
Degenerative test
CDR, CBR, migration, and population make sense.[7]
Total Fertility Rate (TFR)
Face validity
Looks good.
Comparison to other models
IFs forecasts global TFR in 2050 to be 2.0851. The World Bank forecasts global TFR in 2050 to be 2.2. The UNPD forecasts global TFR in 2050 to be 2.2. See spreadsheet for country specific comparisons.
Degenerative test
Population, infant mortality, and contraception use all make sense.[8]
Economic module
The key variables we looked at in the economic module are: GDP (MER), GDP per capita (PPP), and value added (for all sectors).
GDP (MER)
Face validity
Big drop for Libya in 2011 and then jumps around a bit. Another drop in Libya 2022. Yemen and South Sudan also have pretty drastic variations over recent history due to conflict.
Traces
All of the issues raised in face validity tests (both history and forecast) were reflective of the data – we take IMF GDP until 2021. The drop in Libya in 2022 is a potential issue.
GDP per capita (PPP)
Face validity
GDP per capita for Equatorial Guinea is over 192 thousand USD by 2100, the highest in the world.
Traces
Equatorial Guinea is initialized correctly.
Value Added
Face validity
Because VADD is forced to reconcile with GDP, the historical values are often quite different from the value in the initial year.
Agriculture
The biggest changes (in percentage points) in the initial year (2014) for VADD(agriculture) are: Qatar, Kuwait, Bahrain, Trinidad, and UAE.
Manufactures
The biggest changes (in percentage points) in the initial year (2014) for VADD(manufactures) are: Micronesia, Timor-Leste, Gabon, Kuwait, and Hong Kong.
Services
The biggest changes (in percentage points) in the initial year (2014) for VADD(services) are: Puerto Rico, Republic of Congo, Chad, Gabon, and Azerbaijan.
The data for ICT value added is scarce and VADD(energy) and VADD(materials) have no historical analogs.
Health Module
We looked at all types of death and life expectancy as the key variables.
Face validity
Traffic
Looks good
Communicable disease (other)
Small spike in South Sudan in 2016. Spike in Sierra Leone in 2015. Increase in Equatorial Guinea in 2021.
Non-Communicable Disease (NCD)
Steep incline in Hong Kong and Taiwan.
Diabetes
Kosovo jumps around especially after 2090. Mauritius much higher than other island nations.
Malignant neoplasm
Large increase in Zimbabwe. Large increase in Qatar and UAE for men.
Cardiovascular
Spike in 2015 in Ukraine. Spike in 2015 in Belarus. Large increase then decrease for men in UAE.
Digestive
Spike in 2015 for men in Sierra Leone. Moldova very high and peak in 2051.
Respiratory
Spike in 2015 in Hong Kong and Taiwan for females. Kosovo much higher than other countries with a lot of jumping around.
Diarrhea
Spike in 2015 in Yemen and Sierra Leone. Jump in 2015 and 2016 for Equatorial Guinea.
Malaria
Spike in 2016 in South Sudan. Spike in 2015 in Sierra Leone. Spike in 2017 and then decrease in Chad. Increase until 2021 in Equatorial Guinea – almost all other countries decrease.
Respiratory infections
Spike in 2015 in Yemen. Spike in 2016 in South Sudan. Spike in 2015 in Sierra Leone. Large increase then steady decrease in Japan, Singapore, and Thailand. Spike in 2017 in Chad. Increase until 2021 in Equatorial Guinea. Decrease then an increase until 2053 in Seychelles.
Intended injuries
Countries with high levels of intended injuries (Syria, El Salvador, and Guatemala) maintain these high levels or even increase until 2100.
HIV/AIDS
Some countries increase – this “double bubble” is something we have been meaning to look into.
Life expectancy
Drop in 2015 in Yemen, Sierra Leone, Hong Kong, Taiwan, Equatorial Guinea.
Traces
Life expectancy for Malawi, Sierra Leone, Equatorial Guinea, Hong Kong, and Yemen is initialized higher than both the 2015 value (data) and the 2015 forecasted value. This is why it looks like there is a drop in 2015 for these countries.
Comparison to other models
[See spreadsheet attached]
Overall, our initial death rates and forecasts are often very different from the data and forecast from the WHO Global Burden of Disease forecast.[9]
Extreme conditions test
Tests were run by increasing mortality rates for all diseases (hlmortm). The model behaves as expected – deaths increase dramatically and then decrease as the stock of people is diminished.
Sensitivity analysis
Sensitivity analysis was conducted on all types of disease and the model behaved as expected i.e. no large increases or decreases (see spreadsheet attached).
Government consumption
We looked at government spending by destination (GDS), government consumption (GOVCON), government expenditures (GOVEXP), government revenues (GOVREV), and government to household transfers (GOVHHTRN).
Government spending by destination (GDS)
Face validity
Bangladesh military spending in the first year. Micronesian health spending looks weird. Most countries have no R&D spending data. Bhutan has a large jump in infrastructure forecast. Many countries do not have infrastructure spending data (same with ‘other’ spending).
Traces
Bangladesh military spending data (GovtMil%GDPWDI) has 0.001 for both 2015 and 2014 but we initialize at 1.29.
Government consumption (GOVCON)
Face validity
Timor-Leste has large drop in 2014. Countries with big spike in initial year: Puerto Rico, Tajikistan, CAR, Panama, Jamaica, Cyprus, Iran, Guinea Bissau, El Salvador, Vietnam, Chad, Somalia, Afghanistan, Cambodia.
Traces
The following countries seem to have problems with the initialization: Timor-Leste, Puerto Rico, Tajikistan, CAR, Panama, Jamaica, Cyprus, Iran, Guinea Bissau, El Salvador, Vietnam, Chad, Somalia, Afghanistan, and Cambodia. See spreadsheet for details.
Government expenditures (GovExp)
Face validity
Jumps in the initial year in Estonia, Japan, Lithuania, and Canada. Liberia looks strange in 2020.
Traces
The following countries have issues with the initialization: Estonia, Canada, Japan, Lithuania, and Liberia.
Government revenue (GovRev)
Missing data for many countries. Yemen initialized at 0 in 2014. Liberia and Micronesia behave strangely.
Traces
The following countries seem to have issues with the initialization: Congo, Slovenia, Yemen, Liberia, and Micronesia.
Government to household transfers (GOVHHTRN)
Yemen, Equatorial Guinea, Sierra Leone, and Spain initial year looks like a jump. Singapore has a jump in 2026.
Traces
Post 2014 there are jumps in Singapore, Yemen, Equatorial Guinea, Sierra Leone, and Spain.
Education module
The variables we looked at were mean years of education for 15+ age group (EDYRSAG15), gender parity (mean education years 25+), educational enrollment (all levels), and education cost (total, as a percent of GDP).
Years of education for 15+
Face validity
Looks good.
Traces
Mozambique is initialized high and increases quite dramatically until 2100.
Educational enrolment (all levels)
Face validity
Secondary enrolment for Australia, Belgium, and Finland looks strange. Tertiary for Australia looks strange.
Traces
Secondary enrolment initialization issues all seem to be that they are initialized lower than the latest year of data (2013). Tertiary is also initialized low.
Expected years of education (EDYRSSLE)
Face validity
Looks strange for Australia, Belgium, China, Finalnd, New Zealand, Malaysia, Ireland, Syria, Spain, Seychelles, South Africa, Thailand, UK, and the world.
Traces
Most seem to be initialized lower than the historical value (EdExpectedYearsofSchooling) but some are high. See spreadsheet for details.
Education cost
Face validity
Solomon Islands, Bhutan, Congo, Equatorial Guinea all look weird.
Extreme conditions test
Were performed and model behaved well – see spreadsheet for details.
Sensitivity analysis
Were performed and model behaved well – see spreadsheet for details.
Agriculture
The variables we looked at were agricultural production (AGP), agricultural demand (AGDEM), land (LD), yield (YL), ag exports (AGX), ag imports (AGM), and calories per capita (CLPC).
Agricultural production
Face validity
The following countries have discrepancies in the first year: Russia, Kazakhstan, UAE (Crop) Montenegro, Palestine, Ethiopia (meat), China (Fish) Many jumps in total category: Kazakhstan, Russia, Cambodia, Ukraine, Gambia, UAE, Timor-Leste, Senegal, Maldives, Singapore, Qatar, Bahrain
Also, AGP (fish) does not have a historical analog attached to it so it gives an error when clicked.
Traces
See spreadsheet for details.
Agricultural demand (AGDEM)
Face validity
Austria, Oman, and South Sudan have spikes.
Land (LD)
Face validity
Crop land looks strange for: Lebanon, Sudan, New Zealand, Jordan, Tanzania, Botswana, Lesotho, and Cuba. Forest land looks weird for: Tonga and Comoros. ‘Other’ land looks weird for: China. Total land looks weird for: Burundi, Nigeria, Rwanda, Sierra Leone, Uruguay, and Austria.
Traces
See spreadsheet
Yield (YL)
Looks strange for Maldives, Russia, Kazakstan, UAE, and Djibouti.
Ag exports (AGX)
Brazilian crop exports have some spikes. Indian meat looks weird and Chinese fish looks weird.
Ag imports (AGM)
Crop looks strange in China, Nigeria, Tonga, Kosovo, Malawi, Zambia, Comoros, Equatorial Guinea, and Brazil. Fish looks strange in: Nigeria and China. Total looks strange in China and Nigeria as well.
Calories per capita (CLPC)
Sierra Leone, Somalia, and Chad behave very strangely. Angola has some initial year discrepancies.
Next steps
From the operational validation exercises, issues have been identified and we have begun to diagnose these issues as either conceptual model issues or computerized model issues. This was not a fully comprehensive validation – there are still modules that need to be validated: infrastructure, socio-political, and energy. We also need to identify additional models to compare IFs to.
Conceptual model issues
- We preference mortality tables over CDRs
- Death rate forecasts are very different from GBD
- “Double bubble” of AIDs deaths
- Equatorial Guinea GDP per capita gets very high by 2100
Computerized model issues
- VarLinks in IFsVar needs to be updated – the “drivers” display contains parameters that no longer exist
- AGP (fish) does not have a historical analog attached to it so it gives an error when clicked.
- Some GDS and VADD dimensions do not have historical analogs – they give errors when clicked
Issues yet to be diagnosed
- All of the transients in the initial year identified throughout this document
</ol>
- There are transients in almost every module but VADD sticks out because there are likely conceptual model issues that need to be addressed i.e. that energy value added is manufacturing less materials
- All “strange’ behaviour in later years in the forecasts (see above)
[1] World Bank Population Estimates and Projections http://data.worldbank.org/data-catalog/population-projection-tables Accessed September 1st, 2016.
[2] UNPD World Population Prospects, 2015 revision https://esa.un.org/unpd/wpp/DataQuery/ Accessed September 1st, 2016.
[3] World Bank Population Estimates and Projections http://data.worldbank.org/data-catalog/population-projection-tables Accessed September 1st, 2016.
[4] UNPD World Population Prospects, 2015 revision https://esa.un.org/unpd/wpp/DataQuery/ Accessed September 1st, 2016.
[5] CBR degenerative test method: Picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data.
[6] CDR degenerative test method: Picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data
[7] Migration degenerative test method: Picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data
[8] TFR degenerative test method: picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data.
[9] WHO Projections of mortality and burden of disease < http://www.who.int/healthinfo/global_burden_disease/projections2004/en/> accessed September 1st, 2016.