August 2016 model validation and verification (version 7.23)

From Pardee Wiki
Revision as of 09:35, 16 September 2016 by Wikiadmin (talk | contribs)
Jump to navigation Jump to search

This page contains an overview of the model validating and verification of IFs version 7.23 in August 2016. For more detailed analysis of the results, please refer to the excel spreadsheet attached. 

Population module

The key variables in the population module, upon which the validation techniques were conducted, were: population, crude death rate, crude birth rate, migration, and total fertility rate.

Population

Face validity

Initial year looks fine – there are no large initial year discrepancies. There is a large increase in population in Syria starting in 2016. The data indicate that population has decreased dramatically in Syria starting in 2011 (due to conflict) and the model forecasts that the population will bounce back over the short term. Likewise, the population of Lebanon has increased dramatically since 2011 (likely due to Syrian refugees), and the model forecasts population to decrease over the short term (2016 – 2020).

Comparison to other models

IFs forecasts global population in 2050 to be 9,542,324,000. The World Bank’s Population Estimates and Projections forecasts global population to be 9,714,894,000 in 2050.[1] The United Nations Population Division (UNPD) forecasts global population to be 9,725,148,000 in 2050.[2]

Degenerative test

Crude birth rate (CBR), crude death rate (CDR), and migration make sense for 4 most populous countries in 2050: US, Nigeria, China, and India.

Crude birth rate (CBR)

Face validity

Hong Kong, Jamaica, Puerto Rico, and Nigeria have discrepancies in the first year (2014).

Traces

There is no historical data for Hong Kong. Jamaica, Puerto Rico, and Nigeria are initialized at different values than the data (see spreadsheet).

Comparison to other models

IFs forecasts the global CBR in 2050 to be 13.89. The World Bank forecasts the CBR in 2050 to be 14.6.[3] The UNPD forecasts the CBR in 2050 to be 14.4.[4] While these vary by country, overall IFs is lower (see spreadsheet for country specific details).

Degenerative test

Births, TFR, and population are all compatible.[5]


Crude death rate (CDR)

Face validity

Kosovo, Uzbekistan, Azerbaijan, Taiwan have large jumps in 2014.

Traces

There is no historical data for Taiwan but Kosovo, Uzbekistan and Azerbaijan are all initialized with values much different from their historical data. We prioritize mortality tables over CDRs so it is not necessarily an error.

Comparison to other models

Ifs forecasts global CDR in 2050 to be 9.76. The World Bank forecasts global CDR in 2050 to be 9.3. The UNPD forecasts global CDR in 2050 to be 9.4. For country specific comparisons see spreadsheet.

Degenerative test

Deaths, population, CBR, and life expectancy make sense.[6]

Migration

Face validity

Bahrain has no migration in 2014. Turkey, Sudan and Eritrea have jumps in % change.

Traces

Data for Bahrain is “0” – comes from SeriesForecastNetMigrationUNPD.

Comparison to other models

Both UNPD and World Bank use UNPD migration forecast data, the same we use.

Degenerative test

CDR, CBR, migration, and population make sense.[7]

Total Fertility Rate (TFR)

Face validity

Looks good.

Comparison to other models

IFs forecasts global TFR in 2050 to be 2.0851. The World Bank forecasts global TFR in 2050 to be 2.2. The UNPD forecasts global TFR in 2050 to be 2.2. See spreadsheet for country specific comparisons.

Degenerative test

Population, infant mortality, and contraception use all make sense.[8]

Economic module

The key variables we looked at in the economic module are: GDP (MER), GDP per capita (PPP), and value added (for all sectors).

GDP (MER)

Face validity

Big drop for Libya in 2011 and then jumps around a bit. Another drop in Libya 2022. Yemen and South Sudan also have pretty drastic variations over recent history due to conflict.
Traces

All of the issues raised in face validity tests (both history and forecast) were reflective of the data – we take IMF GDP until 2021. The drop in Libya in 2022 is a potential issue.

GDP per capita (PPP)

Face validity

GDP per capita for Equatorial Guinea is over 192 thousand USD by 2100, the highest in the world.
Traces

Equatorial Guinea is initialized correctly.

Value Added

Face validity

Because VADD is forced to reconcile with GDP, the historical values are often quite different from the value in the initial year.

Agriculture

The biggest changes (in percentage points) in the initial year (2014) for VADD(agriculture) are: Qatar, Kuwait, Bahrain, Trinidad, and UAE.

Manufactures

The biggest changes (in percentage points) in the initial year (2014) for VADD(manufactures) are: Micronesia, Timor-Leste, Gabon, Kuwait, and Hong Kong.

Services

The biggest changes (in percentage points) in the initial year (2014) for VADD(services) are: Puerto Rico, Republic of Congo, Chad, Gabon, and Azerbaijan.

The data for ICT value added is scarce and VADD(energy) and VADD(materials) have no historical analogs.

Health Module

We looked at all types of death and life expectancy as the key variables.

Face validity

Traffic

Looks good

Communicable disease (other)

Small spike in South Sudan in 2016. Spike in Sierra Leone in 2015. Increase in Equatorial Guinea in 2021.

Non-Communicable Disease (NCD)

Steep incline in Hong Kong and Taiwan.

Diabetes

Kosovo jumps around especially after 2090. Mauritius much higher than other island nations.

Malignant neoplasm

Large increase in Zimbabwe. Large increase in Qatar and UAE for men.

Cardiovascular

Spike in 2015 in Ukraine. Spike in 2015 in Belarus. Large increase then decrease for men in UAE.

Digestive

Spike in 2015 for men in Sierra Leone. Moldova very high and peak in 2051.

Respiratory

Spike in 2015 in Hong Kong and Taiwan for females. Kosovo much higher than other countries with a lot of jumping around.

Diarrhea

Spike in 2015 in Yemen and Sierra Leone. Jump in 2015 and 2016 for Equatorial Guinea.

Malaria

Spike in 2016 in South Sudan. Spike in 2015 in Sierra Leone. Spike in 2017 and then decrease in Chad. Increase until 2021 in Equatorial Guinea – almost all other countries decrease.

Respiratory infections

Spike in 2015 in Yemen. Spike in 2016 in South Sudan. Spike in 2015 in Sierra Leone. Large increase then steady decrease in Japan, Singapore, and Thailand. Spike in 2017 in Chad. Increase until 2021 in Equatorial Guinea. Decrease then an increase until 2053 in Seychelles.

Intended injuries

Countries with high levels of intended injuries (Syria, El Salvador, and Guatemala) maintain these high levels or even increase until 2100.

HIV/AIDS

Some countries increase – this “double bubble” is something we have been meaning to look into.

Life expectancy

Drop in 2015 in Yemen, Sierra Leone, Hong Kong, Taiwan, Equatorial Guinea.

Traces

Life expectancy for Malawi, Sierra Leone, Equatorial Guinea, Hong Kong, and Yemen is initialized higher than both the 2015 value (data) and the 2015 forecasted value. This is why it looks like there is a drop in 2015 for these countries.

Comparison to other models

[See spreadsheet attached]

Overall, our initial death rates and forecasts are often very different from the data and forecast from the WHO Global Burden of Disease forecast.[9]

Extreme conditions test

Tests were run by increasing mortality rates for all diseases (hlmortm). The model behaves as expected – deaths increase dramatically and then decrease as the stock of people is diminished.

Sensitivity analysis

Sensitivity analysis was conducted on all types of disease and the model behaved as expected i.e. no large increases or decreases (see spreadsheet attached).

Government consumption

We looked at government spending by destination (GDS), government consumption (GOVCON), government expenditures (GOVEXP), government revenues (GOVREV), and government to household transfers (GOVHHTRN).

Government spending by destination (GDS)

Face validity

Bangladesh military spending in the first year. Micronesian health spending looks weird. Most countries have no R&D spending data. Bhutan has a large jump in infrastructure forecast. Many countries do not have infrastructure spending data (same with ‘other’ spending).

Traces

Bangladesh military spending data (GovtMil%GDPWDI) has 0.001 for both 2015 and 2014 but we initialize at 1.29.

Government consumption (GOVCON)

Face validity

Timor-Leste has large drop in 2014. Countries with big spike in initial year: Puerto Rico, Tajikistan, CAR, Panama, Jamaica, Cyprus, Iran, Guinea Bissau, El Salvador, Vietnam, Chad, Somalia, Afghanistan, Cambodia.

Traces

The following countries seem to have problems with the initialization: Timor-Leste, Puerto Rico, Tajikistan, CAR, Panama, Jamaica, Cyprus, Iran, Guinea Bissau, El Salvador, Vietnam, Chad, Somalia, Afghanistan, and Cambodia. See spreadsheet for details.

Government expenditures (GovExp)

Face validity

Jumps in the initial year in Estonia, Japan, Lithuania, and Canada. Liberia looks strange in 2020.

Traces

The following countries have issues with the initialization: Estonia, Canada, Japan, Lithuania, and Liberia.

Government revenue (GovRev)

Missing data for many countries. Yemen initialized at 0 in 2014. Liberia and Micronesia behave strangely.

Traces

The following countries seem to have issues with the initialization: Congo, Slovenia, Yemen, Liberia, and Micronesia.

Government to household transfers (GOVHHTRN)

Yemen, Equatorial Guinea, Sierra Leone, and Spain initial year looks like a jump. Singapore has a jump in 2026.

Traces

Post 2014 there are jumps in Singapore, Yemen, Equatorial Guinea, Sierra Leone, and Spain.

Education module

The variables we looked at were mean years of education for 15+ age group (EDYRSAG15), gender parity (mean education years 25+), educational enrollment (all levels), and education cost (total, as a percent of GDP).

Years of education for 15+

Face validity

Looks good.

Traces

Mozambique is initialized high and increases quite dramatically until 2100.

Educational enrolment (all levels)

Face validity

Secondary enrolment for Australia, Belgium, and Finland looks strange. Tertiary for Australia looks strange.

Traces

Secondary enrolment initialization issues all seem to be that they are initialized lower than the latest year of data (2013). Tertiary is also initialized low.

Expected years of education (EDYRSSLE)

Face validity

Looks strange for Australia, Belgium, China, Finalnd, New Zealand, Malaysia, Ireland, Syria, Spain, Seychelles, South Africa, Thailand, UK, and the world.

Traces

Most seem to be initialized lower than the historical value (EdExpectedYearsofSchooling) but some are high. See spreadsheet for details.

Education cost

Face validity

Solomon Islands, Bhutan, Congo, Equatorial Guinea all look weird.

Extreme conditions test

Were performed and model behaved well – see spreadsheet for details.

Sensitivity analysis

Were performed and model behaved well – see spreadsheet for details.

Agriculture

The variables we looked at were agricultural production (AGP), agricultural demand (AGDEM), land (LD), yield (YL), ag exports (AGX), ag imports (AGM), and calories per capita (CLPC).

Agricultural production

Face validity

The following countries have discrepancies in the first year: Russia, Kazakhstan, UAE (Crop) Montenegro, Palestine, Ethiopia (meat), China (Fish) Many jumps in total category: Kazakhstan, Russia, Cambodia, Ukraine, Gambia, UAE, Timor-Leste, Senegal, Maldives, Singapore, Qatar, Bahrain

Also, AGP (fish) does not have a historical analog attached to it so it gives an error when clicked.

Traces

See spreadsheet for details.

Agricultural demand (AGDEM)

Face validity

Austria, Oman, and South Sudan have spikes.

Land (LD)

Face validity

Crop land looks strange for: Lebanon, Sudan, New Zealand, Jordan, Tanzania, Botswana, Lesotho, and Cuba. Forest land looks weird for: Tonga and Comoros. ‘Other’ land looks weird for: China. Total land looks weird for: Burundi, Nigeria, Rwanda, Sierra Leone, Uruguay, and Austria.

Traces

See spreadsheet

Yield (YL)

Looks strange for Maldives, Russia, Kazakstan, UAE, and Djibouti.

Ag exports (AGX)

Brazilian crop exports have some spikes. Indian meat looks weird and Chinese fish looks weird.

Ag imports (AGM)

Crop looks strange in China, Nigeria, Tonga, Kosovo, Malawi, Zambia, Comoros, Equatorial Guinea, and Brazil. Fish looks strange in: Nigeria and China. Total looks strange in China and Nigeria as well.

Calories per capita (CLPC)

Sierra Leone, Somalia, and Chad behave very strangely. Angola has some initial year discrepancies.

Next steps

From the operational validation exercises, issues have been identified and we have begun to diagnose these issues as either conceptual model issues or computerized model issues. This was not a fully comprehensive validation – there are still modules that need to be validated: infrastructure, socio-political, and energy. We also need to identify additional models to compare IFs to.

 

Conceptual model issues

  1. We preference mortality tables over CDRs
  2. Death rate forecasts are very different from GBD
  3. “Double bubble” of AIDs deaths
  4. Equatorial Guinea GDP per capita gets very high by 2100

Computerized model issues

  1. VarLinks in IFsVar needs to be updated – the “drivers” display contains parameters that no longer exist
  2. AGP (fish) does not have a historical analog attached to it so it gives an error when clicked.
  3. Some GDS and VADD dimensions do not have historical analogs – they give errors when clicked

Issues yet to be diagnosed

  1. All of the transients in the initial year identified throughout this document

    </ol>

    • There are transients in almost every module but VADD sticks out because there are likely conceptual model issues that need to be addressed i.e. that energy value added is manufacturing less materials
    • All “strange’ behaviour in later years in the forecasts (see above)


    [1] World Bank Population Estimates and Projections http://data.worldbank.org/data-catalog/population-projection-tables Accessed September 1st, 2016.

    [2] UNPD World Population Prospects, 2015 revision https://esa.un.org/unpd/wpp/DataQuery/ Accessed September 1st, 2016.

    [3] World Bank Population Estimates and Projections http://data.worldbank.org/data-catalog/population-projection-tables Accessed September 1st, 2016.

    [4] UNPD World Population Prospects, 2015 revision https://esa.un.org/unpd/wpp/DataQuery/ Accessed September 1st, 2016.

    [5] CBR degenerative test method: Picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data.

    [6] CDR degenerative test method: Picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data

    [7] Migration degenerative test method: Picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data

    [8] TFR degenerative test method: picked Africa, Europe, Americas, MidEast, Asia and Pacific, Oceania and compared driver data with variable data.

    [9] WHO Projections of mortality and burden of disease < http://www.who.int/healthinfo/global_burden_disease/projections2004/en/> accessed September 1st, 2016.