International Energy Agency (IEA): Difference between revisions

From Pardee Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= About =
The '''International Energy Agency commenced''' its operation in 1974 under the auspices of the Organization for Economic Co-operation and Development (OECD). The IEA is the energy forum for 26 Member countries, all from the OECD, to improve the world’s energy supply and to promote reliable databases for energy-related information. IEA member governments are committed to sharing energy information, to co-ordinating their energy policies and to co-operating in the development of rational energy programs. IEA publishes monthly reports on electricity, natural gas, prices, and the oil market. The [http://www.worldenergyoutlook.org/publications/ ''World Energy Outlook''] is the IEA's most comprehensive publication, and is considerd " the world’s most authoritative source of energy market analysis and projections."
The '''International Energy Agency commenced''' its operation in 1974 under the auspices of the Organization for Economic Co-operation and Development (OECD). The IEA is the energy forum for 26 Member countries, all from the OECD, to improve the world’s energy supply and to promote reliable databases for energy-related information. IEA member governments are committed to sharing energy information, to co-ordinating their energy policies and to co-operating in the development of rational energy programs. IEA publishes monthly reports on electricity, natural gas, prices, and the oil market. The [http://www.worldenergyoutlook.org/publications/ ''World Energy Outlook''] is the IEA's most comprehensive publication, and is considerd " the world’s most authoritative source of energy market analysis and projections."


The main IEA sources used by IFs are the '''''World Energy Balances (WEB)&nbsp;'''''<b>and </b>'''''World Energy Statistics (WES)''''','''&nbsp;'''databases assocated with the ''World Energy Outlook.'' They contain variables such as the production, trade, and consumption of coal, oil, gas, electricity, heat, renewables, and waste for OECD countries and over 100 non-OECD countries.
The main IEA sources used by IFs are the '''''World Energy Balances (WEB)&nbsp;'''''<b>and </b>'''''World Energy Statistics (WES)''''','''&nbsp;'''databases assocated with the ''World Energy Outlook.'' They contain variables such as the production, trade, and consumption of coal, oil, gas, electricity, heat, renewables, and waste for OECD countries and over 100 non-OECD countries.


= Metadata =
= Data Acquisition =
 
== Data Acquisition ==


Unlike most data used in IFs, IEA data from the WEB and WES are not open source. The data must be purchased from the IEA and is delivered on two CD-ROMs.[http://www.iea.org/bookshop/730-World_Energy_Statistics_and_Balances_2016 [1]]&nbsp;Each disc runs on a database management software program called Beyond 20/20 that comes loaded on the discs, along with the data. Financial support for the purchase of the IEA data is available from a University Library Association grant. Pardee successfully applied for grant funding through this program for the 2017 update with the help of staff at&nbsp;the Anderson Academic Commons at the University of Denver.
Unlike most data used in IFs, IEA data from the WEB and WES are not open source. The data must be purchased from the IEA and is delivered on two CD-ROMs.[http://www.iea.org/bookshop/730-World_Energy_Statistics_and_Balances_2016 [1]]&nbsp;Each disc runs on a database management software program called Beyond 20/20 that comes loaded on the discs, along with the data. Financial support for the purchase of the IEA data is available from a University Library Association grant. Pardee successfully applied for grant funding through this program for the 2017 update with the help of staff at&nbsp;the Anderson Academic Commons at the University of Denver.


== Documentation ==
= Documentation =


Full documentation is available for each dataset detailing its contents, structure, definitions, geographical coverage, etc.&nbsp;
Full documentation is available for each dataset detailing its contents, structure, definitions, geographical coverage, etc.&nbsp;
Line 19: Line 15:
[http://wds.iea.org/wds/pdf/WORLDBES_Documentation.pdf World&nbsp;Energy Statistics 2016 Database Documentation]
[http://wds.iea.org/wds/pdf/WORLDBES_Documentation.pdf World&nbsp;Energy Statistics 2016 Database Documentation]


== Batch Pull ==
= Batch Pull =


The IEA update is performed as a Batch Pull that includes '''138 series''' using the Batch Data Update feature in IFs. In the display, "IEA" is the Source, "IEA Countries" are used for country concordance. For the Code Location in Source Data portion of the update form, the source Excel must be formatted so that each Code in Source term is in a different column. For example, SeriesEnImportsOilProductsIEA's Code in Sourse is&nbsp;"Imports.Oil products," so there should be two columns with "Imports" in one and "Oil products" in the next.
The IEA update is performed as a batch pull that includes '''138 series''' using the Batch Data Update feature in IFs. In the display, "IEA" is the Source and the&nbsp;"IEA Countries" country list is used for country concordance. For the Code Location in Source Data portion of the update form, the source Excel must be formatted so that each Code in Source term is in a different column. For example, SeriesEnImportsOilProductsIEA's Code in Source is&nbsp;"Imports.Oil products," so there should be two columns with "Imports" in one and "Oil products" in the next.


== Series Codes ==
= Series Codes =


Because this is a batch pull, each series needs a Code in addition to a variable name to be imported. The codes are listed in the DataDict, and should match (or be extremely close to) the series name in the IEA source database. For the 2017 update, series on the WEB and WES discs were organized by two parameters, FLOW and PRODUCT, where FLOW is the first term of the Code and PRODUCT is the second term of the Code. Drag and drop FLOW, PRODUCT, UNIT,&nbsp;and COUNTRY to column and row headers to reconfigure the display in Beyond 20/20.
Because this is a batch pull, each series needs a Code in addition to a variable name to be imported. The Codes are listed in the DataDict, and should match&nbsp;the series name in the IEA source database. For the 2017 update, series on the WEB and WES discs were organized by two parameters, FLOW and PRODUCT, where FLOW is the first term of the Code and PRODUCT is the second term of the Code. [NOTE: Drag and drop FLOW, PRODUCT, UNIT,&nbsp;and COUNTRY to column and row headers to reconfigure the display in Beyond 20/20.]


'''Example'''
'''Example'''
Line 39: Line 35:
[[File:IEA Beyond 2020 screenshot.png|800px|IEA Beyond 2020 screenshot.png]]
[[File:IEA Beyond 2020 screenshot.png|800px|IEA Beyond 2020 screenshot.png]]


== Pulling and Formatting the Data ==
= Pulling and Formatting the Data =


=== Pulling ===
== Pulling ==


Ideally, the IEA series are pulled from the discs in bulk. As of the 2017 update, Beyond 20/20 limits the number of records that can be exported at one time, preventing all the data from being exported at once; however, it is possible to pull all records by PRODUCT&nbsp;or FLOW. For example, it is possible to pull all natural gas or oil products series &nbsp;at once from the WEB disc. Pull the data by assembling the correct configuration in Beyond 20/20, then going to File>Save As and saving it as a .xls file. It is also possible to copy and past individual (or a few) series into an Excel by highlighting them, but not a large number of series at once. [NOTE: The directly exported Excel file did not work on Pardee computers due to a policy setting issue, so the data&nbsp;had to be copied and pasted into a new sheet to be manipulated]
Ideally, the IEA series are pulled from the discs in bulk. As of the 2017 update, Beyond 20/20 limits the number of records that can be exported at one time, preventing all the data from being exported at once; however, it is possible to pull all records by PRODUCT&nbsp;or FLOW. For example, it is possible to pull all natural gas or oil products series&nbsp;at once from the WEB disc. Pull the data by assembling the correct configuration in Beyond 20/20, then going to File>Save As and saving it as a .xls file. It is also possible to copy and past individual (or a few) series into an Excel by highlighting them, but not a large number of series at once. [NOTE: The directly exported Excel file did not work on Pardee computers due to a policy setting issue, so the data&nbsp;had to be copied and pasted into a new sheet to be manipulated]


[[File:IEA Beyond 2020 screenshot 2.png|800px|IEA Beyond 2020 screenshot 2.png]]
[[File:IEA Beyond 2020 screenshot 2.png|800px|IEA Beyond 2020 screenshot 2.png]]


=== Formatting ===
== Formatting ==


Once the series have been pulled by PRODUCT or FLOW, they need to be cleaned. Missing data are marked with a "..", "x", or "n," so remove these with a Find+Replace. The data can then be imported with the Batch Import tool using the codes assocated with the FLOW and PRODUCT, along with the country name. It may be necessary to also clean the codes, for example if there is a unit listed parenthetically after the name of the product (i.e. "Solar thermal (TJ-net)").
Once the series have been pulled by PRODUCT or FLOW, they need to be cleaned. Missing data are marked with a "..", "x", or "n," so remove these with a Find+Replace. The data can then be imported with the Batch Import tool using the Codes assocated with the FLOW and PRODUCT, along with the country name. It may be necessary to also clean the Codes, for example if there is a unit listed parenthetically after the name of the product (i.e. "Solar thermal (TJ-net)").


[[File:IEA Excel ex.png|800px|IEA Excel ex.png]]
[[File:IEA Excel ex.png|800px|IEA Excel ex.png]]


== Conversions ==
= Conversions =


Most of the necessary conversion for these series are done automatically through the Batch Import form; however, there 22 series that as of the 2017 update had to be converted by hand from terajoules (TJ) to kilotones of energy equivalent (ktoe). The conversion factor used for these series is from the&nbsp;[https://www.iea.org/statistics/resources/unitconverter/ IEA Unit Converter]:&nbsp;TJ value*0.0238845897. These series include:
Most of the necessary conversion for these series are done automatically through the Batch Import form. However, 22 series had to be converted manually for the the 2017 update from terajoules (TJ) to kilotones of energy equivalent (ktoe). The conversion factor used for these series is from the&nbsp;[https://www.iea.org/statistics/resources/unitconverter/ IEA Unit Converter]:&nbsp;TJ value*0.0238845897. These series include:


{| border="0" cellpadding="0" cellspacing="0" width="320"
{| border="0" cellpadding="0" cellspacing="0" width="320"
|- height="20"
|-
| height="20" width="320" | EnConBiogasIndustrialIEA
| height="20" width="320" | EnConBiogasIndustrialIEA
|- height="20"
|-
| height="20" | EnConBiogasOtherIEA
| height="20" | EnConBiogasOtherIEA
|- height="20"
|-
| height="20" | EnConBiogasTotIEA
| height="20" | EnConBiogasTotIEA
|- height="20"
|-
| height="20" | EnConBiomassIndustrialIEA
| height="20" | EnConBiomassIndustrialIEA
|- height="20"
|-
| height="20" | EnConBiomassOtherIEA
| height="20" | EnConBiomassOtherIEA
|- height="20"
|-
| height="20" | EnConBiomassResidentialIEA
| height="20" | EnConBiomassResidentialIEA
|- height="20"
|-
| height="20" | EnConBiomassTotIEA
| height="20" | EnConBiomassTotIEA
|- height="20"
|-
| height="20" | EnConBiomassTransportIEA
| height="20" | EnConBiomassTransportIEA
|- height="20"
|-
| height="20" | EnConSolarThermalTotIEA
| height="20" | EnConSolarThermalTotIEA
|- height="20"
|-
| height="20" | EnConSolarThermIndustrialIEA
| height="20" | EnConSolarThermIndustrialIEA
|- height="20"
|-
| height="20" | EnConSolarThermOtherIEA
| height="20" | EnConSolarThermOtherIEA
|- height="20"
|-
| height="20" | EnConSolarThermResidentialIEA
| height="20" | EnConSolarThermResidentialIEA
|- height="20"
|-
| height="20" | EnExportsBiomassIEA
| height="20" | EnExportsBiomassIEA
|- height="20"
|-
| height="20" | EnExpProdBiogasCDIEA
| height="20" | EnExpProdBiogasCDIEA
|- height="20"
|-
| height="20" | EnExpProdIndustrialWasteCDIEA
| height="20" | EnExpProdIndustrialWasteCDIEA
|- height="20"
|-
| height="20" | EnExpProdMunicipalWasteNonRenewableCDIEA
| height="20" | EnExpProdMunicipalWasteNonRenewableCDIEA
|- height="20"
|-
| height="20" | EnExpProdMunicipalWasteRenewableCDIEA
| height="20" | EnExpProdMunicipalWasteRenewableCDIEA
|- height="20"
|-
| height="20" | EnExpProdnonspecPrimaryBiomassWasteCDIEA
| height="20" | EnExpProdnonspecPrimaryBiomassWasteCDIEA
|- height="20"
|-
| height="20" | EnExpProdSolarThermalCDIEA
| height="20" | EnExpProdSolarThermalCDIEA
|- height="20"
|-
| height="20" | EnImportsBiomassIEA
| height="20" | EnImportsBiomassIEA
|- height="20"
|-
| height="20" | EnProdBiomassIEA
| height="20" | EnProdBiomassIEA
|- height="20"
|-
| height="20" | EnExpProdnonspecPrimaryBiomassWasteCDIEA
| height="20" | EnExpProdnonspecPrimaryBiomassWasteCDIEA
|}
|}


== DataDict'''<br/>Variable: '''<span style="font-size: 13px;">Variable names were not changed from previous years this data was pulled, and no additional variables were added for the 2017 pull.</span>DataDict ==
= DataDict =
 
'''Variable: '''<span style="font-size: 13px;">Variable names were not changed from previous years this data was pulled, and no additional variables were added for the 2017 pull.</span>


'''Table:&nbsp;'''These were not changed from previous years this data was pulled.&nbsp;
'''Table:&nbsp;'''These were not changed from previous years this data was pulled.&nbsp;


'''Code in Source:&nbsp;'''Identifies the series for batch import; were not changed from previous years this data was pulled.
'''Code in Source:&nbsp;'''These were not changed from previous years this data was pulled.


'''Groups:&nbsp;'''These were not changed from previous years this data was pulled.&nbsp;
'''Groups:&nbsp;'''These were not changed from previous years this data was pulled.&nbsp;
Line 116: Line 114:
'''Definitions and Units:'''&nbsp;These&nbsp;were not changed from previous years this data was pulled.
'''Definitions and Units:'''&nbsp;These&nbsp;were not changed from previous years this data was pulled.


'''Extended Source Defn: '''All were marked as "No Extended Source" for 2017 pull.&nbsp;Units used in this dataset include BBOE and GwHr.
'''Extended Source Defn: '''All were marked as "No Extended Source" for 2017 pull.&nbsp;
 
'''Units:'''&nbsp;These were not changed from previous years; the units used in this dataset are BBOE and GwHr.


'''Years:&nbsp;'''Years for every series were changed to available data provided through the database. Most series begin&nbsp;in 1960 with the exception of EnExportsOilIEA, EnImportsOiliIEA, and EnProdOilIEA, which begin in 1971. All extend&nbsp;through either 2014 or 2015 as of the 2017 update.
'''Years:&nbsp;'''Years for every series were changed to available data provided through the database. Most series begin&nbsp;in 1960 with the exception of EnExportsOilIEA, EnImportsOiliIEA, and EnProdOilIEA, which begin in 1971. All extend&nbsp;through either 2014 or 2015 as of the 2017 update.


'''Source:'''&nbsp;The source used in the 2017 update for all batch pull IEA series is "IEA (International Energy Agency) Batch Pull."
'''Source:'''&nbsp;The source name used in the 2017 update for all batch pull IEA series is "IEA (International Energy Agency) Batch Pull."


'''Original Source:'''&nbsp;The original source used in the 2017 update for all series is the "World Energy Outlook."
'''Original Source:'''&nbsp;The original source used in the 2017 update for all series is the "World Energy Outlook."
Line 130: Line 130:
'''Disaggregation:&nbsp;'''Disaggregations were not changed from previous updates.
'''Disaggregation:&nbsp;'''Disaggregations were not changed from previous updates.


'''Name in source:'''&nbsp;Was updated based on the name of each variable as it is displayed in the Beyond 20/20 database format; generally matches the Code in Source.
'''Name in source:'''&nbsp;Names were updated based on the name of each variable as it is displayed in the Beyond 20/20 database format; generally matches the Code in Source.


'''Decimal places:&nbsp;'''These&nbsp;were not changed from previous years this data was pulled.
'''Decimal places:&nbsp;'''These&nbsp;were not changed from previous years this data was pulled.


'''Country Concordance:&nbsp;'''FAO Countries were used.
'''Country Concordance:&nbsp;'''IEA Countries were used (and updated for the 2017 pull, see below).


'''Formula:&nbsp;'''These&nbsp;were not changed from previous years this data was pulled, and are either blank or convert data from ktoe to BBOE or GwH
'''Formula:&nbsp;'''These&nbsp;were not changed from previous years this data was pulled, and are either blank or convert data to BBOE or GwHr.


== Preprocessor Series ==
= Preprocessor Series =


Of the 138&nbsp;IEA Batch Pull series, 24 are preprocessor. These series should be updated first in any IEA batch update. They include:
Of the 138&nbsp;IEA Batch Pull series, 24 are preprocessor. These series should be first priority in any IEA batch update. They include:


{| border="0" cellpadding="0" cellspacing="0" width="175"
{| border="0" cellpadding="0" cellspacing="0" width="175"
Line 193: Line 193:
|}
|}


== Non-Preprocessor Series ==
= Non-Preprocessor Series =


There are 114 nonpreprocessor series included in the IEA Batch Pull. They include:
There are 114 nonpreprocessor series included in the IEA update. They include:


{| border="0" cellpadding="0" cellspacing="0" width="64"
{| border="0" cellpadding="0" cellspacing="0" width="64"
|- height="20"
|-
| height="20" width="64" | EnConBiodieselTotIEA
| height="20" width="64" | EnConBiodieselTotIEA
|- height="20"
|-
| height="20" | EnConBiodieselTransportIEA
| height="20" | EnConBiodieselTransportIEA
|- height="20"
|-
| height="20" | EnConBiogasIndustrialIEA
| height="20" | EnConBiogasIndustrialIEA
|- height="20"
|-
| height="20" | EnConBiogasolineTotIEA
| height="20" | EnConBiogasolineTotIEA
|- height="20"
|-
| height="20" | EnConBiogasolineTransportIEA
| height="20" | EnConBiogasolineTransportIEA
|- height="20"
|-
| height="20" | EnConBiogasOtherIEA
| height="20" | EnConBiogasOtherIEA
|- height="20"
|-
| height="20" | EnConBiogasTotIEA
| height="20" | EnConBiogasTotIEA
|- height="20"
|-
| height="20" | EnConBiomassIndustrialIEA
| height="20" | EnConBiomassIndustrialIEA
|- height="20"
|-
| height="20" | EnConBiomassOtherIEA
| height="20" | EnConBiomassOtherIEA
|- height="20"
|-
| height="20" | EnConBiomassResidentialIEA
| height="20" | EnConBiomassResidentialIEA
|- height="20"
|-
| height="20" | EnConBiomassTotIEA
| height="20" | EnConBiomassTotIEA
|- height="20"
|-
| height="20" | EnConBiomassTransportIEA
| height="20" | EnConBiomassTransportIEA
|- height="20"
|-
| height="20" | EnConCoalIndustrialIEA
| height="20" | EnConCoalIndustrialIEA
|- height="20"
|-
| height="20" | EnConCoalOtherIEA
| height="20" | EnConCoalOtherIEA
|- height="20"
|-
| height="20" | EnConCoalResidentialIEA
| height="20" | EnConCoalResidentialIEA
|- height="20"
|-
| height="20" | EnConCoalTotIEA
| height="20" | EnConCoalTotIEA
|- height="20"
|-
| height="20" | EnConCoalTransportIEA
| height="20" | EnConCoalTransportIEA
|- height="20"
|-
| height="20" | EnConCombustRenewWasteIndustrialIEA
| height="20" | EnConCombustRenewWasteIndustrialIEA
|- height="20"
|-
| height="20" | EnConCombustRenewWasteOtherIEA
| height="20" | EnConCombustRenewWasteOtherIEA
|- height="20"
|-
| height="20" | EnConCombustRenewWasteResidentialIEA
| height="20" | EnConCombustRenewWasteResidentialIEA
|- height="20"
|-
| height="20" | EnConCombustRenewWasteTotIEA
| height="20" | EnConCombustRenewWasteTotIEA
|- height="20"
|-
| height="20" | EnConCombustRenewWasteTransportIEA
| height="20" | EnConCombustRenewWasteTransportIEA
|- height="20"
|-
| height="20" | EnConElecIndustrialIEA
| height="20" | EnConElecIndustrialIEA
|- height="20"
|-
| height="20" | EnConElecOtherIEA
| height="20" | EnConElecOtherIEA
|- height="20"
|-
| height="20" | EnConElecResidentIEA
| height="20" | EnConElecResidentIEA
|- height="20"
|-
| height="20" | EnConElecTotIEA
| height="20" | EnConElecTotIEA
|- height="20"
|-
| height="20" | EnConElecTransportIEA
| height="20" | EnConElecTransportIEA
|- height="20"
|-
| height="20" | EnConGeothermIndustrialIEA
| height="20" | EnConGeothermIndustrialIEA
|- height="20"
|-
| height="20" | EnConGeothermOtherIEA
| height="20" | EnConGeothermOtherIEA
|- height="20"
|-
| height="20" | EnConGeothermResidentialIEA
| height="20" | EnConGeothermResidentialIEA
|- height="20"
|-
| height="20" | EnConGeothermTotIEA
| height="20" | EnConGeothermTotIEA
|- height="20"
|-
| height="20" | EnConNatGasIndustrialIEA
| height="20" | EnConNatGasIndustrialIEA
|- height="20"
|-
| height="20" | EnConNatGasOtherIEA
| height="20" | EnConNatGasOtherIEA
|- height="20"
|-
| height="20" | EnConNatGasResidentialIEA
| height="20" | EnConNatGasResidentialIEA
|- height="20"
|-
| height="20" | EnConNatGasTotIEA
| height="20" | EnConNatGasTotIEA
|- height="20"
|-
| height="20" | EnConNatGasTransportIEA
| height="20" | EnConNatGasTransportIEA
|- height="20"
|-
| height="20" | EnConOtherBiofuelsIndustrialIEA
| height="20" | EnConOtherBiofuelsIndustrialIEA
|- height="20"
|-
| height="20" | EnConOtherBiofuelsTotIEA
| height="20" | EnConOtherBiofuelsTotIEA
|- height="20"
|-
| height="20" | EnConOtherBiofuelsTransportIEA
| height="20" | EnConOtherBiofuelsTransportIEA
|- height="20"
|-
| height="20" | EnConSolarThermalTotIEA
| height="20" | EnConSolarThermalTotIEA
|- height="20"
|-
| height="20" | EnConSolarThermIndustrialIEA
| height="20" | EnConSolarThermIndustrialIEA
|- height="20"
|-
| height="20" | EnConSolarThermOtherIEA
| height="20" | EnConSolarThermOtherIEA
|- height="20"
|-
| height="20" | EnConSolarThermResidentialIEA
| height="20" | EnConSolarThermResidentialIEA
|- height="20"
|-
| height="20" | EnExportsBiodieselIEA
| height="20" | EnExportsBiodieselIEA
|- height="20"
|-
| height="20" | EnExportsBiogasolineIEA
| height="20" | EnExportsBiogasolineIEA
|- height="20"
|-
| height="20" | EnExportsBiomassIEA
| height="20" | EnExportsBiomassIEA
|- height="20"
|-
| height="20" | EnExportsCombustRenewWasteIEA
| height="20" | EnExportsCombustRenewWasteIEA
|- height="20"
|-
| height="20" | EnExportsElecGwHrIEA
| height="20" | EnExportsElecGwHrIEA
|- height="20"
|-
| height="20" | EnExportsElecIEA
| height="20" | EnExportsElecIEA
|- height="20"
|-
| height="20" | EnExportsOtherBiofuelsIEA
| height="20" | EnExportsOtherBiofuelsIEA
|- height="20"
|-
| height="20" | EnExpProdBioDieselsCDIEA
| height="20" | EnExpProdBioDieselsCDIEA
|- height="20"
|-
| height="20" | EnExpProdBiogasCDIEA
| height="20" | EnExpProdBiogasCDIEA
|- height="20"
|-
| height="20" | EnExpProdBiogasolineCDIEA
| height="20" | EnExpProdBiogasolineCDIEA
|- height="20"
|-
| height="20" | EnExpProdCharcoalCDIEA
| height="20" | EnExpProdCharcoalCDIEA
|- height="20"
|-
| height="20" | EnExpProdIndustrialWasteCDIEA
| height="20" | EnExpProdIndustrialWasteCDIEA
|- height="20"
|-
| height="20" | EnExpProdMunicipalWasteNonRenewableCDIEA
| height="20" | EnExpProdMunicipalWasteNonRenewableCDIEA
|- height="20"
|-
| height="20" | EnExpProdMunicipalWasteRenewableCDIEA
| height="20" | EnExpProdMunicipalWasteRenewableCDIEA
|- height="20"
|-
| height="20" | EnExpProdnonspecPrimaryBiomassWasteCDIEA
| height="20" | EnExpProdnonspecPrimaryBiomassWasteCDIEA
|- height="20"
|-
| height="20" | EnExpProdOtherLiquidbiofuelsCDIEA
| height="20" | EnExpProdOtherLiquidbiofuelsCDIEA
|- height="20"
|-
| height="20" | EnExpProdOtherSourcesCDIEA
| height="20" | EnExpProdOtherSourcesCDIEA
|- height="20"
|-
| height="20" | EnExpProdPrimarySolidGasCDIEA
| height="20" | EnExpProdPrimarySolidGasCDIEA
|- height="20"
|-
| height="20" | EnExpProdSolarThermalCDIEA
| height="20" | EnExpProdSolarThermalCDIEA
|- height="20"
|-
| height="20" | EnExpProdSPVCDIEA
| height="20" | EnExpProdSPVCDIEA
|- height="20"
|-
| height="20" | EnExpProdTideWaveOCeanCDIEA
| height="20" | EnExpProdTideWaveOCeanCDIEA
|- height="20"
|-
| height="20" | EnExpProdWindCDIEA
| height="20" | EnExpProdWindCDIEA
|- height="20"
|-
| height="20" | EnImportsBiodieselIEA
| height="20" | EnImportsBiodieselIEA
|- height="20"
|-
| height="20" | EnImportsBiogasolineIEA
| height="20" | EnImportsBiogasolineIEA
|- height="20"
|-
| height="20" | EnImportsBiomassIEA
| height="20" | EnImportsBiomassIEA
|- height="20"
|-
| height="20" | EnImportsElecGwHrIEA
| height="20" | EnImportsElecGwHrIEA
|- height="20"
|-
| height="20" | EnImportsElecIEA
| height="20" | EnImportsElecIEA
|- height="20"
|-
| height="20" | EnImportsOtherBiofuelsIEA
| height="20" | EnImportsOtherBiofuelsIEA
|- height="20"
|-
| height="20" | EnOutputElecCoalCDIEA
| height="20" | EnOutputElecCoalCDIEA
|- height="20"
|-
| height="20" | EnOutputElecCombustibleRenewableWasteCDIEA
| height="20" | EnOutputElecCombustibleRenewableWasteCDIEA
|- height="20"
|-
| height="20" | EnOutputElecCrudeNGLFeedstocksCDIEA
| height="20" | EnOutputElecCrudeNGLFeedstocksCDIEA
|- height="20"
|-
| height="20" | EnOutputElecElectricityCDIEA
| height="20" | EnOutputElecElectricityCDIEA
|- height="20"
|-
| height="20" | EnOutputElecGasCDIEA
| height="20" | EnOutputElecGasCDIEA
|- height="20"
|-
| height="20" | EnOutputElecGeothermalCDIEA
| height="20" | EnOutputElecGeothermalCDIEA
|- height="20"
|-
| height="20" | EnOutputElecHeatCDIEA
| height="20" | EnOutputElecHeatCDIEA
|- height="20"
|-
| height="20" | EnOutputElecHydroCDIEA
| height="20" | EnOutputElecHydroCDIEA
|- height="20"
|-
| height="20" | EnOutputElecNuclearCDIEA
| height="20" | EnOutputElecNuclearCDIEA
|- height="20"
|-
| height="20" | EnOutputElecOilProductsCDIEA
| height="20" | EnOutputElecOilProductsCDIEA
|- height="20"
|-
| height="20" | EnOutputElecPeatCDIEA
| height="20" | EnOutputElecPeatCDIEA
|- height="20"
|-
| height="20" | EnOutputElecSolarWindOtherCDIEA
| height="20" | EnOutputElecSolarWindOtherCDIEA
|- height="20"
|-
| height="20" | EnOutputElecTotalCDIEA
| height="20" | EnOutputElecTotalCDIEA
|- height="20"
|-
| height="20" | EnProdBiogasolineIEA
| height="20" | EnProdBiogasolineIEA
|- height="20"
|-
| height="20" | EnProdBiomassIEA
| height="20" | EnProdBiomassIEA
|- height="20"
|-
| height="20" | EnProdCoalCDIEA
| height="20" | EnProdCoalCDIEA
|- height="20"
|-
| height="20" | EnProdCombustibleRenewableWasteCDIEA
| height="20" | EnProdCombustibleRenewableWasteCDIEA
|- height="20"
|-
| height="20" | EnProdCombustRenewWasteIEA
| height="20" | EnProdCombustRenewWasteIEA
|- height="20"
|-
| height="20" | EnProdCrudeNGLFeedstocksCDIEA
| height="20" | EnProdCrudeNGLFeedstocksCDIEA
|- height="20"
|-
| height="20" | EnProdElectricityCDIEA
| height="20" | EnProdElectricityCDIEA
|- height="20"
|-
| height="20" | EnProdGasCDIEA
| height="20" | EnProdGasCDIEA
|- height="20"
|-
| height="20" | EnProdGeothermalCDIEA
| height="20" | EnProdGeothermalCDIEA
|- height="20"
|-
| height="20" | EnProdHeatCDIEA
| height="20" | EnProdHeatCDIEA
|- height="20"
|-
| height="20" | EnProdHydroCDIEA
| height="20" | EnProdHydroCDIEA
|- height="20"
|-
| height="20" | EnProdNuclearCDIEA
| height="20" | EnProdNuclearCDIEA
|- height="20"
|-
| height="20" | EnProdOilProductsCDIEA
| height="20" | EnProdOilProductsCDIEA
|- height="20"
|-
| height="20" | EnProdOtherBiofuelsIEA
| height="20" | EnProdOtherBiofuelsIEA
|- height="20"
|-
| height="20" | EnProdPeatCDIEA
| height="20" | EnProdPeatCDIEA
|- height="20"
|-
| height="20" | EnProdSolarWindOtherCDIEA
| height="20" | EnProdSolarWindOtherCDIEA
|- height="20"
|-
| height="20" | EnProdTotalCDIEA
| height="20" | EnProdTotalCDIEA
|- height="20"
|-
| height="20" | EnTPESCoalCDIEA
| height="20" | EnTPESCoalCDIEA
|- height="20"
|-
| height="20" | EnTPESCombustibleRenewableWasteCDIEA
| height="20" | EnTPESCombustibleRenewableWasteCDIEA
|- height="20"
|-
| height="20" | EnTPESCrudeNGLFeedstocksCDIEA
| height="20" | EnTPESCrudeNGLFeedstocksCDIEA
|- height="20"
|-
| height="20" | EnTPESElectricityCDIEA
| height="20" | EnTPESElectricityCDIEA
|- height="20"
|-
| height="20" | EnTPESGasCDIEA
| height="20" | EnTPESGasCDIEA
|- height="20"
|-
| height="20" | EnTPESGeothermalCDIEA
| height="20" | EnTPESGeothermalCDIEA
|- height="20"
|-
| height="20" | EnTPESHeatCDIEA
| height="20" | EnTPESHeatCDIEA
|- height="20"
|-
| height="20" | EnTPESHydroCDIEA
| height="20" | EnTPESHydroCDIEA
|- height="20"
|-
| height="20" | EnTPESNuclearCDIEA
| height="20" | EnTPESNuclearCDIEA
|- height="20"
|-
| height="20" | EnTPESOilProductsCDIEA
| height="20" | EnTPESOilProductsCDIEA
|- height="20"
|-
| height="20" | EnTPESPeatCDIEA
| height="20" | EnTPESPeatCDIEA
|- height="20"
|-
| height="20" | EnTPESSolarWindOtherCDIEA
| height="20" | EnTPESSolarWindOtherCDIEA
|- height="20"
|-
| height="20" | EnTPESTotalCDIEA
| height="20" | EnTPESTotalCDIEA
|}
|}


== Country Concordance ==
= Country Concordance =


There is a unique country concordance table for IEA series called "IEA Countries." This table should be checked for accuracy at the time of each update. For the 2017 update, the following countries had to be updated (to the form listed):
There is a unique country concordance table for the IEA series called "IEA Countries." This table should be checked for accuracy at the time of each update. For the 2017 update, the following countries had to be updated (to the form listed):


{| border="0" cellpadding="0" cellspacing="0" width="279"
{| border="0" cellpadding="0" cellspacing="0" width="279"

Latest revision as of 23:54, 10 September 2018

The International Energy Agency commenced its operation in 1974 under the auspices of the Organization for Economic Co-operation and Development (OECD). The IEA is the energy forum for 26 Member countries, all from the OECD, to improve the world’s energy supply and to promote reliable databases for energy-related information. IEA member governments are committed to sharing energy information, to co-ordinating their energy policies and to co-operating in the development of rational energy programs. IEA publishes monthly reports on electricity, natural gas, prices, and the oil market. The World Energy Outlook is the IEA's most comprehensive publication, and is considerd " the world’s most authoritative source of energy market analysis and projections."

The main IEA sources used by IFs are the World Energy Balances (WEB) and World Energy Statistics (WES), databases assocated with the World Energy Outlook. They contain variables such as the production, trade, and consumption of coal, oil, gas, electricity, heat, renewables, and waste for OECD countries and over 100 non-OECD countries.

Data Acquisition

Unlike most data used in IFs, IEA data from the WEB and WES are not open source. The data must be purchased from the IEA and is delivered on two CD-ROMs.[1] Each disc runs on a database management software program called Beyond 20/20 that comes loaded on the discs, along with the data. Financial support for the purchase of the IEA data is available from a University Library Association grant. Pardee successfully applied for grant funding through this program for the 2017 update with the help of staff at the Anderson Academic Commons at the University of Denver.

Documentation

Full documentation is available for each dataset detailing its contents, structure, definitions, geographical coverage, etc. 

World Energy Balances 2016 Database Documentation

World Energy Statistics 2016 Database Documentation

Batch Pull

The IEA update is performed as a batch pull that includes 138 series using the Batch Data Update feature in IFs. In the display, "IEA" is the Source and the "IEA Countries" country list is used for country concordance. For the Code Location in Source Data portion of the update form, the source Excel must be formatted so that each Code in Source term is in a different column. For example, SeriesEnImportsOilProductsIEA's Code in Source is "Imports.Oil products," so there should be two columns with "Imports" in one and "Oil products" in the next.

Series Codes

Because this is a batch pull, each series needs a Code in addition to a variable name to be imported. The Codes are listed in the DataDict, and should match the series name in the IEA source database. For the 2017 update, series on the WEB and WES discs were organized by two parameters, FLOW and PRODUCT, where FLOW is the first term of the Code and PRODUCT is the second term of the Code. [NOTE: Drag and drop FLOW, PRODUCT, UNIT, and COUNTRY to column and row headers to reconfigure the display in Beyond 20/20.]

Example

Variable: EnImportsOilProductsIEA

Code: Imports.Oil products

FLOW: Imports

PRODUCT: Oil products

IEA Beyond 2020 screenshot.png

Pulling and Formatting the Data

Pulling

Ideally, the IEA series are pulled from the discs in bulk. As of the 2017 update, Beyond 20/20 limits the number of records that can be exported at one time, preventing all the data from being exported at once; however, it is possible to pull all records by PRODUCT or FLOW. For example, it is possible to pull all natural gas or oil products series at once from the WEB disc. Pull the data by assembling the correct configuration in Beyond 20/20, then going to File>Save As and saving it as a .xls file. It is also possible to copy and past individual (or a few) series into an Excel by highlighting them, but not a large number of series at once. [NOTE: The directly exported Excel file did not work on Pardee computers due to a policy setting issue, so the data had to be copied and pasted into a new sheet to be manipulated]

IEA Beyond 2020 screenshot 2.png

Formatting

Once the series have been pulled by PRODUCT or FLOW, they need to be cleaned. Missing data are marked with a "..", "x", or "n," so remove these with a Find+Replace. The data can then be imported with the Batch Import tool using the Codes assocated with the FLOW and PRODUCT, along with the country name. It may be necessary to also clean the Codes, for example if there is a unit listed parenthetically after the name of the product (i.e. "Solar thermal (TJ-net)").

IEA Excel ex.png

Conversions

Most of the necessary conversion for these series are done automatically through the Batch Import form. However, 22 series had to be converted manually for the the 2017 update from terajoules (TJ) to kilotones of energy equivalent (ktoe). The conversion factor used for these series is from the IEA Unit Converter: TJ value*0.0238845897. These series include:

EnConBiogasIndustrialIEA
EnConBiogasOtherIEA
EnConBiogasTotIEA
EnConBiomassIndustrialIEA
EnConBiomassOtherIEA
EnConBiomassResidentialIEA
EnConBiomassTotIEA
EnConBiomassTransportIEA
EnConSolarThermalTotIEA
EnConSolarThermIndustrialIEA
EnConSolarThermOtherIEA
EnConSolarThermResidentialIEA
EnExportsBiomassIEA
EnExpProdBiogasCDIEA
EnExpProdIndustrialWasteCDIEA
EnExpProdMunicipalWasteNonRenewableCDIEA
EnExpProdMunicipalWasteRenewableCDIEA
EnExpProdnonspecPrimaryBiomassWasteCDIEA
EnExpProdSolarThermalCDIEA
EnImportsBiomassIEA
EnProdBiomassIEA
EnExpProdnonspecPrimaryBiomassWasteCDIEA

DataDict

Variable: Variable names were not changed from previous years this data was pulled, and no additional variables were added for the 2017 pull.

Table: These were not changed from previous years this data was pulled. 

Code in Source: These were not changed from previous years this data was pulled.

Groups: These were not changed from previous years this data was pulled. 

Subgroups: These were not changed form previoys years this data was pulled.

Definitions and Units: These were not changed from previous years this data was pulled.

Extended Source Defn: All were marked as "No Extended Source" for 2017 pull. 

Units: These were not changed from previous years; the units used in this dataset are BBOE and GwHr.

Years: Years for every series were changed to available data provided through the database. Most series begin in 1960 with the exception of EnExportsOilIEA, EnImportsOiliIEA, and EnProdOilIEA, which begin in 1971. All extend through either 2014 or 2015 as of the 2017 update.

Source: The source name used in the 2017 update for all batch pull IEA series is "IEA (International Energy Agency) Batch Pull."

Original Source: The original source used in the 2017 update for all series is the "World Energy Outlook."

Notes: Notes were updated to reflect the source disc for the series (WEB or WES), any conversion factors used, and the appropriate initials.

Aggregation: Aggregations were not changed from pervious updates.

Disaggregation: Disaggregations were not changed from previous updates.

Name in source: Names were updated based on the name of each variable as it is displayed in the Beyond 20/20 database format; generally matches the Code in Source.

Decimal places: These were not changed from previous years this data was pulled.

Country Concordance: IEA Countries were used (and updated for the 2017 pull, see below).

Formula: These were not changed from previous years this data was pulled, and are either blank or convert data to BBOE or GwHr.

Preprocessor Series

Of the 138 IEA Batch Pull series, 24 are preprocessor. These series should be first priority in any IEA batch update. They include:

EnExportsCoalIEA
EnExportsNatGasIEA
EnExportsOilIEA
EnExportsOilProductsIEA
EnExportsPeatIEA
EnExportsTotalIEA
EnImportsCoalIEA
EnImportsNatGasIEA
EnImportsOilIEA
EnImportsOilProductsIEA
EnImportsPeatIEA
EnImportsTotalIEA
EnProdBiodieselIEA
EnProdBiogasIEA
EnProdCoalIEA
EnProdGeothermIEA
EnProdHydroIEA
EnProdNatGasIEA
EnProdNuclearIEA
EnProdOilIEA
EnProdSolarPhotoIEA
EnProdSolarThermIEA
EnProdTideWaveOceanIEA
EnProdWindIEA

Non-Preprocessor Series

There are 114 nonpreprocessor series included in the IEA update. They include:

EnConBiodieselTotIEA
EnConBiodieselTransportIEA
EnConBiogasIndustrialIEA
EnConBiogasolineTotIEA
EnConBiogasolineTransportIEA
EnConBiogasOtherIEA
EnConBiogasTotIEA
EnConBiomassIndustrialIEA
EnConBiomassOtherIEA
EnConBiomassResidentialIEA
EnConBiomassTotIEA
EnConBiomassTransportIEA
EnConCoalIndustrialIEA
EnConCoalOtherIEA
EnConCoalResidentialIEA
EnConCoalTotIEA
EnConCoalTransportIEA
EnConCombustRenewWasteIndustrialIEA
EnConCombustRenewWasteOtherIEA
EnConCombustRenewWasteResidentialIEA
EnConCombustRenewWasteTotIEA
EnConCombustRenewWasteTransportIEA
EnConElecIndustrialIEA
EnConElecOtherIEA
EnConElecResidentIEA
EnConElecTotIEA
EnConElecTransportIEA
EnConGeothermIndustrialIEA
EnConGeothermOtherIEA
EnConGeothermResidentialIEA
EnConGeothermTotIEA
EnConNatGasIndustrialIEA
EnConNatGasOtherIEA
EnConNatGasResidentialIEA
EnConNatGasTotIEA
EnConNatGasTransportIEA
EnConOtherBiofuelsIndustrialIEA
EnConOtherBiofuelsTotIEA
EnConOtherBiofuelsTransportIEA
EnConSolarThermalTotIEA
EnConSolarThermIndustrialIEA
EnConSolarThermOtherIEA
EnConSolarThermResidentialIEA
EnExportsBiodieselIEA
EnExportsBiogasolineIEA
EnExportsBiomassIEA
EnExportsCombustRenewWasteIEA
EnExportsElecGwHrIEA
EnExportsElecIEA
EnExportsOtherBiofuelsIEA
EnExpProdBioDieselsCDIEA
EnExpProdBiogasCDIEA
EnExpProdBiogasolineCDIEA
EnExpProdCharcoalCDIEA
EnExpProdIndustrialWasteCDIEA
EnExpProdMunicipalWasteNonRenewableCDIEA
EnExpProdMunicipalWasteRenewableCDIEA
EnExpProdnonspecPrimaryBiomassWasteCDIEA
EnExpProdOtherLiquidbiofuelsCDIEA
EnExpProdOtherSourcesCDIEA
EnExpProdPrimarySolidGasCDIEA
EnExpProdSolarThermalCDIEA
EnExpProdSPVCDIEA
EnExpProdTideWaveOCeanCDIEA
EnExpProdWindCDIEA
EnImportsBiodieselIEA
EnImportsBiogasolineIEA
EnImportsBiomassIEA
EnImportsElecGwHrIEA
EnImportsElecIEA
EnImportsOtherBiofuelsIEA
EnOutputElecCoalCDIEA
EnOutputElecCombustibleRenewableWasteCDIEA
EnOutputElecCrudeNGLFeedstocksCDIEA
EnOutputElecElectricityCDIEA
EnOutputElecGasCDIEA
EnOutputElecGeothermalCDIEA
EnOutputElecHeatCDIEA
EnOutputElecHydroCDIEA
EnOutputElecNuclearCDIEA
EnOutputElecOilProductsCDIEA
EnOutputElecPeatCDIEA
EnOutputElecSolarWindOtherCDIEA
EnOutputElecTotalCDIEA
EnProdBiogasolineIEA
EnProdBiomassIEA
EnProdCoalCDIEA
EnProdCombustibleRenewableWasteCDIEA
EnProdCombustRenewWasteIEA
EnProdCrudeNGLFeedstocksCDIEA
EnProdElectricityCDIEA
EnProdGasCDIEA
EnProdGeothermalCDIEA
EnProdHeatCDIEA
EnProdHydroCDIEA
EnProdNuclearCDIEA
EnProdOilProductsCDIEA
EnProdOtherBiofuelsIEA
EnProdPeatCDIEA
EnProdSolarWindOtherCDIEA
EnProdTotalCDIEA
EnTPESCoalCDIEA
EnTPESCombustibleRenewableWasteCDIEA
EnTPESCrudeNGLFeedstocksCDIEA
EnTPESElectricityCDIEA
EnTPESGasCDIEA
EnTPESGeothermalCDIEA
EnTPESHeatCDIEA
EnTPESHydroCDIEA
EnTPESNuclearCDIEA
EnTPESOilProductsCDIEA
EnTPESPeatCDIEA
EnTPESSolarWindOtherCDIEA
EnTPESTotalCDIEA

Country Concordance

There is a unique country concordance table for the IEA series called "IEA Countries." This table should be checked for accuracy at the time of each update. For the 2017 update, the following countries had to be updated (to the form listed):

Bosnia and Herzegovina
Democratic Republic of the Congo
Hong Kong (China)
Democratic People's Republic of Korea
Libya
Moldova
Mauritius
Niger
Suriname
South Sudan
Tanzania
Viet Nam
Retrieved from "https://pardeewiki.du.edu//index.php?title=International_Energy_Agency_(IEA)&oldid=9165"