Database files in IFs: Difference between revisions

From Pardee Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(34 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= IFs Historical Database Files =
= IFs Historical Database Files =


IFs uses Microsoft Access files to store data and data dictionary (meta-data). All data files are in the “IFs\Data” folder. Data and related files are listed below:
IFs uses Microsoft Access files to store data and data dictionary (meta-data). All data files are in the “C:/My Documents/Users/Public/IFs/Data” folder. Data and related files are listed below:
<ol style="list-style-type:lower-roman;">
 
<li><u>IFsHistSeries.Mdb</u> is the largest and most frequently used IFs data file containing more than three thousand data tables each containing 186 rows (one row of data per country) and several columns (one column per year). The figure below shows data from an IFsHistSeries table.</li>
*<u>IFsHistSeries.Mdb</u> is the largest and most frequently used IFs data file containing more than three thousand data tables each containing 186 rows (one row of data per country) and several columns (one column per year). The figure below shows data from an IFsHistSeries table
<li><u>DataDict.Mdb</u> is the data dictionary file with a table containing one row of meta-data (e.g., definition, unit, source, last date of update) for each of the data tables in IFsHistSeries.Mdb.</li>
 
<li><u>IFs.Mdb</u> is the Microsoft Access file that contains several IFs data tables. One of these table - "Country Translation" - is requied for automated IFs data import/update. Country Translation table maintains (and updates) a concordance list between country names used by IFs and data sources.</li>
[[File:IfsHistSeries.png|RTENOTITLE]]
<li><u>IFsWVSCohort.Mdb</u> is the file that contains data from waves of World Value Survey, a global survey of cultural values conducted by University of Michigan.</li>
 
<li><u>IFsDataImport.Mdb</u>, is an MS Access database that holds the data series imported using IFs software's automated '''single series ''''''import''' interface.</li>
*<u>DataDict.Mdb</u> is the data dictionary file with a table containing one row of meta-data (e.g., definition, unit, source, last date of update) for each of the data tables in IFsHistSeries.Mdb
<li><u>IFsDataImportBatch.Mdb</u>, is the Access database that houses the data series imported using IFs software's automated '''batch''' import interface.</li>
 
</ol>
[[File:DataDict.png|Screenshot from DataDict]]
 
*<u>IFs.Mdb</u> is the Microsoft Access file that contains several IFs data tables. One of these table - "Country Translation" - is requied for automated IFs data import/update. Country Translation table maintains (and updates) a concordance list between country names used by IFs and data sources
*<u>IFsWVSCohort.Mdb</u> is the file that contains data from waves of World Value Survey, a global survey of cultural values conducted by University of Michigan.
*<u>IFsDataImport.Mdb</u>, is an MS Access database that holds the data series imported using IFs software's automated '''single series ''''''import''' interface.
*<u>IFsDataImportBatch.Mdb</u>, is the Access database that houses the data series imported using IFs software's automated '''batch''' import interface.


= IFs Data Table Naming Convention =
= IFs Data Table Naming Convention =
Line 25: Line 30:




 
{| border="1" cellpadding="1" cellspacing="1" width="964"
 
|-
 
| height="20" width="197" | '''Column&nbsp;'''
 
| width="558" | '''Description'''
= dvsdfesfsdfcsd =
| width="209" | '''Example'''
 
|-
== ==
| height="40" | '''Variable Name'''
| width="558" | Name of variable you assign (Automatically generated from import)
| width="209" | WSSJMPSanitationRural%Improved
|-
| height="40" | '''Table'''
| width="558" | Automatically generated&nbsp;
| width="209" | SeriesWSSJMPSanitationRural%Improved
|-
| height="20" | '''Group'''
| width="558" | Group or category you assign to the variable (Automatically generated from import)
| width="209" | Infrastructure, Water, Health
|-
| height="20" | '''Subgroup'''
| width="558" | Subroup you assign to the variable&nbsp;(Automatically generated from import)
| width="209" | Sanitation
|-
| height="20" | '''Series'''
| width="558" | Yes
| width="209" | Yes
|-
| height="20" | '''CoVaTra'''
| width="558" | No
| width="209" | No
|-
| height="20" | '''Cohort'''
| width="558" | No
| width="209" | No
|-
| height="60" | '''Definition'''
| width="558" | Specific definition of the variable that will be displayed in IFs
| width="209" | Proportion of Rural population served with Improved Sanitation (%)
|-
| height="60" width="197" | '''Extended Source Definition'''
| width="558" | Additional information about the variable if needed
| width="209" | Likely to ensure hygenic separation of human excreta from human contact.
|-
| height="20" | '''Units'''
| width="558" | Specification of the units the variable is measured by
| width="209" | Percent;People;Degrees C
|-
| height="20" | '''Currency'''
| width="558" | Currency of the units if they're monetary
| width="209" | &nbsp;USD;Euro;Lira
|-
| height="20" | '''Years'''
| width="558" | Years that data set covers (Automatically generated but make sure its accurate)
| width="209" | 2000-2015
|-
| height="20" | '''Source'''
| width="558" | Source or website the data source is from
| width="209" | World Bank
|-
| height="20" width="197" | '''Original Source'''
| width="558" | URL of data source
| width="209" | [https://washdata.org/ https://washdata.org]
|-
| height="20" | '''Notes'''
| width="558" | Any additional notes and the initials of the puller and vetter of the data
| width="209" | Created individual index;EB;JD
|-
| height="20" | '''Last IFs Update'''
| width="558" | Automatically generated
| width="209" | 10/30/2017
|-
| height="40" | '''Aggregation'''
| width="558" | Tells the model how to aggregate country values to groups within the model.&nbsp;Will automatically generate if classified during import.
| width="209" | POP;GDP;LAND;AVG;SUM
|-
| height="40" | '''Disaggregation'''
| width="558" | Tells the model how to disaggregate country values to subregional classification&nbsp;
| width="209" | GDP
|-
| height="20" | '''TreatNullsAs0s'''
| width="558" | If checked, will treat nulls in the data set as zeroes
| width="209" | &nbsp;
|-
| height="20" | '''Proprietary'''
| width="558" | Usually data series are non-proprietary
| width="209" | &nbsp;
|-
| height="40" | '''Name in Source'''
| width="558" | Exact name of series in the source file and other nformation on how the data was pulled so someone can reimport it in the same way
| width="209" | World File;Sanitation;Sum of Rural Latrines, Septic Tanks, Sewer Connections
|-
| height="20" | '''Used in Preprocessor'''
| width="558" | Filled automatically when script is run before consolidation.&nbsp;<font>If checked, data set is used in preprocessor. Puller/Vetter not responsible for this.</font>
| width="209" | &nbsp;
|-
| height="40" | '''UsedInPreprocessorFileName'''
| width="558" | File name that is used in preprocessor that is filled automatically before consolidation. Puller/vetter not responsible for this.
| width="209" | &nbsp;
|-
| height="20" | '''CompareOtherForecasts'''
| width="558" | If series is a forecast from another source, this will be checked
| width="209" | &nbsp;
|-
| height="20" | '''Code in Source'''
| width="558" | Only used in batch pulls
| width="209" | &nbsp;
|-
| height="20" | '''Decimal Places'''
| width="558" | Decimal places for data in data series
| width="209" | 4
|-
| height="20" | '''Country Concordance'''
| width="558" | Country concordance used to import data into IFs. (Will automatically generate)
| width="209" | IFs Country
|-
| height="20" | '''Formula'''
| width="558" | Option during import to manipulate the data
| width="209" | *100 (to convert to %)
|}

Latest revision as of 18:46, 9 November 2017

IFs Historical Database Files

IFs uses Microsoft Access files to store data and data dictionary (meta-data). All data files are in the “C:/My Documents/Users/Public/IFs/Data” folder. Data and related files are listed below:

  • IFsHistSeries.Mdb is the largest and most frequently used IFs data file containing more than three thousand data tables each containing 186 rows (one row of data per country) and several columns (one column per year). The figure below shows data from an IFsHistSeries table

RTENOTITLE

  • DataDict.Mdb is the data dictionary file with a table containing one row of meta-data (e.g., definition, unit, source, last date of update) for each of the data tables in IFsHistSeries.Mdb

Screenshot from DataDict

  • IFs.Mdb is the Microsoft Access file that contains several IFs data tables. One of these table - "Country Translation" - is requied for automated IFs data import/update. Country Translation table maintains (and updates) a concordance list between country names used by IFs and data sources
  • IFsWVSCohort.Mdb is the file that contains data from waves of World Value Survey, a global survey of cultural values conducted by University of Michigan.
  • IFsDataImport.Mdb, is an MS Access database that holds the data series imported using IFs software's automated single series 'import' interface.
  • IFsDataImportBatch.Mdb, is the Access database that houses the data series imported using IFs software's automated batch import interface.

IFs Data Table Naming Convention

Names of all data tables in IFsHistSeries.MDB start with the prefix “Series”. The “Series” prefix is followed by an issue area prefix, e.g., “Ag” for agriculture or “Ed” for education. This second tier of prefix might be followed by additional prefixes (e.g., "EdSec" for secondary education) or might be absent altogether (e.g., in some of the earlier imports).

Data series names might also contain a suffix, the usual purpose of which is to differentiate among the sources for the same/similar series.

No spaces or symbols (other than %) are allowed in series name.

IFs DataDict Columns

The Datadict.mdb file serves as a reference for all series in the IFsHistSeries.mdb file. Every series in IFsHistSeries has an entry in DataDict containing all of the metadata on that series. The Data Dictionary lists each variable, the groups to which it belongs (e.g., Agriculture, Economics) its subgroup (e.g., Trade, Consumption), and additional identifying information. This information includes whether or not the data is a series (Yes/No), CoVaTrA, Cohort. It also includes a definition of the variable, and a column for an extended definition provided by the data source. The data dictionary has columns identifying the years for which a series has data, the source of the data, the original source (e.g. a series may have been pulled from the FAO website, but may have originated as World Bank research.) and the source name of the series, and an identifier for which team member last updated the series and when. It also includes instructions on how data should be aggregated or disaggregated for provincial models (e.g., by population or GDP distribution). Some additional information is supplied that is used by the model such as whether a datum of 0 should be treated as a null or as a zero, if a series is used in the preprocessor, if it is compared to other forecasts, the number of decimal places to read, and any formulas applied to the data.


Column  Description Example
Variable Name Name of variable you assign (Automatically generated from import) WSSJMPSanitationRural%Improved
Table Automatically generated  SeriesWSSJMPSanitationRural%Improved
Group Group or category you assign to the variable (Automatically generated from import) Infrastructure, Water, Health
Subgroup Subroup you assign to the variable (Automatically generated from import) Sanitation
Series Yes Yes
CoVaTra No No
Cohort No No
Definition Specific definition of the variable that will be displayed in IFs Proportion of Rural population served with Improved Sanitation (%)
Extended Source Definition Additional information about the variable if needed Likely to ensure hygenic separation of human excreta from human contact.
Units Specification of the units the variable is measured by Percent;People;Degrees C
Currency Currency of the units if they're monetary  USD;Euro;Lira
Years Years that data set covers (Automatically generated but make sure its accurate) 2000-2015
Source Source or website the data source is from World Bank
Original Source URL of data source https://washdata.org
Notes Any additional notes and the initials of the puller and vetter of the data Created individual index;EB;JD
Last IFs Update Automatically generated 10/30/2017
Aggregation Tells the model how to aggregate country values to groups within the model. Will automatically generate if classified during import. POP;GDP;LAND;AVG;SUM
Disaggregation Tells the model how to disaggregate country values to subregional classification  GDP
TreatNullsAs0s If checked, will treat nulls in the data set as zeroes  
Proprietary Usually data series are non-proprietary  
Name in Source Exact name of series in the source file and other nformation on how the data was pulled so someone can reimport it in the same way World File;Sanitation;Sum of Rural Latrines, Septic Tanks, Sewer Connections
Used in Preprocessor Filled automatically when script is run before consolidation. If checked, data set is used in preprocessor. Puller/Vetter not responsible for this.  
UsedInPreprocessorFileName File name that is used in preprocessor that is filled automatically before consolidation. Puller/vetter not responsible for this.  
CompareOtherForecasts If series is a forecast from another source, this will be checked  
Code in Source Only used in batch pulls  
Decimal Places Decimal places for data in data series 4
Country Concordance Country concordance used to import data into IFs. (Will automatically generate) IFs Country
Formula Option during import to manipulate the data *100 (to convert to %)