Word Cloud LinksAnimation (3) CSP (2) Data (22) Data Science (3) Distributions (1) Dust (7) Economics (14) Engineering (9) Equipment Venders (1) Faster R (1) GDAL (5) Germany (1) ggplot2 (19) GIS (5) Irradiance (17) Kuwait (1) LaTeX (11) Linux (2) Meteorology (16) Misc Tricks (3) Modeling (6) Natural Gas (2) Nuclear (1) O&M (2) Projects (5) Project Valuations (6) Qatar RE (13) R Colors (9) R Data Import (6) R Data Objects (16) R Data Syntax (6) Renewable Energy (14) RE Policy (7) Resource Assessment (19) R Graphics (19) R Packages (3) R Programming (20) Saudi Arabia (2) Scientific Computing (1) Solar (35) Spatial Analysis (6) Storage (1) UAE (3) Ubuntu (1) Website (2) WECC (2) Wind (5)
Category Archives: R Data Import
Quantitative analysis depends on the ability to load and manage many different types of data and file formats. There are many R data import functions. Some functions ship with base R and others can be found in R packages.
Data Available in R
R is pre-installed with many data sets in the datasets package, which is included in the base distribution of R. R datasets are automatically loaded when the application is started. A list of all data sets in the package is obtained using the following command:
A new method to extract data tables from PDF files is introduced. Most of the data scraping tools available are browser-based. The common tools are also manual in nature and limited to one table at a time. A solution is outlined to extract multiple tables at once. The solution combines the R programming language with the open-source Java program Tabula. The result is a convenient method that transforms documents into databases.
The ability to train a machine to extract data tables from PDF files has several benefits:
Aerosol Optical Depth (AOD) defines the degree to which aerosols prevent the transmission of sunlight by absorption or scattering. AOD is measured using an integrated extinction coefficient over a vertical column of air. The extinction coefficient can be used to analyze solar extinction and the performance of solar power systems as a function of location and time.
The maptools package has a pruneMap() function t0 crop map objects in R. In practice, the function extracts data from SpatialPolygon or SpatialLine objects given a boundary box or specific area of interest. Unfortunately, there is no equivalent function for high resolution, large data, raster images, which are common in many Earth Science applications. The following post defines a custom function to crop raster images in R and to extract data from SpatialGridDataFrames. The function is tested using a raster image from the Shuttle Radar Topography Mission (SRTM; shown at left). The resulting data is then mapped using the image() function in R.
The standard way to read text files into R is to use the read.table() command. However, many users struggle with time delays when loading large data sets. An alternative command that offers significant speed improvements is fread(), or fast read, which can found in the data.table package. The following code loads a tab delimited file with a million elements and reveals that fread() reduces load time by almost 99%, as confirmed by the benchmark performance stats at left. The function is still under development, but it is available for download and doesn’t suffer from stability issues. Instead, expect argument structure and command syntax to change over time.
There are many reasons to work with binary data in R. Solar resource data, solar PV performance data, and real-time grid monitoring data are typically stored and transmitted in binary data formats.
In practice, the ability to access binary data in R is impossible in the absence of a vender or format specific “can opener” and a properly configured scientific programming environment. As a result, many business applications often bypass binary data use altogether or, instead, rely on secondary sources and summary statistics with no ability to validate data integrity and accuracy.