Vignette Rmd source code (Not on CRAN to reduce load on DWD server through daily new builds and checks of the vignette)
Interactive map vignette
use cases vignette
Links to the function help pages lead to 404 on CRAN, but work in locally opened vignettes.
The R package rdwd
contains code to select, download and read weather data from measuring stations across Germany. The German Weather Service (Deutscher Wetterdienst, DWD) provides thousands of datasets with weather observations through the FTP server online at
ftp://opendata.dwd.de/climate_environment/CDC/observations_germany/climate.
For data interpolated onto a 1 km raster, including radar data up to the last hour, see ftp://opendata.dwd.de/climate_environment/CDC/grids_germany/. Raster and binary reading functions for the Radolan data are included in rdwd
since May 2019.
For further details, please consult the DWD FTP server documentation
Please note that, as of June 2019, the old server at ftp://ftp-cdc.dwd.de will be phased out.
To use the observation datasets, rdwd
has been designed to mainly do 3 things:
selectDWD
: facilitate file selection, e.g. for certain station names (with findID
), by geographical location (see map and nearbyStations
), by temporal resolution (res = 1/10 minutes, hourly, daily, monthly, annual), variables (var = temperature, rain, wind, sun, clouds, etc) or observation period (per = historical long term records or the current year)
dataDWD
: download a file (or multiple files, without getting banned by the FTP-server)
readDWD
: read that data into R (including useful defaults for metadata)
selectDWD
uses the result from indexFTP
which recursively lists all the files on an FTP-server (using RCurl::getURL). As this is time consuming, the result is stored in the package dataset fileIndex
. From this, metaIndex
and geoIndex
are derived.
install.packages("rdwd")
# get the latest development version from github, incl. vignettes:
remotes::install_github("brry/rdwd", build_opts="--no-manual")
# For full usage, as needed in indexFTP and selectDWD(..., current=TRUE):
install.packages("RCurl") # is only suggested, not mandatory dependency
On Linux, instead of the last line above, use in the terminal (with lowercase rcurl):
sudo apt-get install r-cran-rcurl
If direct installation from CRAN doesn’t work, your R version might be too old. In that case you’ll need to update R.
library(rdwd)
link <- selectDWD("Potsdam", res="daily", var="kl", per="recent")
file <- dataDWD(link, read=FALSE, dir="DWDdata", quiet=TRUE, force=NA)
clim <- readDWD(file, varnames=TRUE)
str(clim)
## 'data.frame': 550 obs. of 19 variables:
## $ STATIONS_ID : int 3987 3987 3987 3987 3987 3987 3987 3987 3987 3987 ...
## $ MESS_DATUM : POSIXct, format: "2017-11-25" "2017-11-26" ...
## $ QN_3 : int 10 10 10 10 10 10 10 10 10 10 ...
## $ FX.Windspitze : num 9.9 11.5 14.7 12.3 8.2 6 5.5 5.2 10.9 13.9 ...
## $ FM.Windgeschwindigkeit : num 3.7 5.2 6 5.2 3.4 2.3 2.5 1.9 4.5 6 ...
## $ QN_4 : int 3 3 3 3 3 3 3 3 3 3 ...
## $ RSK.Niederschlagshoehe : num 16.1 0.2 3.6 0 0.2 0 0 0 0.8 2.1 ...
## $ RSKF.Niederschlagsform : int 6 6 6 6 6 6 6 0 8 8 ...
## $ SDK.Sonnenscheindauer : num 0 2.72 2.53 1.42 6.62 ...
## $ SHK_TAG.Schneehoehe : int 0 0 0 0 0 0 0 0 0 0 ...
## $ NM.Bedeckungsgrad : num 7.2 5.6 6.7 7.2 5.3 6 7.3 7.2 6 6 ...
## $ VPM.Dampfdruck : num 8 6.8 7 7.7 6.3 6.1 6.2 6.6 5.7 7 ...
## $ PM.Luftdruck : num 998 1005 1002 992 994 ...
## $ TMK.Lufttemperatur : num 3.8 2.7 3.5 5.4 2.1 0.9 1 1.7 -0.4 2.9 ...
## $ UPM.Relative_Feuchte : num 99.2 92.3 89.6 86 89 ...
## $ TXK.Lufttemperatur_Max : num 5.5 5 5.2 6.9 5.6 4.2 3.5 3.1 2.4 5.1 ...
## $ TNK.Lufttemperatur_Min : num 1.6 -0.1 0.9 3.8 -0.3 -1.3 -1.3 -1.8 -2.5 0.8 ...
## $ TGK.Lufttemperatur_5cm_min: num -0.3 -2.2 -1.4 2.6 -2.2 -3.5 -3.2 -3.7 -5.3 -1.7 ...
## $ eor : Factor w/ 1 level "eor": 1 1 1 1 1 1 1 1 1 1 ...
Recent temperature time series:
par(mar=c(4,4,2,0.5), mgp=c(2.7, 0.8, 0), cex=0.8)
plot(clim[,c(2,14)], type="l", xaxt="n", las=1, main="Daily temp Potsdam")
berryFunctions::monthAxis() ; abline(h=0)
mtext("Source: Deutscher Wetterdienst", adj=-0.1, line=0.5, font=3)
Long term climate graph:
link <- selectDWD("Goettingen", res="monthly", var="kl", per="h")
clim <- dataDWD(link, quiet=TRUE)
clim$month <- substr(clim$MESS_DATUM_BEGINN,5,6)
temp <- tapply(clim$MO_TT, clim$month, mean, na.rm=TRUE)
prec <- tapply(clim$MO_RR, clim$month, mean, na.rm=TRUE)
berryFunctions::climateGraph(temp, prec, main="Goettingen")
mtext("Source: Deutscher Wetterdienst", adj=-0.05, line=2.8, font=3)
Weather stations can be selected geographically with the interactive map. All stations within a certain radius around a given lat-long position can be obtained with nearbyStations.
The DWD station IDs can be obtained from station names with
## Potsdam
## 3987
## Warning: source -> withVisible -> eval -> eval -> createBerrysVignettes -
## > rmarkdown::render -> knitr::knit -> call_block -> block_exec -> in_dir -
## > evaluate -> evaluate::evaluate -> evaluate_call -> timing_fn -> handle -
## > findID: ID determined from name 'Koeln' has 4 elements (2665, 2666, 2667,
## 2968).
## Koeln-Bonn Koeln-Botanischer Garten Koeln-Porz-Eil
## 2667 2665 2666
## Koeln-Stammheim
## 2968
File selection by station name/id and folder happens with selectDWD
. It needs an index of all the available files on the server. The package contains such an index (fileIndex
) that is updated (at least) with each CRAN release of the package. The selectDWD documentation contains an overview of the FTP folder structure.
If you find the file index to be outdated (Error in download.file … : cannot open URL), please let me know and I will update it. Meanwhile, use current=TRUE in selectDWD
:
# all files at a given path, with current file index (RCurl required):
links <- selectDWD(res="monthly", var="more_precip", per="hist", current=TRUE)
fileIndex
is created with the function indexFTP
used in the last section of rdwd-package.R.
### This chunk is not evaluated ###
# recursively list files on the FTP-server:
files <- indexFTP("hourly/sun") # use dir="some_path" to save the output elsewhere
berryFunctions::headtail(files, 5, na=TRUE)
# indexFTP uses a folder to resume indexing after getting banned:
gridindex <- indexFTP("radolan","ftp://opendata.dwd.de/climate_environment/CDC/grids_germany/hourly")
gridindex <- indexFTP(gridindex,"ftp://opendata.dwd.de/climate_environment/CDC/grids_germany/hourly", sleep=1)
# with other FTP servers, this should also work...
funet <- indexFTP(base="ftp.funet.fi/pub/standards/w3/TR/xhtml11/", folder="")
p <- RCurl::getURL( "ftp.funet.fi/pub/standards/w3/TR/xhtml11/",
verbose=T, ftp.use.epsv=TRUE, dirlistonly=TRUE)
selectDWD
is designed to be very flexible:
# inputs can be vectorized, and period can be abbreviated:
selectDWD(c("Potsdam","Wuerzburg"), res="hourly", var="sun", per="hist")
## [[1]]
## [1] "ftp://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/hourly/sun/historical/ stundenwerte_SD_03987_18930101_20181231_hist.zip"
##
## [[2]]
## [1] "ftp://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/hourly/sun/historical/ stundenwerte_SD_05705_19510101_20181231_hist.zip"
If res/var/per are left NA, an interactive selection is opened with the available options for the given station.
# Time period can be doubled to get both filenames:
selectDWD("Potsdam", res="daily", var="kl", per="rh")
## [1] "ftp://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/ daily/kl/recent/tageswerte_KL_03987_akt.zip"
## [2] "ftp://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/ daily/kl/historical/tageswerte_KL_03987_18930101_20181231_hist.zip"
There may be a differing number of available files for several stations across all folders. That’s why the default outvec is FALSE (unless per="hr"
).
## [[1]]
## [1] "s_germany/climate/annual/more_precip/historical/jahreswerte_RR_03467_19930101_20181231_hist.zip"
## [2] "s_germany/climate/annual/more_precip/recent/jahreswerte_RR_03467_akt.zip"
## [3] "s_germany/climate/daily/more_precip/historical/tageswerte_RR_03467_19930601_20181231_hist.zip"
## [4] "s_germany/climate/daily/more_precip/recent/tageswerte_RR_03467_akt.zip"
## [5] "s_germany/climate/monthly/more_precip/historical/monatswerte_RR_03467_19930601_20171231_hist.zip"
## [6] "s_germany/climate/monthly/more_precip/historical/monatswerte_RR_03467_19930601_20181231_hist.zip"
## [7] "s_germany/climate/monthly/more_precip/recent/monatswerte_RR_03467_akt.zip"
##
## [[2]]
## [1] "s_germany/climate/annual/more_precip/historical/jahreswerte_RR_05116_19920101_20061231_hist.zip"
## [2] "s_germany/climate/daily/more_precip/historical/tageswerte_RR_05116_19920701_20061231_hist.zip"
## [3] "s_germany/climate/monthly/more_precip/historical/monatswerte_RR_05116_19920701_20061231_hist.zip"
selectDWD
also uses a complete data.frame with meta information, metaIndex
(derived from the “Beschreibung” files in fileIndex
).
## 'data.frame': 96609 obs. of 12 variables:
## $ Stations_id : int 1 1 1 1 1 ...
## $ von_datum : int 18910101 18910101 18910101 18910101 19120101 ...
## $ bis_datum : int 19860630 19860630 19860630 19860630 19860630 ...
## $ Stationshoehe: num 478 478 478 478 478 ...
## $ geoBreite : num 47.8 47.8 ...
## $ geoLaenge : num 8.85 8.85 ...
## $ Stationsname : chr "Aach" "Aach" ...
## $ Bundesland : chr "Baden-Wuerttemberg" "Baden-Wuerttemberg" ...
## $ res : chr "annual" "annual" ...
## $ var : chr "more_precip" "more_precip" ...
## $ per : chr "historical" "recent" ...
## $ hasfile : logi TRUE FALSE TRUE ...
dataDWD
can download (and readDWD
can correctly read) such a data.frame from any folder on the FTP server:
# file with station metadata for a given path:
m_link <- selectDWD(res="monthly", var="more_precip", per="hist", meta=TRUE)
substr(m_link, 50, 1e4) # (Monatswerte = monthly values, Beschreibung = description)
## [1] "ervations_germany/climate/monthly/more_precip/historical/RR_Monatswerte_Beschreibung_Stationen.txt"
Meta files may list stations for which there are actually no files. These refer to nonpublic datasets (The DWD cannot publish all datasets because of copyright restrictions). To request those, please contact cdc.daten@dwd.de or klima.vertrieb@dwd.de.
Any feedback on this package (or this vignette) is very welcome via github or berry-b@gmx.de!