EVE Ref Docs

# Downloading Datasets

To download all the data, you can use wget (opens new window). The only real problem with wget is that it does not support multiple concurrent downloads.

For instance, to download the market orders for a particular year, you can use the following command:

wget -r -np -nc -nv --domains=data.everef.net -R index.html https://data.everef.net/market-orders/history/2023/
  • -r enables recursive retrieval, where wget will examine the HTML and resolve relative links.
  • -np prevents ascending to the parent directory when retrieving recursively.
  • -nc prevents downloading files that already exist locally.
  • -nv prevents printing of the download progress.
  • --domains=data.everef.net restricts the download to the data.everef.net domain.
  • -R index.html prevents downloading of the index.html files.
  • Do not use -N to enforce timestamp checking. The current host doesn't handle the Last-Modified header properly and this will result in all the files being redownloaded on every run.

# Wget on Docker

If you don't have wget installed, you can use the mwendler/wget (opens new window) Docker image. This image hasn't been updated in a while, but it still works.

docker run --rm -v $(pwd):/data -w /data mwendler/wget -r -np -nc -nv --domains=data.everef.net -R index.html --no-check-certificate https://data.everef.net/market-orders/history/2023/

← Fuzzwork Ordersets →