# EVE Ref EVE Online Deep Archive
The Deep Archive (opens new window) archives EVE Online data from data.everef.net (opens new window) into torrents.
# Published Torrents
Below is a list of published torrents.
ℹ️ To help seed this historical EVE Online data, add the RSS feed (opens new window) to your torrent client and configure it for automatic downloads.
# Structure
Each archive is one torrent file, named everef-eve-online-deep-archive-NN.torrent, where NN is a sequential number.
The files are laid out exactly as they were originally on data.everef.net.
When downloading multiple torrents, store them in a custom location and make the torrent client to not create a subfolder. This will make the entire archive stay in the same directory and be easily accessible. In qBittorrent, for instance, this is done by setting "Torrent content layout" to "Don't create subfolder" and works on both normal downloads and RSS download rules. If you don't do this, each torrent archive will be in different folders, but will still work just fine.
Text files named everef-eve-online-deep-archive-NN.txt contain a list of all the files in each torrent file and can
be used to search for specific content.
Alongside the torrent files are minisign (opens new window) signatures of the torrent
files themselves, named everef-eve-online-deep-archive-NN.torrent.minisig.
See below for more on signatures.
# Downloading more than one archive into the same folder
No file path is ever repeated across two of these torrents, and every torrent lays its files out exactly as they
were on data.everef.net. That means every Deep Archive torrent can be downloaded into the same folder, with no
separate subfolder per torrent, and they all fit together into one tree matching the site.
In qBittorrent, choose "Don't create subfolder for torrents with multiple files" when adding each torrent (or set
it as the default for new torrents). In a client without that option, move each everef-eve-online-deep-archive-NN/
folder's contents up a level by hand once it's finished downloading. This is how EVE Ref keeps its own copy of
everything archived so far, in one directory.
# Index snapshot
Each torrent includes everef-eve-online-deep-archive-NN-index.tar.xz: a snapshot, taken when the torrent was built,
of the index.json file from every data.everef.net folder it archives a file from. Those files are rewritten by
the site's own indexer every time it runs, so they're never part of the torrent's own data files; this tarball is
the copy that doesn't change, and it's covered by the .sha256 checksum file like everything else. Extract it with
tar xJf everef-eve-online-deep-archive-NN-index.tar.xz.
# Signature Verification
There are two things to verify: the .torrent file itself and the individual files inside the torrent after downloading.
Torrent files have their own internal hashes, which guards against corruption.
The signatures can be used to verify that the data did indeed come from EVE Ref originally and has not been tampered with.
The public key for both is published at everef.net/.well-known/minisign.pub (opens new window).
# Verifying the .torrent file
curl -O https://data.everef.net/deep-archive/everef-eve-online-deep-archive-NN.torrent
curl -O https://data.everef.net/deep-archive/everef-eve-online-deep-archive-NN.torrent.minisig
curl -O https://everef.net/.well-known/minisign.pub
minisign -V -p minisign.pub -m everef-eve-online-deep-archive-NN.torrent
Or via Docker:
docker run --rm -v "$PWD:/data" jedisct1/minisign -Vm /data/everef-eve-online-deep-archive-NN.torrent -P RWTxrmJ0f1feiHhWrnI/WUEf/18UaRb2aFgif8+y2Q8YR1JGa1pj4juk
# Verifying the downloaded files
Once the torrent has downloaded, the SHA-256 checksum file everef-eve-online-deep-archive-NN.sha256 and signature file
everef-eve-online-deep-archive-NN.sha256.minisig can be used to double-check the content.
See the readme at everef-eve-online-deep-archive-NN-README.md for instructions.
# What gets archived
Every file not yet archived, oldest first, once it's at least 2 days old.
Once a torrent is published, those files will eventually be removed from data.everef.net.
Once deleted, the torrent and its seeders become the only remaining source.
Each file is only ever in one torrent.
For datasets with History Rollups, the yearly rollup archives are what gets archived,
not the daily files they replace.
The daily files of market-orders/, public-contracts/,
esi-scrape/ (opens new window), reference-data/ (opens new window)
and hoboleaks-sde/ (opens new window) are never rolled up, so they are archived as they are.
Archiving is done on an ad-hoc basis without a set schedule. Whenever a new archive has been prepared, it will be added to everef-eve-online-deep-archive.xml (opens new window).
# What doesn't get archived
market-history/andkillmails/, since these effectively function as databases where old files are updated in-place.- Daily
history/<year>/files of datasets that are rolled up, see History Rollups. Datasets that haven't been rolled up yet are only archived once they are. wars/history/(opens new window) for now.- Any
*-latest*or*-current*files, which are updated in-place.
# Tools
- qBittorrent (opens new window) - free bittorrent client
- webtor.io (opens new window) can be used to download individual files in the archives