ShaCache
ShaCache is a caching tool that guarantees own and third party source code
relevant for deploying software is always available in the exact same version
it was once used and signed. ShaCache is used as both download and binary cache
within SlapOS. Download Cache allows to cache downloaded files similar to source
code on the internet. If the original website used to download the file is down,
the download cache will serve as backup while the Binary Cache allows to
download an already-compiled software release.
Why use ShaCache?
ShaCache (or libnetworkcached)
and its two corresponding ERP5 business templates ShaCache
and ShaDir
allow to utilize an ERP5 instance as self-certifying cache server using a NoSQL storage
and REST API. Files are archived using the HTTP POST method with their
SHA256 hash being set as url. When a client wants to download a specific file,
it just needs to provide the key value
(hash) using the GET method and the server will send a response with the file
data.
ShaCache allows to fetch files using their md5-checksum hashes offering a
self-certifying, secure and consistent way of loading content. The goal of
libnetworkcache python library is to abstract HTTP calls. It works as wrapper
of python httplib to use the Networkcache HTTP Server.
__________________
/ \
| |
------>| LIBNETWORKCACHED |
| ----| | <-----
| | \__________________/ |
GET /hash | | | POST / data
| | File |
__|__v______ ______|_____
| | | |
| Client | | Client |
|____________| |____________|
The Networkcache HTTP Server is sub-divided into two Web services:
- ShaCache - a simple HTTP server used to cache files.
- ShaDir - a simple HTTP Server used to cache information, working like a directory.
Getting Started
To run ShaCache server you need an ERP5 instance.
Setting up a ShaCache Server
If you want to run a dedicated ShaCache instance (like http://shacache.org/),
you can setup this up over SlapOS
and Vifib.
In case you are already running an ERP5 instance and just want to add ShaCache, the source
code can be found inside the ERP5 repository business templates
(ShaCache and
ShaDir).
Please refer to the the ERP5 documentation on installing business templates
on how to add ShaCache to your instance. A simple ShaCache client is available on
Gitlab.
Setting up a ShaCache Client
Clients are for uploading and downloading files to/from the ShaCache server
as described above. You can use Easy Install or pip
and just do:
easy_install slapos.libnetworkcache
#or
pip install slapos.libnetworkcache
and use the "networkcache-*" commands to upload/download files. See help
for all commands available
networkcache-download --help
networkcache-upload --help
API
Anyone can download files from ShaCache (with curl/http). Uploads require
being whitelisted as trusted source.
API
The API is straightforward:
PUT / :
parameter: file uploaded
Used to upload/modify an entry
GET /
Return raw content
Raise HTTP error (404) if key does not exist
Examples:
Usage Example
Following is a quick walkthrough of how ShaCache is being used in SlapOS itself to
load any of the softwares being used (OpenSSL in this case).
The buildout.cfg for OpenSSL
on the SlapOS Gitlab repository contains the location of the source file. This is the starting point (#19).
# OpenSSL - a toolkit implementing SSL v2/v3 and TLS v1 protocols as
# well as a full-strength general purpose cryptography
# library.
# http://www.openssl.org/
[buildout]
extends =
../ca-certificates/buildout.cfg
../coreutils/buildout.cfg
../patch/buildout.cfg
../perl/buildout.cfg
../zlib/buildout.cfg
parts =
openssl-output
[openssl]
recipe = slapos.recipe.cmmi
url = https://www.openssl.org/source/openssl-1.0.2k.tar.gz
md5sum = f965fc0bf01bf882b31314b61391ae65
location = ${buildout:parts-directory}/${:_buildout_section_name_}
Accessing ShaDir
buildout and slapos.libnetworkcache use the following
to create a source file's hash':
echo -n https://www.openssl.org/source/openssl-1.0.2k.tar.gz | md5sum -
With the hash, you can curl the ShaCache directory:
curl http://dir.shacache.org/slapos-buildout-c26dfe76c4128b780971d234206ab538
which should return a JSON file similar to this:
[
[
{
"urlmd5": "c26dfe76c4128b780971d234206ab538",
"sha512": "0d314b42352f4b1df2c40ca1094abc7e9ad684c5c35ea997efdd58204c70f22a1abcb17291820f0fff3769620a4e06906034203d31eb1a4d540df3e0db294016",
"file": "openssl-1.0.2k.tar.gz"
},
"SntZvR8Eu9xsEJKtuO3hjDl5MzNdUXB2uC7Lqa0j1jK/31sLSdPkUHlXy67EhfV66U7TXh5umdFX\nEO8Oq+E6Lo7yRDULVcyWCaTw+Qc7F9LHiBRHZeIvEfx5xfIt1Ldi3bYEm8PgW1vTBgrMkTBsgBJq\nVUUuf9Nuk3OaTWLrj0s=\n"
]
]
with the following response API:
[
[
{
"urlmd5": INDEXHASH,
"sha512": CONTENTHASH,
"file": FILENAME
},
USER_SSL_SIGNATURE
],
[...]
]
which means a user with signature USER_SSL_SIGNATURE uploaded the FILENAME containing the CONTENTHASH as Sha512 sum to the INDEXHASH. If 10 users uploaded
this file, there will be 10 entries returned in the list.
Downloading from ShaCache
On your client, simply call:
wget http://download.shacache.org/CONTENTHASH
which triggers download of the actual file using the Sha512 sum of its content
thus making sure no corrupted or inadverted files are downloaded as this would
create a different hash.
Trusted Sources
You can discover whitelisted sources on the ShaCache server by:
for entry, signature on [[ {"urlmd5": INDEXHASH, sha512: CONTENTHASH, file: FILENAME}, USER_SSL_SIGNATURE ], [...]]:
if signature in LIST_OF_SIGNATURES_I_TRUST:
return entry["sha512"]
LIST_OF_SIGNATURES_I_TRUST has to be defined in a config file, in the case of SlapOS
it is stored in SlapOS config (#195):
...
[networkcache]
download-cache-url = http://download.shacache.org/
download-dir-url = http://dir.shacache.org/
# signature certificates of the following uploaders.
# xxx
# yyy
# zzz
# aaa
# Test Agent (Automatic update from tests)
signature-certificate-list =
-----BEGIN CERTIFICATE-----
MIIB4DCCAUkCADANBgkqhkiG9w0BAQsFADA5MQswCQYDVQQGEwJGUjEZMBcGA1UE
CBMQRGVmYXVsdCBQcm92aW5
...
The signatures listed here are trusted. Upload attempts from sources other than
the ones listed will be ignored.
Upload to ShaCache
Uploading to ShaCache thus requires an SSL pair to sign a file before uploading. Should
two or more users upload the same sha512, shacache will calculate the sha512 for both, keep
two entries on ShaDir directory but only keep a single file. See the documentation
below for further information.
Latest Releases
Latest News
Documentation
The following documents provide more details into the use of ShaCache
Tips and Tricks
Tests
Automated test results are published on www.erp5.com.
FAQ
Licence
ShaCache is Free Software, licensed under the terms of the GNU GPL v3 (or later). For details, please see Nexedi licensing.