ShaCache Home ShaCache


    ShaCache is a caching tool that guarantees own and third party source code relevant for deploying software is always available in the exact same version it was once used and signed. ShaCache is used as both download and binary cache within SlapOS. Download Cache allows to cache downloaded files similar to source code on the internet. If the original website used to download the file is down, the download cache will serve as backup while the Binary Cache allows to download an already-compiled software release.

    Why use ShaCache?

    ShaCache (or libnetworkcached) and its two corresponding ERP5 business templates ShaCache and ShaDir allow to utilize an ERP5 instance as self-certifying cache server using a NoSQL storage and REST API. Files are archived using the HTTP POST method with their SHA256 hash being set as url. When a client wants to download a specific file, it just needs to provide the key value (hash) using the GET method and the server will send a response with the file data.

    ShaCache allows to fetch files using their md5-checksum hashes offering a self-certifying, secure and consistent way of loading content. The goal of libnetworkcache python library is to abstract HTTP calls. It works as wrapper of python httplib to use the Networkcache HTTP Server.

                      /                  \
                      |                  |
               ------>| LIBNETWORKCACHED |
               |  ----|                  | <-----
               |  |   \__________________/      |
    GET /hash  |  |                             | POST / data
               |  | File                        |
             __|__v______                 ______|_____
            |            |               |            |
            |   Client   |               |   Client   |
            |____________|               |____________|

    The Networkcache HTTP Server is sub-divided into two Web services:

    • ShaCache - a simple HTTP server used to cache files.
    • ShaDir - a simple HTTP Server used to cache information, working like a directory.

    Getting Started

    To run ShaCache server you need an ERP5 instance.

    Setting up a ShaCache Server

    If you want to run a dedicated ShaCache instance (like, you can setup this up over SlapOS and Vifib.

    In case you are already running an ERP5 instance and just want to add ShaCache, the source code can be found inside the ERP5 repository business templates (ShaCache and ShaDir). Please refer to the the ERP5 documentation on installing business templates on how to add ShaCache to your instance. A simple ShaCache client is available on Gitlab.

    Setting up a ShaCache Client

    Clients are for uploading and downloading files to/from the ShaCache server as described above. You can use Easy Install or pip and just do:

    easy_install slapos.libnetworkcache
    pip install  slapos.libnetworkcache

    and use the "networkcache-*" commands to upload/download files. See help for all commands available

    networkcache-download --help
    networkcache-upload --help


    Anyone can download files from ShaCache (with curl/http). Uploads require being whitelisted as trusted source.


    The API is straightforward:

    PUT / :
       parameter: file uploaded
       Used to upload/modify an entry
     GET /
       Return raw content
       Raise HTTP error (404) if key does not exist


    Usage Example

    Following is a quick walkthrough of how ShaCache is being used in SlapOS itself to load any of the softwares being used (OpenSSL in this case).

    The buildout.cfg for OpenSSL on the SlapOS Gitlab repository contains the location of the source file. This is the starting point (#19).

    # OpenSSL - a toolkit implementing SSL v2/v3 and TLS v1 protocols as
    #           well as a full-strength general purpose cryptography
    #           library.
    extends =
    parts =
    recipe = slapos.recipe.cmmi
    url =
    md5sum = f965fc0bf01bf882b31314b61391ae65
    location = ${buildout:parts-directory}/${:_buildout_section_name_}

    Accessing ShaDir

    buildout and slapos.libnetworkcache use the following to create a source file's hash':

    echo -n | md5sum -

    With the hash, you can curl the ShaCache directory:


    which should return a JSON file similar to this:

    		  "urlmd5": "c26dfe76c4128b780971d234206ab538",
    		  "sha512": "0d314b42352f4b1df2c40ca1094abc7e9ad684c5c35ea997efdd58204c70f22a1abcb17291820f0fff3769620a4e06906034203d31eb1a4d540df3e0db294016",
    		  "file": "openssl-1.0.2k.tar.gz"

    with the following response API:

          "urlmd5": INDEXHASH,
          "sha512": CONTENTHASH,
          "file": FILENAME

    which means a user with signature USER_SSL_SIGNATURE uploaded the FILENAME containing the CONTENTHASH as Sha512 sum to the INDEXHASH. If 10 users uploaded this file, there will be 10 entries returned in the list.

    Downloading from ShaCache

    On your client, simply call:


    which triggers download of the actual file using the Sha512 sum of its content thus making sure no corrupted or inadverted files are downloaded as this would create a different hash.

    Trusted Sources

    You can discover whitelisted sources on the ShaCache server by:

    for entry, signature on [[ {"urlmd5": INDEXHASH, sha512: CONTENTHASH, file: FILENAME}, USER_SSL_SIGNATURE ], [...]]:
      if signature in LIST_OF_SIGNATURES_I_TRUST:
          return entry["sha512"]

    LIST_OF_SIGNATURES_I_TRUST has to be defined in a config file, in the case of SlapOS it is stored in SlapOS config (#195):

    download-cache-url =
    download-dir-url =
    # signature certificates of the following uploaders.
    #   xxx
    #   yyy
    #   zzz
    #   aaa
    #   Test Agent (Automatic update from tests)
    signature-certificate-list =
      -----BEGIN CERTIFICATE-----

    The signatures listed here are trusted. Upload attempts from sources other than the ones listed will be ignored.

    Upload to ShaCache

    Uploading to ShaCache thus requires an SSL pair to sign a file before uploading. Should two or more users upload the same sha512, shacache will calculate the sha512 for both, keep two entries on ShaDir directory but only keep a single file. See the documentation below for further information.

    Latest Releases

    • XXX

    Latest News

    • XXX


    The following documents provide more details into the use of ShaCache

    Tips and Tricks


    Automated test results are published on



    ShaCache is Free Software, licensed under the terms of the GNU GPL v3 (or later). For details, please see Nexedi licensing.