ShaCache Home ShaCache

    ShaCache

    ShaCache is a caching tool that guarantees own and third party source code relevant for deploying software is always available in the exact same version it was once used and signed. ShaCache is used as both download and binary cache within SlapOS. Download Cache allows to cache downloaded files similar to source code on the internet. If the original website used to download the file is down, the download cache will serve as backup while the Binary Cache allows to download an already-compiled software release.

    Why use ShaCache?

    ShaCache (or libnetworkcached) and its two corresponding ERP5 business templates ShaCache and ShaDir allow to utilize an ERP5 instance as self-certifying cache server using a NoSQL storage and REST API. Files are archived using the HTTP POST method with their SHA256 hash being set as url. When a client wants to download a specific file, it needs to provide the key value (hash) using the GET method and the server will send a response with the file data, offering a self-certifying, and secure binary cache of trusted content.

    When considering ShaCache as network cache, the goal of libnetworkcache python library is to abstract HTTP calls. It works as wrapper of python httplib to use the Networkcache HTTP Server. If all relevant files for building and deploying an application were available via ShaCache, this software could be setup independent of the availability of an internet connection.

    
                       __________________
                      /                  \
                      |                  |
               ------>| LIBNETWORKCACHED |
               |  ----|                  | <-----
               |  |   \__________________/      |
    GET /hash  |  |                             | POST / data
               |  | File                        |
             __|__v______                 ______|_____
            |            |               |            |
            |   Client   |               |   Client   |
            |____________|               |____________|
    
    

    The ShaCache HTTP Server is sub-divided into two Web services:

    • ShaCache - a simple HTTP server used to cache files.
    • ShaDir - a simple HTTP Server used to cache information, working like a directory.

    Getting Started

    To run a ShaCache server you need an ERP5 instance. You can run a ShaCache client from the terminal.

    Setting up a ShaCache Server

    If you want to run a dedicated ShaCache ERP5 instance (like http://shacache.org/) you can request an instance via SlapOS or Vifib.

    In case you are already running an ERP5 instance and just want to add ShaCache, the source code can be found inside the ERP5 repository business templates (ShaCache and ShaDir). Please refer to the the ERP5 documentation on installing business templates on how to add the ShaCache templates to your instance.

    Setting up a ShaCache Client

    Clients are for uploading and downloading files to/from the ShaCache server as described above. A simple ShaCache client is available on Gitlab. You can use Easy Install or pip and just do:

    easy_install slapos.libnetworkcache
    #or
    pip install  slapos.libnetworkcache

    and use the "networkcache-*" commands to upload/download files. See help for all commands available

    
    networkcache-download --help
    networkcache-upload --help
    

    API

    Anyone can download files from ShaCache (with curl/http). Uploads require being whitelisted as trusted source.

    API

    The API is straightforward:

    PUT / :
       parameter: file uploaded
       Used to upload/modify an entry
    
     GET /
       Return raw content
       Raise HTTP error (404) if key does not exist

    Examples:

    Usage Example

    Following is a quick walkthrough of how ShaCache is being used in SlapOS itself to load any of the softwares being used (OpenSSL in this case).

    The buildout.cfg for OpenSSL on the SlapOS Gitlab repository contains the location of the source file. This is the starting point (#19).

    # OpenSSL - a toolkit implementing SSL v2/v3 and TLS v1 protocols as
    #           well as a full-strength general purpose cryptography
    #           library.
    # http://www.openssl.org/
    
    [buildout]
    extends =
      ../ca-certificates/buildout.cfg
      ../coreutils/buildout.cfg
      ../patch/buildout.cfg
      ../perl/buildout.cfg
      ../zlib/buildout.cfg
    
    parts =
      openssl-output
    
    [openssl]
    recipe = slapos.recipe.cmmi
    url = https://www.openssl.org/source/openssl-1.0.2k.tar.gz
    md5sum = f965fc0bf01bf882b31314b61391ae65
    location = ${buildout:parts-directory}/${:_buildout_section_name_}
    

    Accessing ShaDir

    buildout and slapos.libnetworkcache use the following to create a source file's hash':

    echo -n https://www.openssl.org/source/openssl-1.0.2k.tar.gz | md5sum -

    With the hash, you can curl the ShaCache directory:

    curl http://dir.shacache.org/slapos-buildout-c26dfe76c4128b780971d234206ab538

    which should return a JSON file similar to this:

    [
    	[
        {
    		  "urlmd5": "c26dfe76c4128b780971d234206ab538",
    		  "sha512": "0d314b42352f4b1df2c40ca1094abc7e9ad684c5c35ea997efdd58204c70f22a1abcb17291820f0fff3769620a4e06906034203d31eb1a4d540df3e0db294016",
    		  "file": "openssl-1.0.2k.tar.gz"
    	  }, 
        "SntZvR8Eu9xsEJKtuO3hjDl5MzNdUXB2uC7Lqa0j1jK/31sLSdPkUHlXy67EhfV66U7TXh5umdFX\nEO8Oq+E6Lo7yRDULVcyWCaTw+Qc7F9LHiBRHZeIvEfx5xfIt1Ldi3bYEm8PgW1vTBgrMkTBsgBJq\nVUUuf9Nuk3OaTWLrj0s=\n"
      ]
    ]

    with the following response API:

    [
      [
        {
          "urlmd5": INDEXHASH,
          "sha512": CONTENTHASH,
          "file": FILENAME
        },
        USER_SSL_SIGNATURE
      ],
      [...]
    ]

    which means a user with signature USER_SSL_SIGNATURE uploaded the FILENAME containing the CONTENTHASH as Sha512 sum to the INDEXHASH. If 10 users uploaded this file, there will be 10 entries returned in the list.

    Downloading from ShaCache

    On your client, simply call:

    wget http://download.shacache.org/CONTENTHASH

    which triggers download of the actual file using the Sha512 sum of its content thus making sure no corrupted or inadverted files are downloaded as this would create a different hash.

    Trusted Sources

    You can discover whitelisted sources on the ShaCache server by:

    for entry, signature on [[ {"urlmd5": INDEXHASH, sha512: CONTENTHASH, file: FILENAME}, USER_SSL_SIGNATURE ], [...]]:
      if signature in LIST_OF_SIGNATURES_I_TRUST:
          return entry["sha512"]
    

    LIST_OF_SIGNATURES_I_TRUST has to be defined in a config file, in the case of SlapOS it is stored in SlapOS config (#195):

    
    ...
    [networkcache]
    download-cache-url = http://download.shacache.org/
    download-dir-url = http://dir.shacache.org/
    
    # signature certificates of the following uploaders.
    #   xxx
    #   yyy
    #   zzz
    #   aaa
    #   Test Agent (Automatic update from tests)
    signature-certificate-list =
      -----BEGIN CERTIFICATE-----
      MIIB4DCCAUkCADANBgkqhkiG9w0BAQsFADA5MQswCQYDVQQGEwJGUjEZMBcGA1UE
      CBMQRGVmYXVsdCBQcm92aW5
      ...
    

    The signatures listed here are trusted. Upload attempts from sources other than the ones listed will be ignored.

    Upload to ShaCache

    Uploading to ShaCache thus requires an SSL pair to sign a file before uploading. Should two or more users upload the same sha512, shacache will calculate the sha512 for both, keep two entries on ShaDir directory but only keep a single file. See the documentation below for further information.

    Latest Releases

    Latest News

    Documentation

    The following documents provide more details into the use of ShaCache

    Tips and Tricks

    Tests

    Automated test results are published on www.erp5.com.

    FAQ

    Licence

    ShaCache is Free Software, licensed under the terms of the GNU GPL v3 (or later). For details, please see Nexedi licensing.