Encrypted Storage Service (ESS) / Storage Layer

Overview

The Storage Layer is designed to manage pointers to encrypted raw data rather than holding the data in a directly accessible format. It acts as the intermediary that:

• Accepts encrypted data along with storage location metadata.

• Stores the data using the appropriate storage method.

• Returns a unique object identifier for each stored data object.

• Maintains a mapping between object IDs and their corresponding storage locations.

• Verifies data integrity by calculating and checking data checksums.

All data handled by this layer is encrypted and remains inaccessible to any internal or external parties without proper authorization.

---

Core Functions

Pointer Management & Object ID Generation

• Data Ingestion: When encrypted data is provided, it comes with a designated storage location and relevant metadata (e.g., bucket details for S3). The Storage Layer validates that it can store the data at the specified location.

• Object ID Generation: Once validated, the layer stores the pointer information and returns a unique object ID to the application. This ID is used for future reference and retrieval.

Data Integrity Verification

• Checksum Calculation: A checksum is computed when data is stored to ensure that the data remains unaltered.

• Verification on Retrieval: When data is queried via its object ID, the Storage Layer verifies the checksum to confirm that the retrieved data is exactly as originally stored.

Data Retrieval

• Pointer Lookup: Upon receiving an object ID, the Storage Layer identifies the appropriate storage method and location.

• Fetching & Verification: The data is fetched from its storage location and the checksum is validated before returning the encrypted data to the requester.

---

Supported Storage Methods

The Storage Layer supports multiple storage solutions to provide flexibility and redundancy:

• S3: Data is stored in Amazon S3 buckets. Metadata includes bucket names and region details.

• MongoDB: Data pointers and associated metadata can be stored in a MongoDB collection.

• IPFS: Data stored in a decentralized manner using the InterPlanetary File System.

• Local Storage: For on-premise or development purposes, data can be stored locally, with metadata specifying the local path or directory.

Each storage method is integrated through modular functions that handle data input, pointer mapping, and integrity verification.