Storage model

This is the documentation for the vault storage model, including the types of files it uses, their internal structure and the encryption algorithms used.

Use this document as reference for any software development that requires interaction with the vault files.

File types

The vault storage model uses different types of files:

• Lock file: File used to prevent multiple instances of PersonalMediaVault accessing the same vault.
• Unencrypted JSON files: Configuration files that do not contain any protected vault data.
• Encrypted JSON files: Used to store metadata.
• Index files: Used to store lists of media asset IDs, in order to make searching faster.
• Encrypted assets: Encrypted files containing the media assets. They can be single-file or multi-file.

Lock file

The lock file has the .lock extension.

It stores in plain text, a decimal number representing the PID of the current Process accessing the vault.

PersonalMediaVault backend should check for the existence of this file and the process before accessing the vault.

Unencrypted JSON files

Unencrypted JSON files have the .json extension.

They follow the JSON format. The schema varies depending on the specific file.

Since they are not encrypted, they just store configuration, like the port it should listen, or the encryption parameters.

Encrypted JSON files

Encrypted JSON files have the .pmv extension.

They take as a base a JSON plaintext, that is encrypted using an algorithm like AES.

They are binary files, with the following structure:

The system is flexible enough to allow multiple encryption algorithms. Currently, there are 2 supported ones:

• AES256_FLAT: ID = 1, Uses ZLIB (RFC 1950) to compress the data, and then uses AES with a key of 256 bits to encrypt the data, CBC as the mode of operation and an IV of 128 bits. This algorithm uses a header containing the following fields:

• AES256_FLAT: ID = 2, Uses AES with a key of 256 bits to encrypt the data, CBC as the mode of operation and an IV of 128 bits. This algorithm uses a header containing the following fields:

Index files

Index files have the .index extension.

They are sorted lists of media assets identifiers. They can store all the existing identifiers, or a fraction of them, for example, for a tag.

Thanks to being sorted, searching for a specific identifier can be achieved using binary search.

They are binary files, consisting of the following fields:

Encrypted assets

Encrypted assets have the .pma extension.

They stored one or multiple encrypted files.

They are also binary files, and they can be of two types:

• Single-File encrypted assets: They store a single file in size-limited chunks. Their name usually starts with s_.
• Multi-File encrypted assets: They store multiple smaller files. Their name usually starts with m_.

Single-File encrypted assets

These asset files are used to store a single and possibly big file in chunks, encrypted each chunk using the same method described by the Encrypted JSON files section.

They are binary files consisting of 3 contiguous sections: The header, the chunk index and the encrypted chunks.

The header contains the following fields:

After the header, the chunk index is stored. For each chunk the file was split into, the chunk index will store a metadata entry, withe the following fields:

After the chunk index, the encrypted chunks are stored following the same structure described in the Encrypted JSON files section.

This chunked structure allows to randomly access any point in the file as a low cost, since you don’t need to decrypt the entire file, only the corresponding chunks. This capability is specially great for video rewinding and seeking.

Multi-File encrypted assets

These asset files are used to store multiple smaller files, meant to be sorted and accessed by an index number.

They are binary files consisting of 3 contiguous sections: The header, the file table and the encrypted files.

The header contains the following fields:

After the header, a file table is stored. For each file stored by the asset, a metadata entry is stored, with the following fields:

After the file table, each file is stored following the same structure described in the Encrypted JSON files section.

This format is useful to store video previews, without the need to use too many files.

Vault folder structure

Media vaults are stored in folders. A vault folder may contain the following files and folders:

Media assets folder

The media assets are stored inside the media folder.

In order to prevent the folder size to increase too much, the assets are distributed evenly in 256 sub-folders. The sub-folder name for each media asset is calculated from its identifier, since it’s a 64 bit unsigned integer, the folder name is the identifier module 256, and the result turned into a 2 character hex lowercased string

Examples: 00, 01, 02…, fd, fe, ff.

Inside each subfolder, the assets are stored inside their own folders, named by turning their identifier into a decimal string. Examples:

• media_id=0 - Stored inside {VAULT_FOLDER}/media/00/0
• media_id=15 - Stored inside {VAULT_FOLDER}/media/0f/15

import (
    "fmt",
    "hex", 
    "path",
)

func GetMediaAssetFolder(vault_path string, media_id uint64) string {
    subFolderName := hex.EncodeToString([]byte{ byte(media_id % 256) });

    return path.Join(vault_path, "media", subFolderName, fmt.Sprint(media_id))
}

The media asset folder may contain up to 3 types of files:

• Media asset metadata file: Named meta.pmv and used to store metadata.
• Single-File assets: Named concatenating the s_ prefix and the asset ID in decimal, with .pma extension.
• Multi-File assets: Named concatenating the m_ prefix and the asset ID in decimal, with .pma extension.

Media asset metadata file

Each media asset folder must contain a file named meta.pmv, being an encrypted JSON file containing the metadata of the media asset.

The file contains the following fields:

The Resolution object has the following fields:

The Subtitle object has the following fields:

The TimeSplit object has the following fields:

The AudioTrack object has the following fields:

The Attachment object has the following fields:

The image notes asset is a JSON file, containing an array of ImageNote objects, with the following fields:

Tag indexes folder

When a tag is added to the vault, a new index file is created inside the tags folder, with a name made by concatenating the tag_ prefix with the tag identifier encoded in decimal, and the .index extension.

import (
    "fmt",
    "path",
)

func GetTagIndexPath(vault_path string, tag_id uint64) string {
	return path.Join(vault_path, "tags", "tag_"+fmt.Sprint(tag_id)+".index")
}

Each tag index file contains the list of media asset identifiers that have such tag.

Credentials file

The credentials file, named credentials.json is an unencrypted JSON file used to store the hashed credentials, along with the encrypted vault key.

The JSON file contains the following fields:

Each Account is an object with the following fields:

Currently, the following methods are implemented:

• AES256 + SHA256 + SALT16 - Identifier: aes256/sha256/salt16

AES256 + SHA256 + SALT16

This algorithm uses a random salt of 16 bytes (128 bits).

The password hash is calculated by using the SHA256 algorithm 2 times on the binary concatenation of the password (as UTF-8) and the random salt:

import (
    "sha256"
)

func ComputePasswordHash(password string, salt []byte) []byte {
	firstHash := sha256.Sum256(append([]byte(password), salt...))
	secondHash := sha256.Sum256(firstHash[:])
	return secondHash[:]
}

The vault ket is encrypted using the AES256 algorithm, using the system defined in the Encrypted JSON files section. Specifically using the AES256_FLAT mode.

The key for the encryption is calculated by hashing with SHA256 the the binary concatenation of the password (as UTF-8) and the random salt:

import (
    "sha256"
)

func ComputeAESEncryptionKey(password string, salt []byte) []byte {
	passwordHash := sha256.Sum256(append([]byte(password), salt...))
	return passwordHash[:]
}

Media ID tracker

The media ID tracker file, named media_ids.json is an unencrypted JSON file used to store the number of used media identifiers, very important to prevent duplicated identifiers.

The JSON file has just one field:

Tasks tracker

The task tracker file, named tasks.json is an unencrypted JSON file used to store the number of used task identifiers, in order to prevent duplicates. It also stores the pending tasks, in order to continue them in case of a vault restart.

The JSON file contains the following fields:

The PendingTask objects have the following fields:

Albums file

The albums file, named albums.pmv is an encrypted JSON file used to store the list of existing albums in the vault.

The file has the following fields:

The Album object has the following fields:

Tags file

The tags file, named tag_list.pmv is an encrypted JSON file used to store the list of existing tags in the vault.

The file has the following fields:

User configuration file

The user configuration file, named user_config.pmv is an encrypted JSON file used to store the vault configuration set by the user.

The file has the following fields:

The VideoResolution object has the following fields:

The ImageResolution object has the following fields:

Main index file

The main index file, named main.index is an index file containing every single media asset identifier existing in the vault.

This file is used to check if a media asset exists and to perform searches when a tag filter is not specified.