CBFS Storage 2022 Python Edition
Version 22.0 [Build 8460]

Introduction

Welcome to CBFS Storage, an all-in-one file storage solution that allows applications to create and embed secure, fully-functional filesystems anywhere. These filesystems, known as Vaults, are completely self-contained and can be stored in files, memory, or any other location your application has read/write access to.

Each vault can store as many files, directories, custom file attributes and tags, alternate streams, and symbolic links as desired; your application has complete control over its contents and capacity. To ensure your data is stored efficiently and securely, CBFS Storage also offers built-in Compression and Encryption functionality, and allows you to provide your own algorithms if desired.

The included CBDisk class offers an easy-to-use toolkit for creating virtual disks whose contents are stored as contiguous blocks of data. It doesn't use Vaults' format but lets you format the virtual disk to any filesystem, such as exFAT or NTFS. CBDisk also enables applications to expose existing disk images as virtual disks.

Included Classes

CBMemoryDriveThe CBMemoryDrive component lets applications create an in-memory vault, accessible to all or some processes as a regular disk.
CBVaultDriveThe CBVaultDrive component lets applications create a vault, manipulate its contents, and mount it as a virtual drive.

Additional Information

You will always find the latest information about CBFS Storage at our web site: www.callback.com. We offer free, fully-functional 30-day trials for all of our products, and our technical support staff are happy to answer any questions you may have during your evaluation.

Please direct all technical questions to support@callback.com. To help support technicians assist you as quickly as possible, please provide an detailed and accurate description of your problem, the results you expected, and the results that you received while using our product. For questions about licensing and pricing, and all other general inquiries, please contact sales@callback.com.

Thank You!

Thank you for choosing CBFS Storage for your development needs. We realize that you have a choice among development tools, and that by choosing us you are counting on us to be a key component in your business. We work around the clock to provide you with ongoing enhancements, support, and innovative products; and we will always do our best to exceed your expectations!

Deployment

The topics in this section provide information regarding the deployment of applications built with CBFS Storage.

Topics

Driver-specific

The topics in this section provide information relevant to the deployment of applications built with the classes that include a kernel-mode driver or make use of a third-party kernel driver: CBVaultDrive or CBMemoryDrive.

This information should be reviewed carefully when designing a deployment strategy for such an application, since CBFS Storage's kernel mode drivers and other supplementary DLLs must be distributed along with the application in order for it to function correctly.

Topics

Prerequisites

The classes create a virtual drive, visible to other processes, which requires a certain level of integration between the classes and the system itself. In order for an application that uses such class to function correctly, the following prerequisites must first be met on the target machine:

  • Windows: The kernel mode drivers must be installed; please refer to the Driver Installation in Windows topic for more information.
  • macOS: macFUSE together with its "FUSE Compatibility Layer" must be installed in the target system (including the final end-users' systems). Note that the product does not include macFUSE, it must be downloaded and installed separately.
  • Linux: The system's kernel must have been compiled with support for FUSE. Also, FUSE 2.9 user-mode libraries must be installed in the development system. This can be achieved using the following commands:
    • RedHat/CentOS and derivative Linux distributions: sudo yum install fuse-devel
    • Debian/Ubuntu and derivative Linux distributions: sudo apt-get install libfuse-dev
    FUSE 2 must also be installed in target systems, where your application is deployed. Modern versions of Linux don't include FUSE 2 by default, so it can be installed using these commands:

    • RedHat/CentOS and derivative Linux distributions: sudo yum install fuse-libs
    • Debian/Ubuntu and derivative Linux distributions: sudo apt-get install libfuse2

Driver Installation in Windows

This topic describes the functionality, available in CBVaultDrive and CBMemoryDrive classes when the class is used in Windows operating system.

At a high level, CBFS Storage consists of a kernel mode driver, a helper DLL, and a user mode library; all of which work together in tandem to provide the product's functionality. Therefore, it is necessary to install the CBFS Storage kernel mode driver and helper DLL when deploying an application built with the CBFS Storage user mode library.

The functionality needed to install the above-mentioned modules is included in the user mode library itself, as well as in a separate installer DLL. The drivers directory, located within the product's Python package (see notes below), contains the following files:

{ClassName}.cab Contains the main drivers, PnP bus drivers, helper DLLs, and the supplementary installation/uninstallation files.
installer/{ClassName}Inst.h A header file for the installer DLL. The installer DLL may be used on the target system to install (or uninstall) the items within {ClassName}.cab.
installer/x64/{ClassName}Inst.dll The C/C++ installer DLL for the x64 (AMD64) processor architecture.
installer/x86/{ClassName}Inst.dll The C/C++ installer DLL for 32-bit x86 processor architecture.
installer/ARM/{ClassName}Inst.dll The C/C++ installer DLL for 32-bit ARM processor architecture.
installer/ARM64/{ClassName}Inst.dll The C/C++ installer DLL for 64-bit ARM processor architecture.

Windows: Note: When the user-mode library is installed or updated on end-user systems, it is required to ensure that the kernel-mode drivers already present in the system are updated to match the version of the installed user-mode library.

Python Package Notes

In the Python edition, the drivers directory described above is included as part of the product's Python package, <install_dir>\cbfsstorage-22.0.xxxx.tar.gz. After installing the package with pip (see the User Mode Library topic), the drivers directory can be found within the product's package directory, which can be located by running pip show cbfsstorage.

Installation and Uninstallation via User Mode Library Methods

The class includes the following methods to install and uninstall the required files; please refer to their documentation for more information:

Important: uninstall must only be used when completely removing the driver. When updating the driver, this method must not be used as it may cause the OS to incorrectly remove the driver on reboot. Please refer to the "Updating the Driver" section, below, for more information.

Installation and Uninstallation via Installer DLL Functions

The installer DLL is a lightweight, stand-alone library that contains only the functionality required for installing and uninstalling the required files. It is available in both 32-bit and 64-bit versions (each of which is capable of installing both 32-bit and 64-bit drivers and helper DLLs); and may be used as desired in installation scripts, setup applications, or any other executable capable of loading dynamically-linked libraries (DLLs).

The functions exposed by the installer DLL mirror the class methods listed above. Each function is available in two forms: those with an *A suffix, which can be used with ANSI/UTF8 strings; and those with a *W suffix, which can be used with Unicode (UTF16) strings.

Updating the Driver

To update the driver, call the install method. The new version of the driver will replace the older version. Please do not call the uninstall method when updating the driver.

Uninstalling the Driver

To uninstall the driver completely, call the uninstall method. If the driver cannot be immediately uninstalled, it will be marked for removal and uninstalled on the next reboot.

Use caution when calling uninstall ; if it gets called and the driver cannot be uninstalled immediately, and then install is subsequently called to install a new version, then upon reboot, the OS will end up uninstalling the newly-installed driver.

Important: The driver should only be uninstalled when the intent is to completely remove it from the system. Do not uninstall the driver to update it.

Reboot Requirements

Depending on the current state of the system, as well as the options chosen when installing or uninstalling the driver, the OS may need to reboot to complete the operation.

For example, the helper DLL must be loaded by Windows Explorer when it starts, and a reboot or restart of Explorer is required for this to occur. When installing or uninstalling the Plug-and-Play (PnP) drivers, a reboot is almost always requested by Windows.

Always check the return value of the install and uninstall methods/functions; it will indicate whether a reboot is required (and if so, which module(s) required it).

Additional Notes

The OS treats major versions of the driver as separate products; they can operate in parallel and do not share any resources. Old major versions may optionally be removed from the system when calling install by passing the appropriate value for its Flags parameter.

For each major version of the product, only one copy of the driver can be installed at any time. When the driver is being installed, its version is checked, and one of the following three things occurs:

  • If no driver with the same major version is currently installed, then the install procedure installs the driver as a new product.
  • If a driver with the same major version and an older minor version is currently installed, then the install procedure updates the existing driver with the new one.
  • If a driver with the same major version and a newer minor version is currently installed, then the install procedure leaves the existing driver unchanged.

When deploying files to a target system, the CAB file must remain present on the system. This file is required for uninstallation of the driver at a later time.

The product's installation code maintains a ProductGUID-based record of driver installations in the Windows Registry, creating a separate registry entry for each different ProductGUID. When the driver is "uninstalled", the corresponding registry entry is removed. The driver is only removed from the system if there are no entries left in the registry that reference the driver.

Windows 7 and Windows 2008 Server R2

Kernel-mode drivers are signed using the SHA2 algorithm. The original releases of Windows 7 and Windows 2008 Server R2 didn't support SHA2. To be able to load the newest versions of the drivers, the system needs to have certain updates installed. The updates are KB976932 (Service Pack 1 of the mentioned systems) and KB4474419 (Security Update).

Required Permissions

By default, Windows only allows installation and uninstallation of the CBFS Storage system files (kernel mode drivers and helper DLLs) to be performed from a user account which is a member of the Administrators group.

On systems where UAC is enabled, the process responsible for installing or uninstalling the system files must run with elevated permissions. Detection of current privileges and elevation of permissions is not within the scope of the class itself.

Some examples of obtaining the required permissions for driver installation and uninstallation are below.

  • Starting the application which uses the class with the "Run as administrator" option.
  • Modifying the Load and unload device drivers setting in the Local Security Policy under the User Rights Assignment section.
  • Including a manifest alongside the application indicating the requirement for elevated permissions. For instance, if a file MyApp.exe.manifest with the content below exists next to the application MyApp.exe, it will prompt for elevated permissions when started (if required).

    <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0"> <assemblyIdentity version="1.0.0.0" processorArchitecture="X86" name="ExeName" type="win32"/> <description>elevate execution level</description> <trustInfo xmlns="urn:schemas-microsoft-com:asm.v2"> <security> <requestedPrivileges> <requestedExecutionLevel level="requireAdministrator" uiAccess="false"/> </requestedPrivileges> </security> </trustInfo> </assembly>

User Mode Library

Windows: Note : The user-mode library must be deployed to end-user systems together with the kernel-mode drivers; the version of the kernel-mode drivers on the end-user systems must be equal or newer to the version of the user-mode library. Thus, when the user-mode library is installed or updated on end-user systems, it is required to ensure that the kernel-mode drivers already present in the system are updated to match the version of the installed user-mode library.

The user mode library comes in two pieces, both of which must be deployed with the application:

  • A Python module named cbfsstorage.py.
  • A native library (unmanaged), named as follows:
    • Windows: pycbfsstorage22.dll (available for x64 and x86 processor architectures)
    • Linux: libpycbfsstorage.so.22.0 (available for x64 and x86 processor architectures)
    • macOS: libpycbfsstorage22.0.dylib (available for x64 and ARM64 processor architectures)

Both of these are included in the product's Python package, <install_dir>\cbfsstorage-22.0.xxxx.tar.gz, which should be installed using pip: cd C:\path\to\install_dir python -m pip install cbfsstorage-22.0.xxxx.tar.gz

Once the product's Python package has been installed, the module can be imported and used: from cbfsstorage import *. Nothing else is required to deploy the application.

As an alternative to installing the module using pip, you can utilize the built-in setuptools module to package the module for deployment or install it to the machine. python setup.py build --build-lib=<app_dir> The above setup command packages the module and native library for deployment. A folder is created in the app_dir directory with the module and the native library packaged inside.

Windows Only: Remember to deploy the drivers too, as they are an integral part of CBFS Storage.

General Information

The topics in this section provide general information about various aspects of the product's functionality.

Topics

Buffer Parameters

Some events include one or more parameters intended for use as a binary data buffer. Depending on the event, these parameters may contain data when the event is fired, or it may be expected that the application populates them with the desired amount of data during the event handler. Some events combine both paradigms, and expect the application to modify the data already present when the event is fired.

The documentation for such events will describe which of the above cases applies to each buffer parameter. Note that, in all cases, buffer parameters point to a pre-allocated block of unmanaged memory, the size of which is specified by the parameter immediately following the buffer parameter. In cases where data is to be written, be sure to write it directly to the pointed-to memory, don't change the value of the buffer parameter itself. Buffer parameters are always of the ctypes.c_void_p type; use the ctypes.memmove() method to read and write data from and to the unmanaged memory region. To obtain a pointer to a Python buffer suitable for use with ctypes.memmove(), call ctypes.c_void_p.from_buffer() and pass it a byte array or some other type that implements Python's buffer interface.

Callback Mode

As discussed in the Vaults topic, the default behavior of CBVAULT class and its relatives, CBVaultDrive and CBMemoryDrive classes is to create a vault using a real file on a local disk. However, the filesystem engine behind these classes doesn't require a vault to be a local file; it can be a remote file, a memory region, or anything else that the application can provide random read/write access to.

Applications that wish to use something other than a local file to store a vault must enable callback mode using the class's callback_mode property. When callback mode is enabled, applications must handle the following events (which map closely to the Windows File API) in order for the class to interact with the vault. For brevity, vaults created and accessed using callback mode are typically referred to as "callback mode vaults".

  • on_vault_close: Fires when the currently-open vault should be closed.
  • on_vault_delete: Fires when a callback mode vault (that is not open) should be deleted.
  • on_vault_flush: Fires when any buffered vault data should be flushed out to storage.
  • on_vault_get_parent_size: Fires when the class needs to know how much free space is available for the currently-open vault to use for automatic growth.
  • on_vault_get_size: Fires when the class needs to know the size of the currently-open vault.
  • on_vault_open: Fires when a callback mode vault should be opened (and, if necessary, created).
  • on_vault_read: Fires when the class needs to read one or more pages of vault data.
  • on_vault_set_size: Fires when the class needs to resize (i.e., shrink or grow) the currently-open vault.
  • on_vault_write: Fires when the class needs to write one or more pages of vault data.
(Note: All API members discussed in this topic are available in both CBVaultDrive, CBMemoryDrive, and CBVAULT, unless otherwise noted.)

Callback mode is an extremely powerful feature for applications that want to fine-tune performance. For example, consider the following scenario: a help authoring tool that keeps a compound file in memory for fast operations. In this scenario, the vault itself would be a help project which the help authoring tool loads into memory and uses the above events to access. When a user needs to save the project, the vault is flushed and the data in memory is copied back to the help project on disk.

Note: An application should not attempt to call class's methods from handlers of the listed events. Doing this is guaranteed to cause a deadlock.

Encryption

The CBVaultDrive, CBMemoryDrive, and CBVAULT classes include strong built-in data encryption support, which can be applied to individual files and alternate streams, entire vaults, or both. Each file, alternate stream, and vault can have its own encryption key. (Note: the API members discussed in this topic are available in all listed classes, unless otherwise noted.)

Encrypting Vaults

To specify a default encryption mode and password to use when creating new vaults, applications can set the vault_encryption and vault_password properties. To change the encryption mode and/or password of an existing vault, use the update_vault_encryption method.

When opening an existing vault, vault_encryption is updated to reflect the vault's encryption mode; and if the vault is encrypted, the password specified by vault_password is used to access it.

Encrypting Files and Alternate Streams

To specify a default encryption mode and password for files and alternate streams, applications can set the default_file_encryption and default_file_password properties. Additionally, the following methods allow applications to set a file or alternate stream's encryption mode and/or password explicitly:

When a file or alternate stream is encrypted, its encryption password must be provided in order to access it; many methods in the class's API provide a Password parameter for this purpose. If the application doesn't explicitly specify a password when calling such a method, then the default_file_password will be used, if possible.

Using Custom Encryption

The class's built-in encryption implementation uses 256-bit AES encryption in XTS mode with PBKDF2 key derivation based on a HMAC-SHA256 key hash. However, applications also can choose to provide their own custom encryption and key derivation implementations. This flexibility allows applications to support more sophisticated security techniques, such as PKI-based encryption, or Digital Rights Management. To get started, do the following:

  1. Choose a custom encryption mode to implement (i.e., one of the VAULT_EM_CUSTOM* options from the table below). This choice will determine:
    • Whether the custom encryption implementation uses a 256-bit, 512-bit, or 1024-bit block size; and,
    • Whether to use built-in key derivation, custom key derivation, or no key derivation.
  2. Implement the on_data_encrypt and on_data_decrypt events.
  3. If a VAULT_EM_CUSTOM*_CUSTOM_KEY_DERIVE mode was chosen, implement the on_key_derive event.
  4. If a VAULT_EM_CUSTOM*_DIRECT_KEY mode was chosen, implement the on_hash_calculate event.

Supported Encryption Modes

The class support the following encryption modes:

VAULT_EM_NONE0x0Don't use encryption.

VAULT_EM_DEFAULT0x1Use default encryption (VAULT_EM_XTS_AES256_PBKDF2_HMAC_SHA256).

VAULT_EM_XTS_AES256_PBKDF2_HMAC_SHA2560x2Use AES256 encryption with PBKDF2 key derivation based on a HMAC_SHA256 key hash.

VAULT_EM_CUSTOM256_PBKDF2_HMAC_SHA2560x3Use event-based custom 256-bit encryption with PBKDF2 key derivation based on a HMAC_SHA256 key hash.

256-bit (32-byte) block size.

VAULT_EM_CUSTOM512_PBKDF2_HMAC_SHA2560x4Use event-based custom 512-bit encryption with PBKDF2 key derivation based on a HMAC_SHA256 key hash.

512-bit (64-byte) block size.

VAULT_EM_CUSTOM1024_PBKDF2_HMAC_SHA2560x5Use event-based custom 1024-bit encryption with PBKDF2 key derivation based on a HMAC_SHA256 key hash.

1024-bit (128-byte) block size.

VAULT_EM_CUSTOM256_CUSTOM_KEY_DERIVE0x23Use event-based custom 256-bit encryption with custom key derivation.

256-bit (32-byte) block size.

VAULT_EM_CUSTOM512_CUSTOM_KEY_DERIVE0x24Use event-based custom 512-bit encryption with custom key derivation.

512-bit (64-byte) block size.

VAULT_EM_CUSTOM1024_CUSTOM_KEY_DERIVE0x25Use event-based custom 1024-bit encryption with custom key derivation.

1024-bit (128-byte) block size.

VAULT_EM_CUSTOM256_DIRECT_KEY0x43Use event-based custom 256-bit encryption with no key derivation.

256-bit (32-byte) block size. Useful for cases where the password is an identifier for an external key and should not be used for key derivation.

VAULT_EM_CUSTOM512_DIRECT_KEY0x44Use event-based custom 512-bit encryption with no key derivation.

512-bit (64-byte) block size. Useful for cases where the password is an identifier for an external key and should not be used for key derivation.

VAULT_EM_CUSTOM1024_DIRECT_KEY0x45Use event-based custom 1024-bit encryption with no key derivation.

1024-bit (128-byte) block size. Useful for cases where the password is an identifier for an external key and should not be used for key derivation.

VAULT_EM_UNKNOWN0xFFUnidentified or unknown encryption.

Event Handling

In CBFS Storage, the purpose of most events is to let an application provide custom data storage, encryption, or compression routines. Events are also used to communicate the progress of long-running operations.

Since the CBDriveVault class's events are typically tied directly to requests from the OS or kernel-mode actors, it's critical that event handlers complete quickly in order to prevent the system from being blocked. To help prevent such blocking, the CBFS Storage system driver enforces request timeouts on a per-virtual-drive basis.

Any event handler has 30 seconds to finish its operations; after this, if an event handler didn't return, the driver completes the request handling and reports an error to the OS.

Events are easy to understand and to use, but keep the following things in mind when implementing an application's event handlers:

  1. Event handlers must not perform any operations, explicitly or implicitly, against files opened in buffered mode. Any file accessed from an event handler must be opened using the Windows API's FILE_FLAG_NO_BUFFERING flag.
  2. Event handlers must not perform any asynchronous procedure calls (APCs).
  3. If the currently-open vault is stored in a file on an NTFS-formatted disk, neither the file containing the vault nor the disk said file resides on can be compressed or encrypted using NTFS features.

Events handlers which violate any of the restrictions described above will cause a system-wide deadlock.

In CBFS Storage, the purpose of most events is to let an application provide custom data storage, encryption, or compression routines. Events are also used to communicate the progress of long-running operations.

Since the CBDriveVault class's events are typically tied directly to requests from the OS or kernel-mode actors, it's critical that event handlers complete quickly in order to prevent the system from being blocked. To help prevent such blocking, the CBFS Storage system driver enforces request timeouts on a per-virtual-drive basis.

Any event handler has 30 seconds to finish its operations; after this, if an event handler didn't return, the driver completes the request handling and reports an error to the OS.

Events are easy to understand and to use, but keep the following things in mind when implementing an application's event handlers:

  1. Event handlers must not perform any operations, explicitly or implicitly, against files opened in buffered mode. Any file accessed from an event handler must be opened using the Windows API's FILE_FLAG_NO_BUFFERING flag.
  2. Event handlers must not perform any asynchronous procedure calls (APCs).
  3. If the currently-open vault is stored in a file on an NTFS-formatted disk, neither the file containing the vault nor the disk said file resides on can be compressed or encrypted using NTFS features.

Events handlers which violate any of the restrictions described above will cause a system-wide deadlock.

Recursive Calls

To ensure stable operation, it is critical to avoid accessing drives and filesystems recursively. Essentially, this means that event handlers must not perform any operations involving the drive or filesystem that the event fired for (i.e., don't read from/write to files on it, don't unmount the media, etc.).

To ensure stable operation, it is critical to avoid accessing drives and filesystems recursively. Essentially, this means that event handlers must not perform any operations involving the drive or filesystem that the event fired for (i.e., don't read from/write to files on it, don't unmount the media, etc.).

Error Reporting and Handling

Error Codes

The CBFS Storage class APIs communicate errors using the error codes defined in their Error Codes pages (available for each class). The CBVaultDrive class also communicate errors using OS-specific error codes; for example, on Windows they use Win32 error codes defined in WinError.h, which is part of the Windows Platform SDK.

Reporting Errors to the Class from Event Handlers

If the event has a ResultCode parameter, the event handler can use it to return the result code of the operation to the class. The ResultCode parameter is set to 0 by default, which indicates the operation was successful.

If an unhandled exception occurs in the event handler, it will be caught by the class, which will fire the OnError event.

In some events, the OS doesn't expect the error code to be returned and either the class or the OS ignores the returned error code. Please, refer to the description of a particular event for more information.

How to Handle Errors Reported by the Class

If an error occurs, the class will throw an exception. The code property of the exception object will contain an error code, and the message property will contain an error message (if available).

Extended Logging in Windows (Driver-based Classes Only)

Some class methods in CBFS Storage are capable of writing extended information about reported errors to the Windows event logs, which can be viewed using the system's eventvwr.exe tool. The user mode part of the class writes to the "Windows Logs \ Application" folder, while the kernel mode part writes to the "Windows Logs \ System" folder.

The information written in the extended logs is meaningful to the Callback Technologies development team, but not to end-users, so extended logging is disabled by default. If issues occur during the installation of the CBFS Storage system drivers, or while using the class, please do the following:

  1. Enable extended logging (see below).
  2. Replicate the issue.
  3. Using Event Viewer (eventvwr.exe), export the event log entries from the locations mentioned above in native format (please restrict the scope of the export to just those entries related to CBFS Storage).
  4. Submit an issue report that includes the exported file.

There are two ways to toggle extended logging for a class:

  1. By toggling the class's LoggingEnabled configuration setting.
  2. By adding a DWORD-typed value named Enabled to the HKEY_LOCAL_MACHINE\SOFTWARE\Callback Technologies\{ClassName}\EventLog registry key and setting it to 0 (disabled) or 1 (enabled).
    • Replace the {ClassName} part of the registry key path with the name of the applicable class.
    • If this registry key, one of its parents, or the value itself does not exist, please create it manually.
    Note that if your code runs in emulated mode (x86 mode on x64 or ARM64 architecture), you need to add the value to the HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Callback Technologies\{ClassName}\EventLog registry key in addition to the "main" registry key.

The system must be rebooted anytime extended logging is enabled or disabled to make the changes take effect.

Driver-specific

The topics in this section provide information specific to the classes on Windows.

Topics

Event Handling

In CBFS Storage, the purpose of most events is to let an application provide custom data storage, encryption, or compression routines. Events are also used to communicate the progress of long-running operations.

Since the CBDriveVault class's events are typically tied directly to requests from the OS or kernel-mode actors, it's critical that event handlers complete quickly in order to prevent the system from being blocked. To help prevent such blocking, the CBFS Storage system driver enforces request timeouts on a per-virtual-drive basis.

Any event handler has 30 seconds to finish its operations; after this, if an event handler didn't return, the driver completes the request handling and reports an error to the OS.

Events are easy to understand and to use, but keep the following things in mind when implementing an application's event handlers:

  1. Event handlers must not perform any operations, explicitly or implicitly, against files opened in buffered mode. Any file accessed from an event handler must be opened using the Windows API's FILE_FLAG_NO_BUFFERING flag.
  2. Event handlers must not perform any asynchronous procedure calls (APCs).
  3. If the currently-open vault is stored in a file on an NTFS-formatted disk, neither the file containing the vault nor the disk said file resides on can be compressed or encrypted using NTFS features.

Events handlers which violate any of the restrictions described above will cause a system-wide deadlock.

In CBFS Storage, the purpose of most events is to let an application provide custom data storage, encryption, or compression routines. Events are also used to communicate the progress of long-running operations.

Since the CBDriveVault class's events are typically tied directly to requests from the OS or kernel-mode actors, it's critical that event handlers complete quickly in order to prevent the system from being blocked. To help prevent such blocking, the CBFS Storage system driver enforces request timeouts on a per-virtual-drive basis.

Any event handler has 30 seconds to finish its operations; after this, if an event handler didn't return, the driver completes the request handling and reports an error to the OS.

Events are easy to understand and to use, but keep the following things in mind when implementing an application's event handlers:

  1. Event handlers must not perform any operations, explicitly or implicitly, against files opened in buffered mode. Any file accessed from an event handler must be opened using the Windows API's FILE_FLAG_NO_BUFFERING flag.
  2. Event handlers must not perform any asynchronous procedure calls (APCs).
  3. If the currently-open vault is stored in a file on an NTFS-formatted disk, neither the file containing the vault nor the disk said file resides on can be compressed or encrypted using NTFS features.

Events handlers which violate any of the restrictions described above will cause a system-wide deadlock.

Recursive Calls

To ensure stable operation, it is critical to avoid accessing drives and filesystems recursively. Essentially, this means that event handlers must not perform any operations involving the drive or filesystem that the event fired for (i.e., don't read from/write to files on it, don't unmount the media, etc.).

To ensure stable operation, it is critical to avoid accessing drives and filesystems recursively. Essentially, this means that event handlers must not perform any operations involving the drive or filesystem that the event fired for (i.e., don't read from/write to files on it, don't unmount the media, etc.).

Custom Drive Icons

This topic describes the functionality, available in CBVaultDrive and CBMemoryDrive classes when the class is used in Windows operating system.

Virtual drives created with the CBVaultDrive class can have a custom icon associated with them to better distinguish them in Windows Explorer. There are a few different ways to accomplish this:

Using Additional Files

If placing additional files into the virtual drive itself is an acceptable condition, Windows provides a couple of file-based mechanisms for specifying a custom icon.

To specify a custom icon for the virtual drive itself, an autorun.inf file can be created based on the information in Microsoft's Autorun.inf article.

Additionally, custom icons can be specified for subdirectories of the virtual drive using desktop.ini files, which can be created based on the information in Microsoft's Desktop.ini article. Note that desktop.ini files cannot be used to specify a custom icon for the root directory of the virtual drive (i.e., they cannot be used to change the icon of the virtual drive itself).

Using Registry Keys

If the virtual drive is assigned a persistent drive letter, using registry keys to assign a custom icon may be a good option. To specify a custom icon using the registry, create a subkey like {DriveLetter}\DefaultIcon (e.g., K\DefaultIcon) under one of the following keys:

  • HKEY_CURRENT_USER\SOFTWARE\Classes\Applications\Explorer.exe\Drives, if the custom icon should only be used for the current user.
  • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\DriveIcons, if the custom icon should be used for all users. (Note that manipulating anything under the HKEY_LOCAL_MACHINE registry hive requires administrative rights.)

Regardless of which key it is created under, the {DriveLetter}\DefaultIcon subkey's (Default) (i.e., "unnamed") value should then be set to the absolute path of the icon file.

Please note that custom icons specified in this manner are only effective so long as the drive letter assigned to the virtual drive remains unchanged over time; if its drive letter changes, the registry keys used to specify the custom icon will need to be updated accordingly.

Using the Class and its Shell Helper DLL

As long as the Helper DLL has been installed to the system using the install method, custom icons can be assigned to a virtual drive directly using the class. This method of specifying custom icons is especially valuable when project constraints preclude placing additional files into the drive or modifying the registry.

Custom icons assigned in this manner function a bit differently than those assigned using the two methods described above, as they are implemented using Windows' icon overlay mechanism. Consequently, the custom icons are restricted to 25% of the original icon's area (except for 16x16 icons); the tables below describe the required sizes and color levels of the assets in the icon file.

Overlay icon sizes map as follows:

Main Icon SizeOverlay Icon Size
16x1610x10
32x3216x16
48x4824x24
256x256128x128

Icon assets must have the following color levels:

Icon SizeColor Level
16x1616 colors
32x3216 colors
48x48256 colors
256x25632-bit color

Because it's possible to specify multiple different overlay icons (e.g., to represent different drive states), icons are assigned through the class using a two-step process:

  1. Register the desired icon(s) using the register_icon method. (Note that administrative rights are required to execute this method successfully.)
  2. Switch between the registered icon(s) using the set_icon and reset_icon methods.

Icons are copied to a temporary location when registered; and removed from said location when unregistered using the unregister_icon method.

It is important to keep in mind that Windows limits the number of registered overlay icons to 15 (this is a global limit for the entire system, and it cannot be changed). Since other applications on the system (e.g., OneDrive, Dropbox, etc.) may have registered multiple overlay icons, it's not uncommon to get into a situation where various applications are competing to have their overlay icons registered.

Overlay icons are registered by placing values in the following keys in the Registry:

  • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\ShellIconOverlayIdentifiers
  • HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\explorer\ShellIconOverlayIdentifiers (64-bit Windows only)
If necessary, it's up to the application (or better yet, the user) to decide whether or not to remove other entries; however, doing so too aggressively will likely have a negative impact on the user's experience with other applications.

Mounting Points

A mounting point is a name that can be used to access a volume. When the filesystem driver mounts a volume, it must make that volume accessible by creating one or more mounting points for it.

Windows:

Mounting points can be global (visible in all user sessions) or local (visible only to a specific user session). The add_mounting_point method creates global mounting points by default; applications must include the STGMP_LOCAL flag in the Flags parameter value to create local mounting points. (Note: The STGMP_MOUNT_MANAGER flag is not compatible with the STGMP_LOCAL flag.)

When creating a local mounting point, applications can specify a specific user session for it to be visible in by passing that session's Authentication ID for the AuthenticationId parameter (retrieval of Authentication IDs is discussed in a later section). If no Authentication ID is provided (i.e., 0 is passed), the local mounting point is created in the current user session; and if the application does this while running with elevated rights, then the local mounting point will only be visible in the elevated session, and consequently won't be available to applications in other sessions (such as, e.g., Windows Explorer).

When mounting points are added or removed, a system message (WM_DEVICECHANGE) is broadcast. It instructs Windows Explorer to refresh the list of drives available. However, these messages cannot cross user session boundaries; so if, for example, the application is running as a service, Windows Explorer may not receive the broadcast and thus fail to refresh the list of drives. To address this issue, CBFS Storage includes a Helper DLL which, among other things, helps ensure that Windows Explorer always refreshes the list of drives regardless of which user session the application is running in; please refer to that topic for more information.

Types of Mounting Points

There are a handful of different mounting point types, each of which exposes volumes in a slightly different manner:

  • Drive letter mounting points
  • Folder mounting points
  • Network mounting points
  • UNC path mounting points

Each type of mounting point is discussed in more detail below.

Drive Letter Mounting Points

Drive letter mounting points are one of the more commonly-used mounting point types thanks to users' familiarity with them. To create a drive letter mounting point, pass a string composed of a single character in the A-Z range followed by a colon (e.g., Z:) for the add_mounting_point method's MountingPoint parameter.

If the value passed for the add_mounting_point method's Flags parameter includes the STGMP_AUTOCREATE_DRIVE_LETTER flag, the class will assign a drive letter automatically. In this case, the value passed for the MountingPoint parameter must not include a drive letter.

Folder Mounting Points

A folder mounting point makes a volume accessible through a folder located on another (pre-existing) NTFS volume. Folder mounting points are always visible to all users in the system, and their creation requires administrative privileges.

To create a folder mounting point using the add_mounting_point method, include the STGMP_MOUNT_MANAGER flag in the Flags parameter , and pass the target folder's absolute path for the MountingPoint parameter (e.g., C:\MountedDrives\MyMountingPoint). The target folder must already exist, must reside on an NTFS volume, and must be empty; otherwise, the call will fail.

Authentication IDs

An Authentication ID is a locally unique identifier (LUID) assigned to a logon session (or, "user session"), retrievable through the access token that represents said session. Applications can obtain the Authentication ID of a session from an access token or by enumerating logon sessions.

To obtain an Authentication ID from an access token, call the Windows API's GetTokenInformation function and pass either TokenGroupsAndPrivileges or TokenStatistics for the TokenInformation parameter. The resulting value will be a reference to a structure (TOKEN_GROUPS_AND_PRIVILEGES or TOKEN_STATISTICS, respectively) containing the needed Authentication ID.

To enumerate logon sessions, use the Windows API's LsaEnumerateLogonSessions function, which returns a list of existing logon session IDs (that is, Authentication IDs). To obtain additional information about a particular logon session (e.g., in order to determine if it's the desired one), use the Windows API's LsaGetLogonSessionData function. Network Mounting Points

Network mounting points are similar to other mounting point types, except that the system treats them as "remote devices". This distinction is useful since:

  • Windows Explorer makes fewer requests for files located on remote devices.
  • Some applications are more tolerant of timeouts and delays when working with remote devices.

Therefore, when an application is designed to work with some slow or remote storage medium, it's recommended that it use a network mounting point. When using network mounting points, it's important that the Helper DLL be used so that Windows Explorer displays the correct drive status.

To create a network mounting point using the add_mounting_point method, include the STGMP_NETWORK flag in the Flags parameter, and pass a string of the form <Local Name>;<Server Name>;<Share Name> for the MountingPoint parameter.

  • <Local Name> is the name to use for the mounting point on the local system; it can be a drive letter or a name for use in a UNC path. Alternatively, it can be left empty, in which case the volume will only be accessible via the network path (see below) or the drive letter will be assigned automatically if the STGMP_AUTOCREATE_DRIVE_LETTER flag is used.
    • Note: This "local name" is not related to the concept of "local and global mounting points" discussed in the overview.
  • <Server Name> and <Share Name> are used to create a network path of the form \\<Server Name>\<Share Name>. This network path is not shared by default (see notes following examples below).

The set of characters allowed in server names, is defined in this document. The set of characters allowed in share names, is defined in this document.

With the above information in mind, here are some examples of valid MountingPoint parameter values when creating network mounting points:

  • Y:;MyServer;VirtualShare: Creates a network mounting point accessible both via the drive letter Y: and via the network path \\MyServer\VirtualShare.
  • MyMountingPoint;MyServer;VirtualShare: Creates a network mounting point accessible both via the UNC path \\.\MyMountingPoint and via the network path \\MyServer\VirtualShare
  • ;MyServer;VirtualShare: Creates a network mounting point accessible only via the network path \\MyServer\VirtualShare.

As stated above, the network paths created for network mounting points are not shared (i.e., visible to other computers on the network) by default. To have the class create an actual network share when add_mounting_point is called, applications must include either the STGMP_NETWORK_READ_ACCESS or the STGMP_NETWORK_WRITE_ACCESS flag in the Flags parameter value, and use empty string for the <Server Name> segment of the MountingPoint parameter value (the local computer's name is used). Note that when the mounting point is shared in this way, a local resource is created and then shared. The name of the resource is derived from the Share Name defined above. However, the set of allowed characters for such name is not strictly defined. Additionally, sharing is done using a call to NetShareAdd Windows API function, which can be called by Administrators, System Operators, and Power Users.

UNC Path Mounting Points

UNC path mounting points make a volume available via a specific name, and unlike other mounting points types, they are not displayed anywhere in Windows Explorer; the UNC path must already be known.

UNC path mounting points consist of the \\.\ prefix, followed by a name (e.g., \\.\CBDrive1). The mounting-point-related class methods expect just the name (i.e., the UNC path with the \\.\ prefix omitted). So to add a new UNC path mounting point like, e.g., \\.\CBDrive1, call the add_mounting_point method and pass CBDrive1 for the MountingPoint parameter.

Linux and macOS:

A virtual drive / filesystem is mounted to a directory, which must exist at the time of mounting and be empty; otherwise, the call will fail.

Helper DLL

This topic describes the functionality, available in CBVaultDrive and CBMemoryDrive classes when the class is used in Windows operating system.

The Helper DLL is integrated into Windows Explorer and offers functionality designed to provide users with a consistent and pleasant experience. It is recommended that the Helper DLL be installed alongside the system driver, which can be accomplished by including the MODULE_HELPER_DLL flag when calling the install method.

The Helper DLL is distributed in the same .cab file as the system driver; its name is CBVaultDriveShellHelper22.dll, and it is shipped in both 32-bit and 64-bit variants. Its functionality is described below.

Mounting Point Change Broadcasts

Anytime a mounting point is added or removed, the system driver will send a notification to the Helper DLL, which then broadcasts a system message instructing Windows Explorer to refresh the list of drives. Without this functionality, Windows Explorer will not refresh the list of drives if a mounting point is added or removed from a Windows service or another user session.

Network Mounting Point

When a network mounting point is used, the Helper DLL provides the functionality that allows Windows Explorer to correctly display the current status of, and interact with, the virtual drive. Without this functionality, the virtual drive will display as "Disconnected", which may result in unexpected behavior.

Custom Icons

When custom icons are used for a virtual drive, the Helper DLL ensures that they are properly displayed in Windows Explorer.

Threading and Concurrency

Through the use of multithreading, the CBVaultDrive class provides powerful concurrency features to help applications maximize their performance. For data integrity purposes, the class also strictly enforces the order in which events fire in certain situations, and allows applications to specify the extent to which events should be fired concurrently.

Please note that, even when configured for minimal concurrency, the class always fires events in the context of some worker thread, not in the thread the class was originally created on. Therefore, applications must be sure to synchronize operations between event handlers and other threads as necessary (including, but not limited to, calls to the class instance, unless a method is explicitly documented as callable within events).

Configuring Event Concurrency

Generally speaking, CBVaultDrive will always enforce per-file event serialization; that is, it always fires events relating to the same file in sequence (though technically-speaking, there is one optional exception to this behavior, discussed at the end of this section). For example, if there are multiple read or write operations pending against a given file, then an event will be fired for the first operation, and after its event handler has returned, another event will be fired for the second operation; and so on.

When the serialize_events property is set to seOnMultipleThreads, the MinWorkerThreadCount and MaxWorkerThreadCount configuration settings control the minimum and maximum number of worker threads the class can use for firing events. By default, both are set to 0, which indicates that the class's system driver should automatically choose appropriate values based on how many CPU cores the system has. These settings are both ignored if serialize_events is set to values other than seOnMultipleThreads.

PID Re-use

This topic describes the functionality, available in CBVaultDrive and CBMemoryDrive classes when the class is used in Windows operating system.

When using various rules that are based on process IDs (PID), you need to be aware that Windows tends to reuse PID numbers. Once the process with a certain PID is finished in any way, Windows can re-use this PID for another process being started. And it does reuse PIDs quite frequently for the purpose of keeping PID numbers low.

Such reuse can cause unexpected and sometimes unpleasant consequence for your application. To counteract it, you can take one or both actions:

  1. Open a handle to the process with the needed PID and not close it as long as your rule exists. Windows documentation states that as long as there exists an open handle to a process, its PID is not reused.
  2. Track completion of the process with the given PID (either by monitoring the state of the process by its handle or using CBProcess component of the CBFS Filter product) and once the process is finished, delete the corresponding rule.

Troubleshooting

This topic describes the functionality, available in CBVaultDrive and CBMemoryDrive classes when the class is used in Windows operating system.

Note: The below information applies to the operations of the class on Windows .

CBFS Storage is a complex product that operates in both user mode and kernel mode simultaneously; so when a serious issue occurs, it's critical that we are able to obtain sufficient information about the circumstances of the failure.

In order to help us assist you in a more expedient manner, please collect the information described in the instructions below when reporting a serious issue (i.e., one that causes the system to crash or hang). Our development team cannot effectively diagnose such issues without this information.

Also, please note that these sorts of issues commonly involve environmental differences and other factors that are either unforeseen or otherwise out of our control. It is also not unheard-of for a crash to appear attributable to one thing while in fact being caused by something completely different. Rest assured that we are committed to assisting you as best we can, and we thank you ahead-of-time for your patience and understanding throughout the support process.

System Crashes (BSODs)

If you encounter a consistently-reproducible system crash (BSOD) that you suspect may be due to CBFS Storage, please obtain a crash dump and include it when reporting the issue to us. Our development team is unable to diagnose system crashes without the information these dumps contain.

Ensure that your system is set up to generate crash dumps, and to not restart automatically after a crash, by following the steps found in Microsoft's Enabling a Kernel-Mode Dump File article. The options available in the memory dump dropdown vary depending on your version of Windows; please choose the first one from the following list that is present in yours:

  • Complete
  • Full
  • Automatic
  • Kernel

Once your system is set up to generate crash dumps, perform the same action that caused the BSOD originally to trigger the crash again. When it occurs, be sure to copy the information on the BSOD screen exactly so that it can be included in your submission (a picture of the screen in which all of the information is legible is also acceptable). Here are some examples of the specific information we're looking for:

Recent versions of Windows:

What failed: cbfs***22.sys Stop Code: FILE_SYSTEM

Older versions of Windows:

STOP: 0x00000022 (0x00240076, 0xF7A07AA8, 0xF7A077A8, 0xF7800C82) cbfs***22.sys - Address F7800C82 base at F77CD000, DateStamp 447d6975

After you've copied this information, reboot and check that the memory dump file was created at %SYSTEMROOT%\MEMORY.DMP (typically this is C:\Windows\MEMORY.DMP; if you changed the dump file location in the crash dump settings, check the location you specified instead). It will be a very large file that is too big to attach to an email, so please upload it to a file sharing site of your choice and generate a sharing link that our development team can use to download it.

Finally, submit a support issue to us that includes the link to your dump file, all of the information from the BSOD screen (if you took a picture, attach it or provide another sharing link), a description of how the BSOD was triggered, and any other information that you feel is relevant.

System Hangs

If you encounter a consistently-reproducible system hang that you suspect may be due to CBFS Storage, you'll need to collect the same information as described above. But in order to obtain a crash dump, you'll first need to configure your system so that you can trigger a crash from the keyboard once it hangs. To make this possible, follow these steps (adapted from Microsoft's Forcing a System Crash from the Keyboard article):

  1. First, using the instructions provided in the section above, configure your system to generate crash dumps, and to not restart automatically after a crash.
  2. Next, you must enable keyboard-initiated crashes in the registry by creating a new value named CrashOnCtrlScroll, and setting it equal to a REG_DWORD value of 0x01, in all of the following registry keys:
    • HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\i8042prt\Parameters
    • HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\kbdhid\Parameters
    • HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\hyperkbd\Parameters
  3. Finally, you must restart the system in order for these settings to take effect.

After these steps are complete, you'll be able to trigger a keyboard-initiated crash by using the following hotkey sequence: hold down the Right CTRL key, and press the SCROLL LOCK key twice.

At this point, you can perform the same action that caused the system to hang originally to trigger the hang again. Once the system hangs, use the hotkey sequence to force it to crash, and then follow the rest of the instructions from the section above to collect and submit the necessary information.

Application Crashes

Sometimes, an application crashes while the OS continues to operate, and the name of one of the modules of CBFS Storage is present in the crash information. A crashing application can be the one that uses CBFS Storage or some third-party process. If the crash occurs repeatedly, it is possible to make use of a User-Mode Crash Dump to locate or narrow down the source of the crash. Generation of crash dumps is disabled by default. Before you reproduce the crash, you need to Enable Collecting User-Mode Crash Dumps.

After you enable the crash dump, you don't need to reboot, you can proceed to reproduction of the crash immediately. After the crash re-occurs, you can pick the dump file from its location. The default locations of user-mode dump files are:

  • For regular applications: %LOCALAPPDATA%\CrashDumps
  • For System services: %WINDIR%\System32\Config\SystemProfile
  • For Network and Local services: %WINDIR%\ServiceProfiles
If you changed the dump file location in the crash dump settings in the Registry, check the location you specified instead.

A crash dump can be a large file (depending on the settings) that is too big to attach to an email, so please upload it to a file sharing site of your choice and generate a sharing link that our development team can use to download it.

Finally, submit a support issue to us that includes the link to your dump file, a description of how the BSOD or a manual crash was triggered, and any other information that you feel is relevant.

File Features

The topics in this section provide information about file-related features.

Topics

Alternate Streams

Every file stored in a CBFS Storage vault contains a primary stream of data that holds the file's contents. In addition to this primary stream, files in a vault may also contain one or more alternate streams of data whose contents are determined by the application. By taking advantage of the flexibility that alternate streams offer, applications can support a wide range of use-cases; for example:

  • Storing a file's metadata and/or security information.
  • Saving supplementary information associated with a file (e.g., song lyrics for music files).
  • Maintaining a history of file content revisions.
  • Providing multiple representations of the same file (e.g., HTML, RTF, and plain versions of text content).

Alternate streams are addressed using names like <FileName>:<StreamName>, so an alternate stream named "altstream" could be addressed as \path\to\filename.ext:altstream. Alternate streams can be created and accessed just like files using open_file, open_file_ex, delete_file, etc.; and can even be compressed and/or encrypted individually if an application desires.

To enumerate a file's alternate streams, call the find_first method with a mask like <FileName>:<StreamNameMask>. The <StreamNameMask> part can be * to enumerate all streams in a file. A file's main stream, which is always nameless, can be accessed explicitly using the name <FileName>: (note the trailing colon).

(Note: All API members discussed in this topic are available in both CBVaultDrive, CBMemoryDrive, and CBVAULT, unless otherwise noted.)

Compression

The CBFS Storage filesystem stores data in a vault as a series of one or more pages. To reduce space usage, CBFS Storage can compress files and alternate streams with an application-selected compression algorithm using the following mechanism, which is optimized to provide optimal performance for both sequential and random data access:

  1. A block of data composed of a specific number of pages is passed to the compression routine, which attempts to compress the data.
  2. If the compressed data can be stored using fewer pages than before, it is written to the vault. Otherwise, the original (uncompressed) data is written instead.
  3. Steps 1 and 2 are repeated until all of the data pages associated with the file or alternate stream have been processed.

Compressing Files and Alternate Streams

To specify a default compression mode for files and alternate streams, applications can set the default_file_compression property (and, if applicable, the DefaultFileCompressionLevel configuration setting). Additionally, the following methods allow applications to set a file or alternate stream's compression mode explicitly:

Using Custom Compression

CBFS Storage includes built-in support for zlib and RLE data compression. However, applications can also choose to provide their own custom compression implementation using the on_data_compress and on_data_decompress events.

Supported Compression Modes

CBFS Storage supports the following compression modes:

VAULT_CM_NONE0Don't use compression.

VAULT_CM_DEFAULT1Use default compression (zlib).

VAULT_CM_CUSTOM2Use event-based custom compression.

Compression level is not used.

VAULT_CM_ZLIB3Use zlib compression.

Valid compression levels are 1-9.

VAULT_CM_RLE4Use RLE compression.

Compression level is not used.

File Tags

CBFS Storage allows applications to attach arbitrary metadata to any file, directory, or alternate stream using file tags. There are two kinds of file tags, both of which are stored as key-value pairs:

  1. Raw file tags use numeric Ids as keys and store raw binary data.
    • Valid Id values are those in the range 0x0001 to 0xCFFF (inclusive).
    • A tag should contain at least one (1) byte of data.
    • The maximum size of a raw file tag's binary data is 65531 bytes.
  2. Typed file tags use string keys and store typed values.
    • Names may be up to 4095 characters long (not including the null terminator), and are stored in UTF-16LE format internally.
    • The maximum size of a typed file tag's value is 65529 - (name_length * 2) bytes (where name_length is measured in characters, including the null terminator).

Each file, directory, and alternate stream can have up to 1024 typed file tags and 53247 raw file tags attached to it at once. The following methods are used to manage and interact with file tags:

Applications can also use the find_first_by_query method to search for files and directories whose file tags match a specified query; please refer to that method's documentation, as well as the Query Language topic, for more information. Note that the query language only works with typed file tags; it does not support raw file tags.

(Note: All API members discussed in this topic are available in both CBVaultDrive, CBMemoryDrive, and CBVAULT, unless otherwise noted.)

Query Language

In addition to searching by name, applications can search for files and directories based on their File Tags (metadata) using the CBFS Storage query language.

The query language includes a wide variety of Language Elements and supports all common Data Types. Search queries are interpreted as UTF-16LE strings, and may contain any valid arrangement of language elements, typed file tag names, and constants. For example:

  • From = 'John Smith': Selects all files received from John Smith.
    • From is the name of a file tag.
    • = is the equality operator (== is also supported).
    • 'John Smith' is a string constant.
  • SendData - Today > 3: Selects all files which were sent over 3 days ago.
    • SendData is the name of a file tag.
    • - is the subtraction operator.
    • Today is an intrinsic constant which returns the current system date.
    • > is the greater than operator.
    • 3 is a numeric constant.

When parsing an expression from a search query, the query engine converts all of its operands to the same data type using a specific set of rules; please refer to the Type Conversion topic for more information.

To find the first match for a query, call the find_first_by_query method, passing the desired search query for the Query parameter; and then call find_next to find other matches, if necessary. Be sure to call find_close when finished so that the class can release the resources allocated for the search operation.

(Note: All API members discussed in this topic are available in both CBVaultDrive, CBMemoryDrive, and CBVAULT, unless otherwise noted.)

Please refer to the File Tags topic for more information about how to manage file tags, and please refer to the other topics in this section for more information about the query language:

Language Elements

The CBFS Storage query language supports a wide variety of language elements, all of which are described below.

Logical Operators

Operator Operand type(s) Description
NOT, !, ~ Boolean Logical negation (NOT)
NOT, !, ~ Number Bitwise NOT
AND, & Boolean Logical AND
AND, & Number Bitwise AND
OR, | Boolean Logical OR
OR, | Number Bitwise OR

Arithmetic Operators

Operator Operand type(s) Description
+ Number, DateTime Addition
+ String String concatenation
- Number Negation
- Number, DateTime Subtraction
* Number Multiplication
/ Number Division (attempting to divide by zero will cause an exception)

Addition and subtraction operations involving DateTime operands behave as follows:

  • When adding a Number (n) and a DateTime, the result is a DateTime whose value has increased by n whole days.
  • When subtracting a Number (n) from a DateTime, the result is a DateTime whose value has decreased by n whole days.
  • When subtracting a DateTime from another DateTime, the result is a Number which reflects the difference as a number of whole days. The query evaluator converts both operands to whole days before performing the subtraction; "leftover" time is truncated as part of the conversion.

Relational Operators

Operator Operand type(s) Description
=, == All types Equal to
<>, != All types Not equal to
< All types Less than
> All types Greater than
<= All types Less than or equal to
>= All types Greater than or equal to

Conditions

Condition Operand type Description
IS [NOT] NULL All types Returns True if the value is/isn't NULL, and False otherwise.
IS [NOT] True Boolean Returns True if the value is/isn't True, and False otherwise.
IS [NOT] False Boolean Returns True if the value is/isn't False, and False otherwise.
[NOT] LIKE '...' [ESCAPE '...'] String Returns True if the value does/doesn't match the specified pattern; see notes below.

Keep the following in mind when using the LIKE condition:

  • Two kinds of wildcards are supported: %, which matches a string of any length; and _, which matches any single character. For example:
    • From LIKE '% Smith': Selects all files received from people with the last name "Smith".
    • From LIKE 'John Sm_th': Selects all files received from people with the first name "John" and a last name that is five characters long, begins with "Sm", and ends with "th" (Smith, Smyth, Smeth, etc.).
  • To search for values which include wildcard characters, the optional ESCAPE parameter can be used to specify a wildcard escape character. For example:
    • From LIKE 'John!_Smith' ESCAPE '!': Selects all files received from "John_Smith".
    • From LIKE 'John!_%' ESCAPE '!': Selects all files received from a name that begins with "John_".

File Variables

File variables represent some piece of information about the current file the query is being evaluated against.

Variable Type Description
FileName String The name of the current file.
FullName String The fully-qualified name of the current file, starting from the root directory /.
Path String The full path to the current file, including the final path separator (not including the file name).
IsFile Boolean True if the current file is not a directory, and False otherwise.
IsDirectory Boolean True if the current file is a directory, and False otherwise.
IsLink Boolean True if the current file is a symbolic link, and False otherwise.
LinkDestination String If the current file is a symbolic link, the link's target; otherwise acts the same as FullName.
CreationTime DateTime The current file's creation date and time.
LastAccessTime DateTime The current file's last access date and time.
ModificationTime DateTime The current file's last modification date and time.
Size Number The size of the current file (always 0 for directories).
Attributes Number The current file's attribute, encoded as a number.
IsEncrypted Boolean True if the current file is encrypted, and False otherwise.
IsCompressed Boolean True if the current file is compressed, and False otherwise. May be True for directories that contain files compressed by default.

Intrinsics

"Intrinsics" are the functions and constants built into the query language.

Intrinsic Operand type(s)Return typeDescription
D(value) String DateTime Converts a String to a DateTime; please refer to the Type Conversion topic for more information.
IsNull(value) All types Boolean Returns True if the value is NULL, and False otherwise.
IsNotNull(value) All types Boolean Returns True if the value is not NULL, and False otherwise.
Min(value1, value2) All types All types Returns the smaller of the two values.
Max(value1, value2) All types All types Returns the larger of the two values.
Now DateTime Returns the current system date and time.
Today DateTime Returns the current system date.
True Boolean Boolean True.
False Boolean Boolean False.

Precedence

The table below lists the query language's elements in order of descending precedence. Any legal expression within a query string may be surrounded with parentheses () in order to override precedence or increase readability.

PrecedenceLanguage Elements
1

All File Variables

All Intrinsics (except D(value); see note)

2

-: Arithmetic negation

NOT, !, ~: Logical/bitwise negation

D(value): Explicit String to DateTime conversion

3

*: Multiplication

/: Division

4

+: Addition/string concatenation

-: Subtraction

5

=, ==: Equal to

<>, !=: Not equal to

<: Less than

>: Greater than

<=: Less than or equal to

>=: Greater than or equal to

IS [NOT], [NOT] LIKE: All Conditions

6AND, &: Logical/bitwise AND
7OR, |: Logical/bitwise OR

Note: The query engine treats the D(value) function as an operator, so its precedence is lower than the other intrinsics.

Data Types

The CBFS Storage query language supports the following operand data types:

Type Description
NULL Empty value. Operations with NULL operand(s) always result with NULL.
Boolean Boolean; either False or True (and False < True).
String String of UTF-16LE (2-byte Unicode) characters.
DateTime Describes the date and time.
Number Signed 64-bit integer.

Type Conversion

When CBFS Storage parses a query, it will attempt to convert operands to the same type before evaluating an expression. The right-hand operand is converted to match the type of the left-hand operand if possible; otherwise, if the right-hand operand is of the String type, the left-hand operand is converted to String.

Supported Data Type Conversions

Convert To Convert From Notes
String Number, Boolean, NULL Typical conversion rules apply; Boolean values become "True" or "False".
String DateTime The format string used by the conversion is YYYY-MM-DD hh:mm:ss.fff.
Boolean String "True" and "False" are the recognized string values.
DateTime String The parsing pattern used by the conversion is YYYY[-]MM[-]DD[[tT ]hh[[:]mm[[:]ss[.fff]]]]; see notes below.
Number String The conversion recognizes numbers formatted as signed base-10 integers.

In addition to the implicit conversion mentioned above, a String can be converted to a DateTime explicitly using the intrinsic function D(value). The implicit and explicit conversions both use the parsing pattern shown above, which has a number of optional parts:

  • The date separators - may be omitted if the month and day are both two-digit values. They must both be present if the month and/or day is a single-digit value.
  • The time portion may be omitted; if present, it must be specified as one of the following: hours only; hours and minutes; hours, minutes, and seconds; or hours, minutes, seconds, and milliseconds.
    • The time separators : may be omitted if all included time elements are two-digit values. They must be present if any time elements are single-digit values.
    • Milliseconds, if present, must always be separated by a . character, and must always be a three-digit value.
  • When the time portion is present, it may immediately follow the date portion (i.e., with no separator), or it may be separated from the date portion using a T, a t, or a single space.

Vaults

What is a Vault?

The key functionality CBFS Storage provides is the ability to create and store an entire filesystem (complete with files, directories, metadata, and much more) in a standalone container called a vault.

A vault is typically stored as a real file on a local disk (similar to, e.g., an SQLite database file), but applications can technically store it using any data location by using Callback Mode.

Internally, a vault's storage space is divided up into chunks of equal size called pages. A vault's page size is specified at creation-time, and cannot be changed later. Applications do not have direct access to vault pages, but awareness of their existence is helpful for understanding certain class APIs.

Vaults can be created and accessed using both the CBVAULT and CBVaultDrive classes; but only the latter allows a vault to be mounted as a virtual drive. Please refer to the other topics in this section for more information about vaults:

Multipart Vaults

CBFS Storage is capable of storing a single vault across multiple files on disk; this is known as a multipart vault. To create a multipart vault, set the PartSize configuration setting to a non-zero value before the vault is created. CBFS Storage will automatically create, resize, and delete individual part files as necessary over time (please refer to the Vault Size topic for more information).

Multipart vaults are typically used by applications which operate in environments with file size constraints. For example, if an application needed to store a 16GB vault on a FAT32 filesystem, it could use a multipart vault with a 4GB part size.

Existing vaults cannot be converted between multipart and non-multipart, and a multipart vault's size cannot be changed after creation. Also, multipart vaults are not supported in Callback Mode (since it already gives applications full control over how/where a vault is stored); the PartSize configuration setting is simply ignored.

(Note: All API members discussed in this topic are available in both CBVaultDrive, CBMemoryDrive, and CBVAULT, unless otherwise noted.)

Using RootData

All CBFS Storage vaults contain a special data stream called RootData which can be used for application-defined purposes. A vault's RootData stream can only be accessed using the open_root_data method, because the stream itself is not part of the vault's filesystem hierarchy. The standalone nature of the RootData stream means that:

The RootData stream is also exempt from whole-vault encryption. This exemption is intentional; it allows applications which utilize whole-vault encryption to store information about said encryption (e.g., encrypted session keys, certificates, access control lists (ACLs), etc.) within the vault itself, thus simplifying application design.

(Note: All API members discussed in this topic are available in both CBVaultDrive, CBMemoryDrive, and CBVAULT, unless otherwise noted.)

Vault Corruption

CBFS Storage vaults have a complex internal structure which may become corrupted if a vault is not closed properly or if some operation is interrupted. Typically, such things are caused by an application crash, a system crash, or (when operating in Callback Mode) due to an error in an event handler. Corruption can also occur if a vault's raw data is modified externally, either intentionally or due to storage failure.

When a vault is open, the is_corrupted property can be queried to determine if has been corrupted. If a vault is corrupted, any operation may fail with a VAULT_ERR_VAULT_CORRUPTED error code.

Applications can attempt to fix a corrupted vault by calling the check_and_repair method. Always create a vault backup before calling check_and_repair, as it's possible for data to be lost during the repair process in cases of severe corruption.

Journaling

To reduce the chances of vault corruption in the event of a crash, CBFS Storage can make use of journaling. Journaling works by wrapping vault modification operations in transactions, as follows:

  1. A new transaction is opened by writing information about a change to a journal located within the vault.
  2. The changes themselves are written to the vault.
  3. The transaction is committed by writing another entry in the journal.

If a crash occurs, any interrupted modification operations will appear in a vault's journal as pending transactions. The next time CBFS Storage opens the vault, it will discover any pending transactions and automatically to recover them. During the transaction recovery process, each transaction is either committed or rolled back, depending on its last known state.

Overall, journaling is an effective technique for maintaining data integrity. However, keep the following considerations in mind:

  • When journaling is enabled, all file data changes incur additional write operations; this has a significant impact on overall write performance.
  • Journaling does not provide any kind of data redundancy or consistency; it cannot protect against corruption caused by bit-rot, storage failures, or external modification of a vault's physical data.

CBFS Storage implements journaling as an operational mode rather than a vault attribute, so there is no such thing as a "journaled vault" or a "non-journaled vault". Applications control whether journaling mode is used by setting the JournalingMode parameter of the open_vault methodwhen opening a vault. Therefore, the same vault might be opened with journaling enabled at one point, and opened without journaling enabled at another point.

The filesystem engine will always perform transaction recovery when a vault is opened (if there are pending transactions in its journal), even if journaling is disabled.

(Note: All API members discussed in this topic are available in both CBVaultDrive, CBMemoryDrive, and CBVAULT, unless otherwise noted.)

Vault Size

By default, a vault grows automatically as more data is written to it, and shrinks automatically when its free space percentage reaches the threshold defined by the auto_compact_at property.

Applications can use the following properties to both control, and obtain information about, a vault's size. Please refer to each one's documentation for more information.

  • vault_size_max: Specifies the maximum size a vault can be; 0 (unlimited) by default.
  • vault_size_min: Specifies the minimum size a vault can be; 0 by default.
  • vault_size: Reflects a vault's actual size; and can also be used to explicitly resize a vault, keeping in mind the following:
    • A vault cannot shrink more than its available free space allows (i.e., not by more than vault_free_space bytes).
    • A vault cannot shrink beyond vault_size_min bytes.
    • If vault_size_max is not 0 (unlimited), a vault cannot grow beyond vault_size_max bytes.
    • If a vault grows enough to reach/exceed its auto_compact_at threshold, it will automatically shrink again when the next automatic compaction occurs.
  • vault_free_space: Reflects the actual amount of free space a vault has available.
  • possible_size: Reflects the maximum size a vault could possibly be.
  • possible_free_space: Reflects the maximum amount of free space a vault could possibly have available.

(Note: All API members discussed in this topic are available in both CBVaultDrive, CBMemoryDrive, and CBVAULT, unless otherwise noted.)

For CBVaultDrive, please note that the size of a vault backed by a storage volume or partition (i.e., one created/opened using the format_volume/open_volume methods) is identical to the size of said storage volume or partition itself. Such vaults, once created, cannot be resized using any of the properties or methods discussed above.

Constants

All constants are defined in the cbfsstorage module.

Error Codes

VAULT_ERR_INVALID_VAULT_FILE -1 The specified file is not a CBFS Storage vault.

VAULT_ERR_INVALID_PAGE_SIZE -2 The specified page size is not valid.

VAULT_ERR_VAULT_CORRUPTED -3 The vault is corrupted.

VAULT_ERR_TOO_MANY_TRANSACTIONS -4 Too many transactions active.

VAULT_ERR_FILE_ALREADY_EXISTS -5 A file, directory, symbolic link, or alternate stream with the specified name already exists.

VAULT_ERR_TRANSACTIONS_STILL_ACTIVE -6 One or more transactions are still active.

VAULT_ERR_TAG_ALREADY_EXISTS -7 The specified file tag already exists.

VAULT_ERR_FILE_NOT_FOUND -8 The specified file, directory, symbolic link, or alternate stream was not found.

VAULT_ERR_PATH_NOT_FOUND -9 The specified path was not found.

VAULT_ERR_SHARING_VIOLATION -10 The specified file or alternate stream is already open in an exclusive access mode.

VAULT_ERR_SEEK_BEYOND_EOF -11 Cannot seek beyond the end of a file or alternate stream.

VAULT_ERR_NO_MORE_FILES -12 There are no other files, directories, symbolic links, or alternate streams that match the search criteria.

VAULT_ERR_INVALID_FILE_NAME -13 The specified name is not valid.

VAULT_ERR_VAULT_ACTIVE -14 The requested operation cannot be performed while a vault is open.

VAULT_ERR_VAULT_NOT_ACTIVE -15 A vault must be open before the requested operation can be performed.

VAULT_ERR_INVALID_PASSWORD -16 The specified password is incorrect.

VAULT_ERR_VAULT_READ_ONLY -17 The requested operation cannot be performed; the vault is open in read-only mode.

VAULT_ERR_NO_ENCRYPTION_HANDLERS -18 Can't use custom encryption; no custom encryption event handlers provided.

VAULT_ERR_OUT_OF_MEMORY -19 Out of memory.

VAULT_ERR_SYMLINK_DESTINATION_NOT_FOUND -20 A symbolic link's destination file could not be found.

VAULT_ERR_FILE_IS_NOT_SYMLINK -21 The specified file is not a symbolic link.

VAULT_ERR_BUFFER_TOO_SMALL -22 The specified buffer is too small to hold the requested value.

VAULT_ERR_BAD_COMPRESSED_DATA -23 Decompression failed (possible due to corruption).

VAULT_ERR_INVALID_PARAMETER -24 Invalid parameter.

VAULT_ERR_VAULT_FULL -25 The vault is full (and cannot be automatically resized).

VAULT_ERR_INTERRUPTED_BY_USER -26 Operation interrupted by user.

VAULT_ERR_TAG_NOT_FOUND -27 The specified file tag was not found.

VAULT_ERR_DIRECTORY_NOT_EMPTY -28 The specified directory is not empty.

VAULT_ERR_HANDLE_CLOSED -29 The file or alternate stream was closed unexpectedly; the handle is no longer valid.

VAULT_ERR_INVALID_STREAM_HANDLE -30 Invalid file or alternate stream handle.

VAULT_ERR_FILE_ACCESS_DENIED -31 Access denied.

VAULT_ERR_NO_COMPRESSION_HANDLERS -32 Can't use custom compression; no custom compression event handlers provided.

VAULT_ERR_NOT_IMPLEMENTED -33 Not implemented in this version of CBFS Storage.

VAULT_ERR_DRIVER_NOT_INSTALLED -35 The CBFS Storage system driver has not been installed.

VAULT_ERR_NEW_VAULT_VERSION -37 The specified vault cannot be opened, it was created using a newer version of CBFS Storage.

VAULT_ERR_FILE_IS_NOT_DIRECTORY -38 The specified file is not a directory.

VAULT_ERR_INVALID_TAG_DATA_TYPE -39 The specified file tag data type is not valid.

VAULT_ERR_VAULT_FILE_DOES_NOT_EXIST -40 The specified vault storage file does not exist.

VAULT_ERR_VAULT_FILE_ALREADY_EXISTS -41 The specified vault storage file already exists.

VAULT_ERR_CALLBACK_MODE_FAILURE -42 Some callback mode event handler has returned an unidentified error.

VAULT_ERR_EXTERNAL_ERROR -43 External library could not be initialized or used.

File Attributes

VAULT_FATTR_FILE 0x00000001 The entry is a file.

VAULT_FATTR_DIRECTORY 0x00000002 The entry is a directory.

VAULT_FATTR_DATA_STREAM 0x00000004 The entry is an alternate data stream.

VAULT_FATTR_COMPRESSED 0x00000008 The file or stream is compressed.

VAULT_FATTR_ENCRYPTED 0x00000010 The file or stream is encrypted.

VAULT_FATTR_SYMLINK 0x00000020 The entry is a symbolic link.

VAULT_FATTR_READONLY 0x00000040 The file is read-only.

This attribute is not used by CBFS Storage, but it can be set and retrieved.

VAULT_FATTR_ARCHIVE 0x00000080 The file requires archiving.

This attribute is not used by CBFS Storage, but it can be set and retrieved.

VAULT_FATTR_HIDDEN 0x00000100 The file is hidden.

This attribute is not used by CBFS Storage, but it can be set and retrieved.

VAULT_FATTR_SYSTEM 0x00000200 The file is a system file.

This attribute is not used by CBFS Storage, but it can be set and retrieved.

VAULT_FATTR_TEMPORARY 0x00000400 The file is temporary.

This attribute is not used by CBFS Storage, but it can be set and retrieved.

VAULT_FATTR_DELETE_ON_CLOSE 0x00000800 The file should be deleted when the last handle to the file is closed.

This attribute is currently not supported by CBFS Storage.

VAULT_FATTR_RESERVED_0 0x00001000 Reserved.

VAULT_FATTR_RESERVED_1 0x00002000 Reserved.

VAULT_FATTR_RESERVED_2 0x00004000 Reserved.

VAULT_FATTR_RESERVED_3 0x00008000 Reserved.

VAULT_FATTR_NO_USER_CHANGE 0x0000F03F A mask which includes all attributes that cannot be changed.

Applications cannot use the set_file_attributes method to directly change any of the following attributes: FILE, DIRECTORY, DATA_STREAM, COMPRESSED, ENCRYPTED, SYMLINK, RESERVED_0, RESERVED_1, RESERVED_2, RESERVED_3.

VAULT_FATTR_USER_DEFINED 0x7FF00000 A mask for application-defined attributes.

Applications can use the set_file_attributes method to set custom attributes, so long as their values are covered by this mask.

VAULT_FATTR_ANY_FILE 0x7FFFFFFF A mask which includes any and all attributes.

Check and Repair Flags

VAULT_CR_CHECK_ONLY 0x00000001 Check only, don't attempt any repairs.

VAULT_CR_CHECK_ALL_PAGES 0x00000002 Check all vault pages, including empty ones.

(When this flag is not present, only the vault pages which are marked as occupied are checked.)

Format Flags

VAULT_FMF_FAST_FORMAT 0x00000001 Perform a fast format; only initialize the pages necessary for storing the filesystem structure.

When this flag is not set, all pages of the new vault are initialized.

Vault Journaling Modes

VAULT_JM_NONE 0 No journaling is used.

This mode ensures the fastest operations but if the application crashes, corruption of the vault is possible.

VAULT_JM_METADATA 1 Journaling is used only for metadata (filesystem structure and directory contents).

This mode is a balance between speed and reliability.

VAULT_JM_FULL 2 Journaling is used for both filesystem structure and file data and metadata.

This is the slowest but the most reliable option.

Search Flags

VAULT_FF_NEED_NAME 0x00000001 Include entry names (without paths) when returning search results.

VAULT_FF_NEED_FULL_NAME 0x00000002 Include fully-qualified entry names when returning search results.

VAULT_FF_NEED_ATTRIBUTES 0x00000004 Include entry attributes when returning search results.

VAULT_FF_NEED_SIZE 0x00000008 Include entry sizes when returning search results.

VAULT_FF_NEED_METADATA_SIZE 0x00000010 Include entry metadata sizes when returning search results.

VAULT_FF_NEED_TIMES 0x00000020 Include entry times when returning search results.

VAULT_FF_NEED_LINK_DEST 0x00000040 Include symbolic link destinations when returning search results.

VAULT_FF_EMULATE_FAT 0x00001000 Inserts . and .. pseudo-entries into search results for all directories except the root one.

VAULT_FF_RECURSIVE 0x00002000 Search recursively in all subdirectories.

VAULT_FF_CASE_INSENSITIVE 0x00004000 Forces case-insensitive search, even if the vault is case-sensitive.

Vault Open Modes

VAULT_OM_CREATE_NEW 0 Creates a new vault if possible, failing if one already exists.

VAULT_OM_CREATE_ALWAYS 1 Creates a new vault, overwriting an existing one if necessary.

VAULT_OM_OPEN_EXISTING 2 Opens a vault if it exists; fails otherwise.

VAULT_OM_OPEN_ALWAYS 3 Opens a vault if it exists; creates a new one otherwise.

Vault State Flags

VAULT_ST_FIXED_SIZE 0x00000001 The vault is fixed-size.

VAULT_ST_READ_ONLY 0x00000002 The vault was opened in read-only mode.

Please refer to the read_only property for more information.

VAULT_ST_CORRUPTED 0x00000004 The vault is corrupted.

Applications can use the check_and_repair method to try to repair vault corruption. Please refer to the Vault Corruption topic for more information.

VAULT_ST_TRANSACTIONS_USED 0x00000008 The vault was opened in journaling mode.

Please refer to the use_journaling property for more information.

VAULT_ST_ACCESS_TIME_USED 0x00000010 Last access times are being tracked.

Please refer to the use_access_time property for more information.

VAULT_ST_ENCRYPTED 0x00000020 The vault is encrypted with whole-vault encryption.

Please refer to the Encryption topic for more information.

VAULT_ST_VALID_PASSWORD_SET 0x00000040 The correct whole-vault encryption password has been provided.

Please refer to the Encryption topic for more information.

VAULT_ST_PHYSICAL_VOLUME 0x00000080 The vault is backed by a storage volume or partition formatted with the CBFS Storage filesystem.

This flag only applies when using the CBVaultDrive class.

VAULT_ST_PARTED 0x00000100 The vault's contents are split across multiple files on disk.

Please refer to the Multipart Vaults topic for more information.

Tag Data Types

VAULT_TDT_RAWDATA 0x0 The tag is untyped and must be addressed by Id.

VAULT_TDT_BOOLEAN 0x1 The tag contains Boolean data and must be addressed by name.

VAULT_TDT_STRING 0x2 The tag contains String (UTF-16LE) data and must be addressed by name.

VAULT_TDT_DATETIME 0x3 The tag contains DateTime data and must be addressed by name.

VAULT_TDT_NUMBER 0x4 The tag contains numeric (signed 64-bit) data and must be addressed by name.

VAULT_TDT_ANSISTRING 0x5 The tag contains AnsiString (8-bit string) data and must be addressed by name.

Path Separator Characters

VAULT_PSC_BACKSLASH 92 Backslash ('\\').

Windows path separator.

VAULT_PSC_SLASH 47 Forward slash ('/').

Unix-style path separator.

Compression Modes

VAULT_CM_NONE 0 Don't use compression.

VAULT_CM_DEFAULT 1 Use default compression (zlib).

VAULT_CM_CUSTOM 2 Use event-based custom compression.

Compression level is not used.

VAULT_CM_ZLIB 3 Use zlib compression.

Valid compression levels are 1-9.

VAULT_CM_RLE 4 Use RLE compression.

Compression level is not used.

Encryption Modes

VAULT_EM_NONE 0x0 Don't use encryption.

VAULT_EM_DEFAULT 0x1 Use default encryption (VAULT_EM_XTS_AES256_PBKDF2_HMAC_SHA256).

VAULT_EM_XTS_AES256_PBKDF2_HMAC_SHA256 0x2 Use AES256 encryption with PBKDF2 key derivation based on a HMAC_SHA256 key hash.

VAULT_EM_CUSTOM256_PBKDF2_HMAC_SHA256 0x3 Use event-based custom 256-bit encryption with PBKDF2 key derivation based on a HMAC_SHA256 key hash.

256-bit (32-byte) block size.

VAULT_EM_CUSTOM512_PBKDF2_HMAC_SHA256 0x4 Use event-based custom 512-bit encryption with PBKDF2 key derivation based on a HMAC_SHA256 key hash.

512-bit (64-byte) block size.

VAULT_EM_CUSTOM1024_PBKDF2_HMAC_SHA256 0x5 Use event-based custom 1024-bit encryption with PBKDF2 key derivation based on a HMAC_SHA256 key hash.

1024-bit (128-byte) block size.

VAULT_EM_CUSTOM256_CUSTOM_KEY_DERIVE 0x23 Use event-based custom 256-bit encryption with custom key derivation.

256-bit (32-byte) block size.

VAULT_EM_CUSTOM512_CUSTOM_KEY_DERIVE 0x24 Use event-based custom 512-bit encryption with custom key derivation.

512-bit (64-byte) block size.

VAULT_EM_CUSTOM1024_CUSTOM_KEY_DERIVE 0x25 Use event-based custom 1024-bit encryption with custom key derivation.

1024-bit (128-byte) block size.

VAULT_EM_CUSTOM256_DIRECT_KEY 0x43 Use event-based custom 256-bit encryption with no key derivation.

256-bit (32-byte) block size. Useful for cases where the password is an identifier for an external key and should not be used for key derivation.

VAULT_EM_CUSTOM512_DIRECT_KEY 0x44 Use event-based custom 512-bit encryption with no key derivation.

512-bit (64-byte) block size. Useful for cases where the password is an identifier for an external key and should not be used for key derivation.

VAULT_EM_CUSTOM1024_DIRECT_KEY 0x45 Use event-based custom 1024-bit encryption with no key derivation.

1024-bit (128-byte) block size. Useful for cases where the password is an identifier for an external key and should not be used for key derivation.

VAULT_EM_UNKNOWN 0xFF Unidentified or unknown encryption.

File Open Modes

VAULT_FOM_CREATE_NEW 0 Creates a new file or alternate stream if possible, failing if one already exists.

VAULT_FOM_CREATE_ALWAYS 1 Creates a new file or stream, overwriting an existing one if necessary.

VAULT_FOM_OPEN_EXISTING 2 Opens a file or stream if it exists; fails otherwise.

VAULT_FOM_OPEN_ALWAYS 3 Opens a file or stream if it exists; creates a new one otherwise.

Progress Operations

VAULT_PO_FORMATTING 0 Formatting a vault.

VAULT_PO_CHECKING_1 1 Checking a vault (stage 1).

VAULT_PO_CHECKING_2 2 Checking a vault (stage 2).

VAULT_PO_CHECKING_3 3 Checking a vault (stage 3).

VAULT_PO_CHECKING_4 4 Checking a vault (stage 4).

VAULT_PO_CHECKING_5 5 Checking a vault (stage 5).

VAULT_PO_PAGE_CORRUPTED 8 Processing a corrupted vault page.

VAULT_PO_PAGE_ORPHANED 9 Processing an orphaned vault page.

VAULT_PO_COMPRESSING 10 Compressing a file or alternate stream.

VAULT_PO_DECOMPRESSING 11 Decompressing a file or alternate stream.

VAULT_PO_ENCRYPTING 12 Encrypting a vault, file, or alternate stream.

VAULT_PO_DECRYPTING 13 Decrypting a vault, file, or alternate stream

VAULT_PO_COMPACTING 14 Compacting a vault.

VAULT_PO_RESIZING 15 Resizing a vault.

VAULT_PO_CALCULATING_SIZE 16 Calculating a vault's size.

VAULT_PO_COPYING_FILES_TO_VAULT 17 Copying files to a vault.

VAULT_PO_COPYING_FILES_FROM_VAULT 18 Copying files from a vault.

Copy Flags

VAULT_CFF_OVERWRITE_NONE 0x00000000 Never overwrite destination files.

VAULT_CFF_OVERWRITE_IF_NEWER 0x00000001 Overwrite a destination file only if the source file is newer.

VAULT_CFF_OVERWRITE_ALL 0x00000002 Always overwrite destination files.

VAULT_CFF_INCLUDE_SUBDIRS_WITH_CONTENTS 0x00010000 Include all subdirectories in source directory, and their contents, recursively.

VAULT_CFF_INCLUDE_SUBDIRS_NO_CONTENTS 0x00020000 Include all subdirectories in the source directory, without their contents.

VAULT_CFF_FIRE_COPY_EVENTS 0x40000000 Fire events related to file copying.

When the flag is set, the class fires the on_file_before_copy and on_file_after_copy events.

Module Flags

MODULE_DRIVER_PNP_BUS 0x00000001 PnP Bus Driver (.sys file).

This module must be installed if the application wishes to make use of Plug-and-Play (PnP) storage features class in Windows. PnP storage devices are those visible as disks in the Device Manager, and the system treats such storage devices differently from other purely virtual devices.

The virtual disk driver must be re-installed anytime this module is added or removed.

MODULE_DRIVER_BLOCK 0x00000002 Virtual disk driver (.sys file).

The product's virtual disk driver module, which provides core functionality; it must be installed for the class to function correctly.

MODULE_DRIVER_FS 0x00000004 Filesystem driver (.sys file).

The product's filesystem driver module, which provides core functionality; it must be installed for the class to function correctly.

MODULE_HELPER_DLL 0x00010000 Shell Helper DLL (CBVaultDriveShellHelper2022.dll)

This module provides supplementary functionality for the class; please refer to the Helper DLL topic for more information.

Note: Not applicable when calling the get_driver_status method.

Desired Access Flags

STG_DACCESS_READ 0x00000001 Grant/deny read access.

STG_DACCESS_WRITE 0x00000002 Grant/deny write access.

STG_DACCESS_READWRITE 0x00000003 Grant/deny read and write access.

Install Flags

INSTALL_REMOVE_OLD_VERSIONS 0x00000001 Uninstall drivers and helper DLLs from previous component versions (e.g., 2017).

INSTALL_KEEP_START_TYPE 0x00000002 Keep the driver's current start type setting in the registry.

If this flag is not set (default), the installation logic will reset the driver's start type setting in the Windows registry to the default value. Setting this flag causes the installation logic to preserve the current value, which may be necessary if the user (or the application itself) set it previously.

INSTALL_OVERWRITE_SAME_VERSION 0x00000004 Install files when their version is the same as the version of already installed files.

If this flag is not set (default), the installation logic will overwrite the existing file only if the version number of the file being installed is larger than the version of the file being overwritten. Setting this flag causes the installation logic to overwrite the file even when it has the same version.

Uninstall Version Flags

UNINSTALL_VERSION_PREVIOUS 0x00000001 Uninstall modules from previous product versions.

UNINSTALL_VERSION_CURRENT 0x00000002 Uninstall modules from the current product version.

UNINSTALL_VERSION_ALL 0x00000003 Uninstall modules from all product versions.

Module Status Flags

MODULE_STATUS_NOT_PRESENT 0x00000000 The specified module is not present on the system.

MODULE_STATUS_STOPPED 0x00000001 The specified module is in the Stopped state.

MODULE_STATUS_RUNNING 0x00000004 The specified module is loaded and running.

Mounting Point Flags in Windows

STGMP_SIMPLE 0x00010000 Create a simple mounting point.

Simple mounting points may be local or global; and when local, can be made visible in either the current user session or another one.

This flag cannot be combined with STGMP_MOUNT_MANAGER or STGMP_NETWORK, and is implied if neither of those flags are present.

STGMP_MOUNT_MANAGER 0x00020000 Create a mounting point that appears to the system as a physical device.

When the storage_type property is set to STGT_DISK_PNP, mounting points created using the system mount manager appear as physical devices in the Disk Management snap-in of the Microsoft Management Console (mmc.exe).

This flag is a necessary prerequisite for creating a folder mounting point, which makes a drive accessible via an otherwise empty directory on another NTFS volume.

This flag cannot be combined with STGMP_SIMPLE, STGMP_NETWORK, or STGMP_LOCAL.

Only one mounting point of this type can be added to a virtual drive.

STGMP_NETWORK 0x00040000 Create a network mounting point.

Network mounting points can be further configured using the various STGMP_NETWORK_* flags described below. Applications that plan to make use of network mounting points must be sure to install the Helper DLL before doing so, otherwise Windows Explorer will not correctly recognize the "network" drive.

This flag cannot be combined with STGMP_SIMPLE or STGMP_MOUNT_MANAGER.

STGMP_LOCAL 0x10000000 Specifies that a local mounting point should be created.

This flag specifies that a local mounting point should be created rather than a global one. When this flag is set, applications must also pass an appropriate value for the add_mounting_point method's AuthenticationId parameter.

Passing 0 for AuthenticationId will make the mounting point visible in the current user session. To make the mounting point visible in a different user session instead, pass the target session's Authentication ID.

This flag is valid when combined with STGMP_SIMPLE or STGMP_NETWORK; it cannot be combined with STGMP_MOUNT_MANAGER. Please note that a mounting point can be made available to other computers as a network share, and network shares are always globally visible on the local machine, even if this flag is set.

STGMP_NETWORK_ALLOW_MAP_AS_DRIVE 0x00000001 Indicates that users may assign a drive letter to the share (e.g., using the 'Map network drive...' context menu item in Windows Explorer).

STGMP_NETWORK_HIDDEN_SHARE 0x00000002 Indicates that the share should be skipped during enumeration.

Such shares are only accessible when their name is already known to the accessor.

STGMP_NETWORK_READ_ACCESS 0x00000004 Makes a read-only share available for the mounting point.

When this flag is specified, the <Server Name> part of the MountingPoint parameter value must be empty. Please refer to the Mounting Points topic for more information. This flag makes the class use the Windows API's NetShareAdd function. As per MSDN, "Only members of the Administrators, System Operators, or Power Users local group can add file shares with a call to the NetShareAdd function."

STGMP_NETWORK_WRITE_ACCESS 0x00000008 Makes a read/write share available for the mounting point.

When this flag is specified, the <Server Name> part of the MountingPoint parameter value must be empty. Please refer to the Mounting Points topic for more information. This flag makes the class use the Windows API's NetShareAdd function. As per MSDN, "Only members of the Administrators, System Operators, or Power Users local group can add file shares with a call to the NetShareAdd function."

STGMP_NETWORK_CLAIM_SERVER_NAME 0x00000010 Specifies that the server name is unique.

When this flag is specified, the driver handles IOCTL_REDIR_QUERY_PATH[_EX] requests by instructing the OS to direct all requests going to the <Server Name> part of the MountingPoint parameter's value to the driver instead.

This flag should be used when the <Server Name> is unique within the local system (e.g., when the application's name is used). Using this flag allows the system to avoid delays caused by certain network requests made by various processes.

This flag is also required for "net view" command to be able to show the share in the list.
STGMP_DRIVE_LETTER_NOTIFY_ASYNC 0x20000000 Causes the method to return immediately without waiting for mounting notifications to be sent to the system.

STGMP_AUTOCREATE_DRIVE_LETTER 0x40000000 Tells the component that it should assign the drive letter automatically.

When this flag is specified, the class will automatically assign a drive letter from the list of available letters. The assigned letter is added to the end of the list of mounting points, and can be retrieved from there.

Do not include a drive letter in the MountingPoint parameter's value when specifying this flag.

Mounting Point Flags in Linux and macOS

STGMP_LOCAL_FUSE 0x10000000 Creates a mounting point, accessible only for current user.

If this flag is not passed, the "-oallow_other" option of FUSE is used.

STGMP_SYMLINK_DEBUG 0x40000000 Prints debug messages to stderr

The messages generated by the class are printed.

STGMP_SYMLINK_SYSTEM_DEBUG 0x20000000 Prints debug messages generated by the FUSE library to stderr

STGMP_NETWORK_MACOS 0x00040000 Create a network mounting point (macOS only).

If this flag is not passed, the "-olocal" option of macFUSE is used.

Storage Type Values

STGT_DISK 0x00000000 Create a regular disk device.

STGT_CDROM 0x00000001 Create a CD-ROM or DVD device.

STGT_DISK_PNP 0x00000003 Create a plug-and-play storage device.

Important: The CBFS Storage system driver must be installed in PnP mode for this option to function properly.

Storage Characteristics Flags

STGC_FLOPPY_DISKETTE 0x00000001 The storage is a floppy disk device.

This flag is not supported when storage_type is set to STGT_DISK_PNP.

STGC_READONLY_DEVICE 0x00000002 The storage is a read-only device.

STGC_WRITE_ONCE_MEDIA 0x00000008 The storage device's media can only be written to once.

This flag is not supported when storage_type is set to STGT_DISK_PNP.

STGC_REMOVABLE_MEDIA 0x00000010 The storage device's media is removable.

Users may remove the storage media from the virtual drive at any time. (Note that this flag does not indicate that the virtual drive itself is removable.)

STGC_AUTOCREATE_DRIVE_LETTER 0x00002000 The system should automatically create a drive letter for the storage device.

Deprecated: Include the STGMP_AUTOCREATE_DRIVE_LETTER flag in the value passed for the add_mounting_point method's Flags parameter instead.

When this flag is present, the storage_guid property must be set. This flag only works when storage_type is set to STGT_DISK_PNP.

STGC_SHOW_IN_EJECTION_TRAY 0x00004000 The storage device should be shown in the 'Safely Remove Hardware and Eject Media' menu in the system notification area (system tray).

This flag only works when storage_type is set to STGT_DISK_PNP.

STGC_ALLOW_EJECTION 0x00008000 The storage device can be ejected.

Users may eject the virtual drive at any time. When the virtual drive is ejected, it is destroyed.

This flag only works when storage_type is set to STGT_DISK_PNP.

STGC_RESERVED_1 0x00010000 Reserved, do not use.

STGC_RESERVED_2 0x00020000 Reserved, do not use.

Share Modes

FILE_SYS_SHARE_READ 0x00000001 Enables subsequent open operations on a file to request read access.

Otherwise, other processes cannot open the file if they request read access. If this flag is not specified, but the file has been opened for read access, file creation or opening fails.

FILE_SYS_SHARE_WRITE 0x00000002 Enables subsequent open operations on a file to request write access.

Otherwise, other processes cannot open the file if they request write access. If this flag is not specified, but the file has been opened for write access or has a file mapping with write access, file creation or opening fails.

FILE_SYS_SHARE_DELETE 0x00000004 Enables subsequent open operations on a file to request delete access.

Otherwise, other processes cannot open the file if they request delete access. If this flag is not specified, but the file has been opened for delete access, the function fails.

Note: Delete access allows both delete and rename operations.

Copyright (c) 2023 Callback Technologies, Inc. - All rights reserved.
CBFS Storage 2022 Python Edition - Version 22.0 [Build 8460]