File System Virtualization

This is the second part in our series File System Virtualization – The New Perspective.

In this part, we’ll demonstrate how to implement a virtual disk plugin for one of the most popular cloud storage services, Google Drive. This plugin will provide you with transparent access to cloud files and folders.

Note: This plugin requires our previously implemented virtual disk service.

Contents

Plugin API
Virtual Disk Interface
Google Drive API
Sample API Implementation
Local File Caching
Plugin in Action
Conclusion
References

Plugin API

First of all, we’ll need a common cloud plugin interface that can be used by all future plugins that we decide to implement:

C++

class ICloudWorker
{
public:
    virtual ~ICloudWorker(){}
 
    virtual std::wstring GetPluginName() = 0;
    virtual std::shared_ptr<ICloudItem> GetRoot() = 0;
    virtual void GetSpaceInformation(...) = 0;
};

Since a cloud file can represent either an actual file or a directory, we can work with such files using the generic ICloudItem interface:

C++

class ICloudItem
{
public:
    virtual ~ICloudItem() {}
 
    virtual ItemType GetType() const = 0;
    virtual const std::string& GetId() const = 0;
    virtual __int64 GetVersion() const = 0;
    virtual const std::wstring& GetName() const = 0;
    virtual const std::string& GetCloudName() const = 0;
    virtual __int64 GetCreationTime() const = 0;
    virtual __int64 GetLastAccessTime() const = 0;
    virtual __int64 GetLastWriteTime() const = 0;
    virtual __int64 GetChangeTime() const = 0;
    virtual __int64 GetSize() const = 0;
    virtual void Rename(...) = 0;
    virtual void SetInfo(...) = 0;
    virtual void SetSize(...) = 0;
    virtual void Update(...) = 0;
};

Plugins take the form of a Windows dll file. This allows you to load and unload them on demand through a virtual disk service. Here’s a possible wrapper for such a plugin:

C++

class CProtocolLibHolder
{
public:
    explicit CProtocolLibHolder(LPCWSTR libraryFileName);
    virtual ~CProtocolLibHolder();
 
    IProtocolManager *GetProtocol(IServiceManager *pServiceManager);
 
private:
    void LoadProtocolLib();
    void FreeProtocolLib();
};

And here’s an example of LoadProtocolLib method implementation:

C++

void CProtocolWrapper::CProtocolLibHolder::LoadProtocolLib()
{
    _ASSERTE(NULL == m_hModule);
    if (NULL == m_hModule)
    {
        // load library
        m_hModule = ::LoadLibrary(m_LibraryFileName);
        if (NULL == m_hModule)
        {
            throw cmn::WinException("Failed to load module");
        }
 
        try
        {
            // get pointers to functions
            (FARPROC&)m_pfnGetProtocol = ::GetProcAddress(m_hModule, GET_PROTOCOL_PROC_NAME);
            if (NULL == m_pfnGetProtocol)
            {
                throw cmn::WinException("Failed to get proc address");
            }
 
            InitFunc *m_pfnInit = NULL;
            (FARPROC&)m_pfnInit = ::GetProcAddress(m_hModule, INIT_PROC_NAME);
            if(m_pfnInit)
                m_pfnInit(m_hModule, CMfDiskService::GetLog());
        }
        catch(const cmn::WinException&)
        {
            this->FreeProtocolLib();
            throw;
        }
    }
}

The dll plugin needs to export only three functions, and it works as a wrapper for ICloudWorker:

C++

namespace google
{
    class Protocol : public cmn::CommonProtocolManager
    {
    protected:
        virtual std::unique_ptr<cmn::ICloudWorker> CreateCloudWorker(MF_CONNECT_INFO* info);
    };
}
 
void InitProtocolLib(...)
{
    // Do module initialization
}
 
void UninitProtocolLib(...)
{
    // Do module cleanup
}
 
IProtocolManager* GetProtocol(IServiceManager*)
{
    try
    {
        return new google::Protocol();
    }
    catch (const std::exception&)
    {
        return NULL;
    }
}

Virtual Disk Interface

When you create a plugin API, you’ll need to implement a Google Drive protocol for your disk service. Using the ICloudWorker interface, you can create a single common protocol class without the need to create separate implementations for each cloud plugin you may add in the future.

Google Drive API

Google Drive provides a RESTful API for interacting with files in the cloud. Both API versions 2 and 3 are currently supported. You can learn more in Google’s API Reference. We’ll cover only part of this material for the sake of simplicity.

Bindings exist for some common programming languages such as Java, JavaScript, C#, Objective-C, PHP, and Python. Unfortunately, C++ isn’t among them so you’ll need to write some boilerplate code too. Tools such as curl or the more modern cpprestsdk can significantly reduce the amount of routine work.

You should authorize every request coming to the Drive API using OAuth2 supported by cpprestsdk. That’s beyond the scope of this article, though.

The Drive API supports the following file and folder requests:

create
update
copy
delete
list
get
emptyTrash
export
watch
generateIds

Your Virtual Disk plugin must implement at least the bare minimum of requests required to correct file I/O handling. These requests are create, update, delete, get, and list.

Related services

Web Application Development Services & Solutions

Sample API Implementation

Once you define the necessary API functions, you can start implementing them.

You can represent a Drive cloud item using either a file or folder class.

C++

class FileItem : public BaseCloudItem<FileItem>
{
public:
    FileItem(const Metadata& metadata, Operations& operations);
 
    virtual void Download(...);
    virtual void Upload(...);
};
class FolderItem : public BaseCloudItem<FolderItem>
{
public:
    FolderItem(const Metadata& metadata, Operations& operations);
 
    virtual boost::shared_ptr<cmn::IChildrenList> GetChildrenList(...);
 
    virtual cmn::CloudItemPtr CreateItem(...);
    virtual void DeleteItem(...);
    virtual std::string RenameItem(...);
    virtual std::string MoveItem(...);
};

Every item also has associated metadata serialized into the JSON object.

C++

class Metadata
{
public:
    Metadata(const Json::Value& metadata);
 
    bool IsFolder() const;
    bool IsDownloadable() const;
 
    const std::string& GetId() const;
    const std::string& GetTitleWindows() const;
    const std::string& GetTitleOriginal() const;
    unsigned __int64 GetSize() const;
 
    unsigned __int64 GetCreatedDate() const;
    unsigned __int64 GetLastViewedDate() const;
    unsigned __int64 GetModifiedDate() const;
 
    __int64 GetVersion() const;
    const std::string& GetDownloadUrl() const;
    bool IsTrashed() const;
 
private:
    ...
};

Finally, you can write the code for actual Drive API requests. You should use the get operation for this. All other operations can be implemented similarly. This example uses curl to handle the HTTP protocol:

C++

void google::Operations::get(const Metadata& metadata, HANDLE file)
{
    const std::string& url = metadata.GetDownloadUrl();
    std::wstring fileName(utils::Utf8ToUtf16(metadata.GetTitleWindows()));
    cmn::Request request = m_requestCreator->Create();
    cmn::FileWriteResultHandler handler(file);
    request.Perform(url, &handler);
 
    if (!::SetEndOfFile(file))
    {
        throw cmn::WinException("Failed to set end of file");
    }
}
 
const std::string& google::Metadata::GetDownloadUrl() const
{
    return ParseString(m_metadata, "downloadUrl");
}
 
void cmn::Request::Perform(const char* url, IRequestResultHandler* resultHandler)
{
    RequestData data(m_curl->m_curl.get(), m_eventHandler.get(), &m_header, resultHandler, &m_errorHandler);
 
    CURL_SETOPT_CHECK(m_curl->m_curl.get(), CURLOPT_WRITEDATA, &data);
    CURL_SETOPT_CHECK(m_curl->m_curl.get(), CURLOPT_HEADERDATA, &data);
    CURL_SETOPT_CHECK(m_curl->m_curl.get(), CURLOPT_HTTPHEADER, m_curl->m_headers);
    CURL_SETOPT_CHECK(m_curl->m_curl.get(), CURLOPT_URL, url);
 
    PerformLoop();
}

Local File Caching

Since the Drive API doesn’t support partial file downloads, you should synchronize all files locally before making changes. That way all file system operations will be performed in the cache once the file download is complete.

C++

class ICloudCache
{
public:
    virtual ~ICloudCache () {}
 
    virtual bool Exists() const;
    virtual void Create();
    virtual void Delete();
 
    virtual void Disconnect() = 0;
 
    virtual void CacheCreateFile(...) = 0;
    virtual void CacheCloseFile(...) = 0;
    virtual void CacheReleaseFile(...) = 0;
    virtual void CacheQueryFileInfo(...) = 0;
    virtual void CacheSetFileSize(...) = 0;
    virtual void CacheSetFileBasicInfo(...) = 0;
    virtual void CacheDeleteFile(...) = 0;
    virtual void CacheRenameFile(...) = 0;
    virtual void CacheQueryDirContents(...) = 0;
    virtual void CacheReadFile(...) = 0;
    virtual void CacheWriteFile(...) = 0;
    virtual void CacheQueryVolumeInfo(...) = 0;
    virtual void CacheSetVolumeInfo(...) = 0;
    virtual void CacheLockFile(...) = 0;
    virtual void CacheOnFileDeleted(...) = 0;
};

Here’s a possible implementation for CacheCreateFile:

C++

void cache::FileFolderCache::CacheCreateFile(
    const std::wstring& relativePath,
    DWORD attributes,
    DWORD createDisposition,
    ACCESS_MASK access,
    WORD shareAccess,
    DWORD* createInfo)
{
    HANDLE handle = NULL;
    DWORD result = m_ntdll.NtCreateFile(
        GetFullPath(relativePath),
        attributes,
        createDisposition,
        access,
        shareAccess,
        &handle,
        createInfo);
 
    if (result != ERROR_SUCCESS)
    {
        throw cmn::WinException("Failed to create file", result);

After applying changes to a file, you should synchronize it back to the cloud. You should do this asynchronously to ensure that multiple changes in the same file are consolidated and to reduce the number of network I/O operations.

C++

void cache::FileFolderCache::CacheWriteFile(
    FILE_HANDLE hFileHandle,
    LARGE_INTEGER byteOffset,
    DWORD dwLength,
    PVOID pvBuffer,
    PDWORD pdwBytesWritten)
{
    // Write data into physical file
    cmn::ItemWrapperPtr item = m_fileWorker.WriteFile(
        hFileHandle, byteOffset, dwLength, pvBuffer, pdwBytesWritten);
 
    if (item == NULL)
    {
        return;
    }
 
    //
    // Queue file for further cloud synchronization
    //
    cmn::CloudItemInfo info(item->GetId(), item->GetType(), m_paths.GetPath(hFileHandle), item->GetVersion());
 
    m_contextHolder.AddItemAsChanged(info);
}

To reduce latency and improve the user experience, the cache also utilizes the readahead technique: the plugin downloads all cloud files in advance starting from the drive root. At the time the system accesses the necessary file, it has already been located in the local storage.

Plugin in Action

When your plugin is complete, you can test it.

But first, you should enable the Drive API in the Google Developers Console. You can do this by following these simple steps:

Use this wizard to create or select a project in the Google Developers Console and automatically enable the API. Click continue and then go to credentials.
On the add credentials to your project page, click cancel.
At the top of the page, select the OAuth consent screen tab. Select the email address option, enter a product name if not already set, and click save.
Select the credentials tab, click the create credentials button, and select OAuth client ID.
Select the other application type, enter the name “Virtual Disk plugin,” and click create.
Click OK to dismiss the dialog box that opens.
Save both the client ID and secret for use in the mounting tool.

Now you can mount and access the disk:

Your cloud storage will now be visible as a local disk in Windows Explorer:

You can seamlessly access all disk files.

Related services

Cloud Computing & Virtualization Development

Conclusion

In this article, we’ve shown the steps required to implement a cloud storage plugin for a virtual disk using file system virtualization. This plugin is designed to work with Google Drive, but what we’ve created is a solid interface foundation allowing for quick integration with other cloud plugins as well: DropBox, Box, Adobe Creative Cloud, and more.