HEP 8 - Hazelcast Distributed File System Layer

General Proposal Information

HEP: 8
Title: Hazelcast Distributed File System Layer
HEP Shortname: HZDFS
Author: Sven Ruppert (sven DOT ruppert AT gmail DOT com)
Sponsor: 
Signed-Of: Christoph Engelbert
Lead: Sven Ruppert
Created: 2016/02/29
Status: Draft
Type: Feature
Component: Integration
Discussion: https://gitter.im/hazelcast-incubator/hzdfs
Specification: 
Project: https://github.com/hazelcast-incubator/hzdfs

Process Information

Start: 2016/Q1 
Depends:
Effort: L
Duration: L
Release: -

Summary

HZDFS the Hazelcast Distributed Filesystem Layer is a filesystem with simple access rights management and the possibility to store files larger than 2GB (chunked) in Hazelcast.

Goals

  • Chunking of big files

  • Access rights management

  • Directory Service
  • User management
  • Streaming support (micro batching?)
  • Navigating directory trees and files
  • Java 7 Path API support?
  • Support for files larger than 2GB
  • Java Input-/OutputStream API?
  • Java 7 (Async-)ByteChannel API?

Non-Goals

  • The implementation is not supposed to be fully POSIX compatible

  • No operating system adapters are part of this HEP

Motivation

It will allow to use memory-consumption based eviction policy in case when in-memory-format is OBJECT

Success Metrics

Typical CRUD operations on files and directories are available, as well as necessary mechanisms are in place to store files bigger than 2GB (chunking) and path walking / searching is possible. Files also need to support reading in a streaming fashion.

Description

HZDFS the Hazelcast Distributed Filesystem Layer is supposed to build a filesystem with access rights management. Files will be stored in chunks and distributed evenly over all nodes with a directory service and lookup system being part of the clustering system.

Testing

Typical amount of unit tests as well as integration testing and long running tests utilizing internal eviction strategies and underlying external resources.


Risks

No risk, because nothing else is based on it until now and it will layered on top of the existing Hazelcast services.


Dependencies