Skip to content

Instantly share code, notes, and snippets.

@miminus
Last active July 22, 2016 08:56
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save miminus/1ac3d911d18052f3828a38a923cffb41 to your computer and use it in GitHub Desktop.
Save miminus/1ac3d911d18052f3828a38a923cffb41 to your computer and use it in GitHub Desktop.
一种分布式文件系统

####Defination: SeaweedFS is a simple and highly scalable distributed file system, two objectives as follows:

  • to store billions of files!
  • to serve the files fast!

####Fetures:

  • implement only a key->file mapping ,like "NoSQL" ,called "NoFS"
  • there are master-server and volume-server:

######master-server:

  • the central master only manages file volumes

######volume-server:

  • volume servers manage files and their metadata

######master server - volume server

  • The actual data is stored in volumes on storage nodes
  • One volume server can have multiple volumes
  • All volumes are managed by a master server
  • master server contains volume id to volume server mapping
  • 每个 VolumeServer 都维护个 n 个 Volume , 每个 Volume 都有一个专属 VolumeId, Needle 属于 Volume 里面的一个单元

######purposes for these designs:

  • relieves concurrency pressure from the central master
  • spreads file metadata into volume servers, allowing faster file access (just one disk read operation).

######seaweedfs - structers

  • 拓扑结构:【DataCenter, Rack, DataNode】
  • 存储模块:【Store, Volume, Needle】
  • 在 MasterServer 维护的拓扑结构里, 是把 VolumeServer 的相关信息存储在 DataNode 里

######fid fid: 3,01637037d6

a volume id: 3
one file key: 01
a file cookie: 637037d6.

  • volume id : an unsigned 32-bit integer
  • file key : an unsigned 64-bit integer
  • file cookie : an unsigned 32-bit integer - to prevent URL guessing

######replication parameter options:

  • SeaweedFS applies the replication strategy at a volume level
  • 000: no replication
  • 001: replicate once on the same rack
  • 010: replicate once on a different rack, but same data center
  • 100: replicate once on a different data center
  • 200: replicate twice on two different data center
  • 110: replicate once on a different rack, and once on a different data center

#######Storage Size

  • each volume can be 8x2^32 bytes (32GiB)
  • There can be 2^32 volumes
  • total system size is 8 x 2^32 bytes x 2^32 = 8 x 4GiB x 4Gi = 128EiB
  • Each individual file size is limited to the volume size

######TTL 定时删除

####参考链接

link1
link2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment