Filecoin Improvement Proposals

FIP-0017: Three-messages Lightweight Sector updates Source

Authornicola, lucaniz, irenegia
Discussions-Tohttps://github.com/filecoin-project/FIPs/issues/131
StatusDraft
TypeCore
Created2021-07-27

Spec Sections

Simple Summary

A protocol for updating any sector with new data without re-sealing.

Abstract

The intuition is that the miner “xor”-s (techically field addition) the new data with some unpredictable randomness into the last layer of the Proof of Replication.

The miner generates three messages:

  • Declares an update with DeclareUpdate with the new deals
  • Gets the encoding randomness from the chain and generates a PreCommitReplicaUpdate
  • Gets the challenges seed from the chain and generates ProveCommitReplicaUpdate

Change Motivation

Since 90+% of sectors in the Filecoin Network are CC sectors, having a protocol that allows for updating CC sectors to store real data without incurring in a full re-sealing would massively improve our network in terms of the amount of real data stored through it.

  1. It would allow decoupling sealing latency from deal-making speed - offering storage clients an improved experience for how quickly their data can land in on-chain deals
  2. It would unlock the 8EiB of storage already committed to the Filecoin network to be quickly used for deals - enabling a 100PiB+ client to make deals for their entire dataset with a single miner like f0127595 which already has 120PiB of committed capacity.
  3. It makes utilizing existing committed capacity much cheaper for miners (since they’ve already invested the sealing cost), increasing the chances they pay clients to add FIL+ data to these sectors.

What this FIP does not support (see Future Work section)

  • Deal updates
  • Moving deals across sectors

Specification

Three-messages Update Protocol

  1. Miner publishes Miner.DeclareUpdate(deals []DealID, sectorId SectorID):
    1. Input
      1. deals is the full list of deals
    2. Validity
      1. Fail if caller is not a worker account
      2. Fail if miner is in debt
      3. Fails if miner has deals in the sector ID which are not in deals and that have not expired yet.
    3. Collaterals
      1. Requires deposit of SectorUpdateDeposit
      2. Requires deposit of new deal collateral and new initial pledge based on the new deals
    4. Side effects
      1. Stores deals for this sector in the actor (TODO: we must extend the miner actor to support this)
      2. Commits the miner to complete the update protocol, failure to do so by a certain time will make the miner lose the deposit. That time is: currentEpoch + SectorUpdateExpiry.
  2. Miner runs ReplicaUpdate routine:
    1. Gets SectorUpdateRandomness from the declared update epoch plus PreCommitReplicaUpdateRandDelay epochs
    2. Evaluates the replica newReplica[i] = Enc(sectorKey[i], data[i], rand), where
      1. newReplica is the vector of Fr elements of the new replica
      2. sectorKey is the vector of Fr elements of the last layer of the sealing process
      3. data is the vector of Fr elements of the deals data ordered and padded as done in the sealing subroutine
      4. rand is the SectorUpdateRandomness
    3. Generate a new commitment UpdatingSectorCID
  3. Miner publishes PreCommitReplicaUpdate(rui ReplicaUpdateInfo)
    1. Input
      1. rui.SealedSectorCID
      2. (TODO: check what other inputs are needed)
    2. Side effects
      1. Store the new commitment on-chain (TODO: define where this should happen)
  4. Miner runs ReplicaUpdateProve routine
    1. Let seed = SectorUpdateProveSeed, taken from precommit epoch plus PreCommitReplicaUpdateChallengeDelay.
    2. Let rand = SectorUpdateRandomness, taken from declare update epoch plus ProveCommitReplicaUpdateRandDelay
    3. Generate a SNARK that proves
      1. For challenge i=0..SectorUpdateProveChallenges, c = ChallengeGen(seed, i)
        1. Encoding: we have that newReplica[c] = Enc(sectorKey[c], data[c], rand)
        2. Inclusion proofs:
          1. newReplica[c] is the opening of UpdatingSectorCID at position c
          2. sectorKey[c] is the opening of CommRLast from SealedSectorCID at position c
          3. data[c] is the opening of UnsealedSectorCID at position c
      2. The following was correctly computednewReplica[c] = Enc(sectorKey[c], data[c], seed)
  5. Miner publishes Miner.ProveCommitReplicaUpdate(proof):
    1. Verify the proof of correct data update:
      1. Generate challenges from SectorUpdateRandomness
      2. Get SectorUpdateProveSeed (seed)
      3. Get UpdatingSectorCID commitment for this sector
      4. Get deals for this sector and generate UnsealedSectorCID from the CIDs of the deals
      5. Verify that proof is valid for using inputs: SectorUpdateRandomness, SectorUpdateSeed, UpdatingSectorCID, Challenges
    2. Collateral:
      1. Return SectorUpdateDeposit
    3. Side effects:
      1. Update SealedSectorCID with UpdatingSectorCID
      2. TODO: Remove book keeping for the in-progress update
      3. TODO: Keep the last DeclareUpdate epoch to simplify future retrieval

Retrieving updated data

  1. From the replica:
    1. Get SectorUpdateRandomness for the last DeclareUpdate
    2. Regenerate sectorKey by re-sealing the sector
    3. For each i: Dec(sectorKey[i], replica[i], rand)

Algorithms

Encoding

The current encoding algorithm is the following:

Enc(sectorKey[i], data[i], rand) = sectorKey[i] + data[i] * rand

Note that it requires re-encoding the entire sector with the new randomness.

See future work on how to improve this.

Design Rationale

Parameters choice

  • PreCommitReplicaUpdateRandDelay: 150 epochs
    • PreCommitReplicaUpdateRandDelay > Finality in order to avoid miner to adaptively choose data Ddeals (can be 150-300 epochs)
  • ProveCommitReplicaUpdateChallengeDelay: 150 epochs
    • ProveCommitReplicaUpdateChallengeDelay> Finality in order to avoid miner to be able to do long forks to grind better challenges
  • SectorUpdateProveChallenges: (TODO choose number of challenges)

TODO

Pledge

  • SectorUpdateDeposit: DeclareUpdate requires a pledge which covers both PreCommitReplicaUpdate (to ensure that they cannot grind on data) and ProveCommitReplicaUpdate (to ensure that they cannot grind on on-chain randomness) and it is given back at ProveCommitReplicaUpdate
  • InitialPledge: We must update the IP of the sector based on its new QA Power. (TODO: explain why and when)

Backwards Compatibility

All sectors can be upgraded.

Breaking immutability of sectors and deals

Test Cases

TODO

Security Considerations

Loosing epsilon-Replication guarantees

This changes changes the security property of Filecoin. Retrievability of the files and space-hardness are still guaranteed. TODO: expand on this.

Incentive Considerations

TODO

Product Considerations

TODO

Implementation

  • TODO: Initial Pledge should be updated in one of the new methods, we believe it should be PreCommit.
  • TODO: How do we best represent deals in Declare update? Do we relist all the deals or we show a diff of deals?
  • TODO: If we were to do 10x the snark (which shouldn’t be a big burden) we can remove the PreCommit step and fold it into ProveCommit - is this something that we should try doing?

Future work

  • DeclareDeals to support deal transfer: allow moving deals from sector to sector
  • CapacityDeals: allow purchasing a capacity of the storage of a miner instead of the storage of a specific deal
  • Update protocol that does not require to perform an operation on a full sector.
  • DealUpdates: a license to terminate a deal in place for a new one (e.g. a user wants to update a portion of their files)

Copyright and related rights waived via CC0.

Citation

Please cite this document as:

nicola, lucaniz, irenegia, "FIP-0017: Three-messages Lightweight Sector updates [DRAFT]," Filecoin Improvement Proposals, no. 0017, July 2021. [Online serial]. Available: https://fips.fission.app/fips/fip-0017.