• vithigar@lemmy.ca
      link
      fedilink
      English
      arrow-up
      28
      arrow-down
      2
      ·
      edit-2
      5 months ago

      That wouldn’t distribute the load of storing it though. Anyone on the torrent would need to set aside 100PBs of storage for it, which is clearly never going to happen.

      You’d want a federated (or otherwise distributed) storage scheme where thousands of people could each contribute a smaller portion of storage, while also being accessible to any federated client. 100,000 clients each contributing 1TB of storage would be enough to get you one copy of the full data set with no redundancy. Ideally you’d have more than that so that a single node going down doesn’t mean permanent data loss.

      • hellofriend@lemmy.world
        link
        fedilink
        English
        arrow-up
        7
        ·
        5 months ago

        Not sure you’d be able to find 100k people to host a 1TB server though. Plus, redundancy would be better anyway since it would provide more download avenues in case some node is slow or has gone down.

        • vithigar@lemmy.ca
          link
          fedilink
          English
          arrow-up
          7
          ·
          5 months ago

          Yes, it’s a big ask, because it’s a lot of data. Any distributed solution will require either a large number of people or a huge commitment of storage capacity. Both 100,000 people and 1TB per node is a lot to ask for, but that’s basically the minimum viable level for that much data. Ten million people each committing 50GB would be great, and offer sufficient redundancy that you could lose 80% of the nodes before losing data, but that’s not a realistic number to expect to participate.

      • uis@lemm.ee
        link
        fedilink
        English
        arrow-up
        6
        ·
        edit-2
        5 months ago

        That wouldn’t distribute the load of storing it though. Anyone on the torrent would need to set aside 100PBs of storage for it, which is clearly never going to happen.

        Torrents are designed for incomplete storage of data. You can store and verify few chunks without any problem.

        You’d want a federated (or otherwise distributed) storage scheme where thousands of people could each contribute a smaller portion of storage, while also being accessible to any federated client.

        Torrents. You may not have entirety of data, but you can request what you need from swarm. The only limitation is you need to know in which chunk data you need.

        Ideally you’d have more than that so that a single node going down doesn’t mean permanent data loss.

        True.

        • vithigar@lemmy.ca
          link
          fedilink
          English
          arrow-up
          4
          ·
          5 months ago

          True. Until you responded I actually completely forgot that you can selectively download torrents. Would be nice to not have to manually manage that at the user level though.

          Some kind of bespoke torrent client that managed it under the hood could probably work without having to invent your own peer-to-peer protocol for it. I wonder how long it would take to compute the torrent hash values for 100PB of data? :D

          • uis@lemm.ee
            link
            fedilink
            English
            arrow-up
            1
            ·
            edit-2
            5 months ago

            ~300MB/s on one core of 13-years old i5 SHA-256(used in BitTorrent v2). Newer cores can about half a gig per one. Less than 3 days on one core then. Less than day on 3 cores.*

            * assuming no additional performance penalty for increased power consumption and memory bandwith usage

            My guess storage bandwidth would be biggest bottleneck.

            Found relatively old article(in Russian, just search for openssl and look at graph that mentions SHA-512 which is SHA-2 too) that says i7-2500 all-cores throughput is slightly over 1GB/s.

    • gravitas_deficiency@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      7
      arrow-down
      2
      ·
      5 months ago

      It’d be a lot more complicated than that, I think, if one wanted to effectively be able to address it like a file system, as well as holistically verify the integrity of the data and preventing unintentional and unwanted tampering