Skip to content

Instantly share code, notes, and snippets.

@balupton

balupton/scale.md

Last active January 16, 2021 00:03
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save balupton/3df7ed8bc2824e600bfc33da1f20df09 to your computer and use it in GitHub Desktop.
Save balupton/3df7ed8bc2824e600bfc33da1f20df09 to your computer and use it in GitHub Desktop.
Download ZIP
Massively Scalable Web Infrastructure
Raw
scale.md

Massively Scalable Web Infrastructure

Todo

These items still need to be categorised below:

  • FoundationDB seems an open-source version of Fauna
  • Git Funding Projects
  • General Funding Projects
  • Resilio Sync
  • S3 Torrent / WebSeed
  • BAT vs SMT
  • ENS vs HNS
  • Cloudflare IPFS Gateway
  • Cloudflare ENS Resolver

Cryptos

  • BTC a ponzi scheme investment (attractive over other ponzi schemes because of its programmed inflation mechanism) however requires off-chain solutions to scale its ineffecient proof of work algorithm, which may undermine its security
  • Ethereum an ecology of ecosystems within it, and the largest ecology, as such largest value outside BTC for the sake of BTC investment
    • Until ETH2 solves the high gas fees via sharding; off-chain solutions, dubbed layer 2 solutions/networks, are required, some of which are still compliant with the crypto ethos:
    • Imitators:
  • Cryptocurrency anonymity
  • Consensus Mechanisms

Centralised

Comparisons:

Global Data Blockchains

Blockchain for publishing and payments, generally complemented by p2p tech for distribution.

Issues with global data blockhains:

  • To avoid abuse, someone has to pay to publish, be it you, or the millionaires who control it who will gift you the ability to post if you follow their incentives, such as de-annonymising yourself.
  • Whoever has the most initial money wins, as their influnce compounds the rewards they obtain. Constant schemes must be centrally devised and enforced to counter-influence this natural pattern.
  • Decentralised generally just means a public ledger, yet control, influence, and architecture is usually still monopolised, with it changing to the whims of the largest benefactors
    • For instance, constant tweaks are made to incentives and disincentives, such as wealth redistribution based on behaviour, as well as handouts from reserves to nepotism
  • Your data becomes part of a global public ledger, which privacy is handled via end-to-end encryption, which may or may not be implemented
  • The largest benefactors are the same people who benefit from existing infrastructure: those mining big data
  • In the end, you don't control your data, nor do you have a sense of controlling your data, it is still delegated
  • They are the same pathologies as the existing status quo, just with new tech.

Trust Networks

Peer to Peer

Global DHT

DHT protocols:

Implementations:

Issues with global DHT

Anonymised Global DHT

  • Elixxir cMix: still in alpha stages
  • Tribler: anonymous overlay network for bittorrent
  • I2P: anonymous overlay network for all sorts, including bittorrent

Gossip

local sync via gossip, remote sync via servers

Transports

HTTP sync, peer discovery left up to implementor

Resources for innovators

Tooling

Chat

Q&A

End-to-end encrypted youtube videos?

Could an encryption ability, that obscures the video and audio data, but not the file itself, be performed? Such that youtube's downscale encoding also unknowingly downscales the original data too for those downscaled sizes. With the goal that one would be confident enough to use youtube for confidential one-on-one chats?

For instance, a pixel location swap (e.g. pixel at X:Y swaps to the pixel at A:B), which could be derived from say a salt, or done randomly for each pixel with the mapping becoming the decryption key. This approach probably would not tolerate compression that well, so perhaps may need to upscale the video first, put the pixels into big blocks of the same colour, then upload to youtube. A color swap may also work.

For instance, a pixel location swap (e.g. pixel at X:Y swaps to the pixel at A:B), which could be derived from say a salt, or done randomly for each pixel with the mapping becoming the decryption key.

More advanced algorithms could be tailored to the actual encoders that youtube uses.

DHTs on the Edge?

I imagine the cloudflare workers stack (either KV or Durable Objects) could be used to facilitate a peer discovery service, and multiple workers from different accounts could sync data to facilitate a multi-owner DHT, which could use some form of cryptography to verify what the worker says who is who is actually who is who. Such operating on the edge with cloudflare power would solve a lot of the current performance issues with DHT technology?

DHT Discovery Keys

How do they work, which ones support it?

DHT Conflicts

What happens when multiple cients claim a key to multiple values? How does attestation to the correction value for that key occur? Is there any transparency about this?

DHT Spam

How is preventation of spamming the DHT occur? Wouldn't it get flooded by spam nonsense, promotional materials, and malicious material such as worms and viruses?

DHT Experation

How do entries in a DHT expire? Is it kept around forever, but less common data pushed more to the sidelines of the DHT network? Is redudant data pruned (stuff that is plenty available)? Is long-term inaccessible data trimmed? What about conflicting data, how is that trimmed/pruned?

Tips

Centralised Decentralisation

While one would hope this is an oxymoron, it is used by many blockchain companies who have a public append-only ledger (a blockchain) but which they control the operations, be it via the smart contracts, the apps, the economy (adapting wealth redistribution, aka incentive/disincentive schemes, to obtain their centralised goals.

Examples of this are:

  • A claim of being decentralised, yet just uses p2p distribution to reduce costs while using a purely centralised controlled database.
  • A claim of being censorship resistent, yet their centralised influence allows blocklist propagation of say DMCAs and other things they consider undesirable.
  • A claim of freedom, yet actively design the protocol to punish or even dox, what a centralised influence considers negative behaviour.
  • A claim of being by and for the people, yet has a centralised wealth distribution scheme towards centralised objectives.

Perhaps Monopolised Decentralisation is another spin on this faux-decentralisation.

For instance, compare LBRY's marketing claims:

What if anyone in the world could publish a piece of digital content, anyone else in the world could access it, for free or for payment, and that entire system worked end-to-end without any centralized authority or point of control?

Treat users like adults. LBRY doesn't play nanny. It encourages individual people to express their own preferences, rather than force our own onto them. We enable consumers to make their own choices about where and who they want to purchase digital content from.

Operate openly, inclusively, and transparently. Anyone can publish or interact with the LBRY network. No one needs permission from us or anyone else. LBRY encourages all parties to participate in the network, rather than the creation of walled gardens. LBRY is a completely open specification and all code is open source.

With their practices:

Content Discovery Although the namespace provided by the LBRY protocol is helpful towards discovery, much as the web would be much less useful without search engines or aggregators, LBRY needs its own discovery mechanisms. Search features can be constructed from the catalogue of metadata provided in the blockchain as well as the content transaction history available in the blockchain or observed on the network. All of this data, along with user history, allows for the creation of content recommendation engines and advanced search features. Discovery on LBRY can also take the form of featured content. Clients can utilize featured content to provide additional visibility for new content that consumers might not otherwise be looking for.

LBRY Credits: Additionally, some credits are awarded on a fixed basis. The total breakdown looks like this:

  • 10% for organizations, charities, and other strategic partners. Organizations like the EFF, ACLU, and others that have fought for digital rights and the security and freedom of the internet.
  • 20% for adoption programs. We'll be giving out lots of bonus credits, especially in the early days of LBRY, in order to encourage participation. We will also look to award credits broadly, ensuring the marketplace is egalitarian.
  • 10% for us. For operational costs as well as profit.
  • 60% earned by LBRY users, via mining the LBRY cryptocurrency.

LBRY is an improvement over BitTorrent in combatting unsavory content in at least four ways:

More records. LBRY contains a public ledger of transactions recording name purchases and published content. As many purchases make it onto the ledger as well, this means infringing actions are frequently recorded forever, or are at a minimum, widely observable.

Updatable URLs. Once a BitTorrent magnet hash is in the wild, there is no mechanism to update or alter its resolution whatsoever. If a LBRY name is pointing to infringing content, it can be updated or removed (but not by us).

Stiffer penalties. Penalties for profiting off of infringement are far stronger and can involve jail time, while infringement without profit only results in statutory damages. This serves as a far stronger deterrent for all infringing uses than BitTorrent provides.

Expensive or impossible. Off-chain settlement will be a requirement for efficient purchases at any significant network size. Settlement providers, ourselves included, will be able to block purchases for infringing content. At significant traffic volume, if infringing content can't be outright removed or blocked, transaction fees will make it prohibitively expensive. And of course, let's not forget that LBRY users are still subject to the DMCA and other laws governing intellectual property. Users who publish infringing content are still subject to penalties for doing so in exactly the same way they would be via BitTorrent. LBRY only adds to the suite of options available. This makes LBRY a strict improvement over BitTorrent with regards to illegal usages, which provides none of the mechanisms listed.

Prove decentralization doesn't mean infringement. Existing decentralized publishing protocols offer no way for right holders to combat or capture profits from illegally shared content. LBRY's service layer, blacklisting mechanisms, and naming system all improve the status quo.

Acknowledge modern digital realities and ethical norms. Prohibition has failed at every turn and in every iteration. Regulating human behavior only works when it aligns with moral norms that are shared by the majority of the population

And with their roadmap

Creator Analytics and Reporting Creators must receive prompt, regular, and useful reports about how their content is performing and behaving on the LBRY network. Creators have an in-app dashboard allowing them to view content performance, including views, follows, tips/supports, and claim staking status. Creators receive regular content reports via email. Creators receive notifications related to content performance (for example, when trending, or entering/exiting top for a tag) as well as when URL changes happen or could happen.

Stronger Data Network While the LBRY data network is relatively strong (especially for popular content), we must ensure it remains strong as lbry.tv increases in usage. Additionally, users who are contributing to the network are not sufficiently rewarded. Ensure all data is well-hosted without our participation. Utilize disk space provided by users pro-actively. Offer rewards and benefits for this. Test, rework, and re-enable code for market-based data payments. Host-only setup process is user-friendly to do at scale. Form partnerships with data providers in other countries.

Creator-Owned Relationships Users have personal profiles and preferences about sharing info with creators. Users that follow creators share certain information with them subject to preferences. Creators are able to access and use this information. The system provides proper incentives (whether emotional or financial) to encourage usage.

Improved Content Discovery Providing decentralized content discovery mechanisms is absolutely central to the success of the LBRY protocol. While LBRY is possibly already the best system in the entire world for this, our design has plenty of room for improvement, particularly around inaccurate or malicious contributions. This issue is substantial and everything necessary is not likely to be enumerated below. Develop finer metrics to ascertain whether various algorithms are doing a better or worse job at returning relevant results. The problem of inaccurate tagging is eliminated or substantially reduced. Requesting trending content, whether by tag, channel, network wide, or other factors returns content that is maximally desired and germane to the request. Incorporate more user-specific information into claim search results. For example, leveraging what channels users follow or what content they like. Investigate weighted web-of-trust algorithms. Clearly document and communicate how discovery algorithms work. Run a campaign to encourage broader participation in the problem solving process. Investigate and implement other changes to align user behavior. Seek to maximize incentive that when content is prominent and easily discovered it is also desired. Continue to experiment with home page, search, following, and other discovery paradigms based on our own experiences and through listening and collecting to user feedback, both explicit and implicit.

This is faux-decentralisation, it and solutions like it, are just Google/Facebook wanabees using new tech to save money and for a marketing spin. Same shit (monopolies influencing your life), new flavour (monopolies saving money through new tech).

Compare it with local-first approaches like Scuttlebutt which by design impede such centralised aggregation, mining, and enforcement.

We need a future that:

And instead:

  • by design restores real human mutual connections
  • by design restores solidarity with the self as the captain
  • by design restores your ability to quit the perverse
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment