hicetnunc IPFS pinning

pinning.md

Hicetnunc.xyz IPFS Pinning

💡 These steps will become easier, less technical, and more accessible as more open tools begin to emerge around Hicetnunc pinning. The steps below assume macOS but should work similarly across other platforms. This gist can be seen as a working draft toward more polished documentation; if you see any issues please post a comment below.

Basic Idea

Hicetnunc.xyz aims to be "decentralized" which means the OBJKTs are owned by the users, not the platform. So, in theory, if hicetnunc disappears, another marketplace could emerge on the same (user-owned) assets. But, this paradigm of decentralization means that you own the assets; so the responsibility to maintain them lies on the users, not the platform.

Of course, hicetnunc and some of its users will probably also make an effort to help maintain all the assets on its platform; but you should not rely purely on that, as it goes against the core ethos of decentralization (it means you are placing your trust entirely upon a central authority).

The way you maintain your OBJKTs (that you mint, collect, or are simply interested in seeing last) is by 'pinning' them. The assets (images, meta data, etc) are stored in a peer-to-peer manner, shared across many devices (i.e. peers or nodes). If an asset is not being pinned, i.e. nobody is interested in it, there is a chance it could get garbage collected and potentially lost forever. So, we first need to run a "IPFS Node" to ensure we are connected to the p2p network, and then we need to "pin" the assets we want to maintain to ensure that they won't be garbage collected. Once pinned, as long as we continue to run our IPFS Node, those resources should not be garbage collected.

Requirements

• NodeJS
• A Terminal, such as macOS Terminal
• The IPFS Desktop App (or CLI if you prefer that)
• You will need some free disk space depending on how many assets you plan to pin, and how large these assets are

1. Install Requirements

Install the IPFS App from the downloads page and then run/open it. It should say "Connected to IPFS" once it's running. This is known as running an IPFS Node (also called a daemon).

Install Node.js (note: the name has no relation to IPFS Nodes) and ensure it's accessible as a command from your Terminal. Once installed, quit and re-open terminal and run node --version to ensure it's been installed correctly.

2. Download hicetnunc_tools

Go to the GitHub repository below (thanks to @antic for doing all the hard work).

https://github.com/atomantic/hicetnunc_tools

Click the green button "Code" with the arrow, then select Download ZIP. Unzip to a folder of your choice.

3. cd into hicetnunc_tools

Open your Terminal and cd into the newly unzipped folder (you can drag and drop the folder into Terminal on macOS). This sets the working directory of the shell to that folder.

4. Pin files

Now you can run the "pin" command:

node pin.js your-tz-address

Change the address to your own tz address, here's what it looks like if I pin my resources:

node pin.js tz1XHADaUcMSkTN9gdmtRqcnrrZfs4tNkCPg

This should pin your own mints and your collected work. You can re-run this command when you mint/collect new work to ensure it too is pinned. You can also pin other addresses to ensure you are pinning your favourite NFTs.

IPFS Node

If you quit/close the IPFS Desktop App (and your "IPFS Node"), and it's no longer running (i.e. not even in your task bar), then you are essentially not maintaining your OBJKTs and their pinned state across the network.¹ You can re-open the IPFS App which will restart the IPFS Node, and that will continue pinning the same resources as before. (You don't need to re-pin them whenever you restart the IPFS Node.)

Presumably, hicetnunc as a platform, and some of its users, may still be pinning your work, but if you care about your NFTs, it's best not to rely only on that assumption. If nobody is pinning your work, it could eventually be garbage collected. It's not always clear how long before collection occurs; perhaps hours or days depending on node configuration.

Long-Term Pinning

For more persistent pinning (i.e. doesn't quit with your computer), something like an always-on Raspberry Pi might be a good solution, or a cloud based provider. More details will come as I continue to explore this space. The more devices you have pinning your work, the more fail-safes you have that are maintaining your work in case one goes offline.

Future...

The more people are pinning everybody's work (rather than just their own work), the more everybody will collectively benefit. Some users are already pinning the entire hicetnunc platform, and there is some discussion in the Discord around economizing this (ie. providing tez rewards for pinning a lot of hicetnunc assets).

There are also likely many other (perhaps better) ways of pinning than described here, and perhaps some information here that may be incorrect. This is an area of hicetnunc (and NFTs in general) that needs more research, better tools, and better documentation.

---

¹ – When the IPFS Node is not running, the pinned files/resources should still exist on your computer's hard disk, but the IPFS network won't be able to access them, and nobody else in the network will see them, so it's like being offline. If that happens, and nobody else on the network is pinning/hosting the resources, there's a chance the NFT will point to a broken URL. Turning your IPFS Node back on should then bring your NFT back, so long as you still have the files on disk. (Thanks @marchingsquare for these details.)

Thanks for the writeup. I can imagine an "art screen" ie Meural, Samsung Frame, Electric Object (sadly now dead) adding support for HEN, with pinning. Maybe the possibility of adding it to Google TV / Firesticks.

Thank you so much for this, so useful, really great stuff and the code works brilliantly. On my end though I am running into two concerning issues that are unrelated to the work you have done here.

When I run this on the example address it works fine, but when I use my address it does not work. I believe this is caused by the fact that I have over 1,000 objkts and the issues that seems to present the API, but I'm not sure:

The other problem I'm noticing is that at least on my end the vast majority of objkts end up with an error when I pin for the example address:

Does this mean that these links have already broken? Or does it maybe mean I already have a backup of these on IPFS (I already did this probably about a month ago I think)?

Edit: just spot-checked some examples of the ones it errored on and I think I do see those links still up and running, so I'm not sure what went wrong the first time. Without changing anything now it looks good:

I would post an issue in the repo here : https://github.com/atomantic/hicetnunc_tools

@atomantic may have a better idea as he wrote those scripts

yeah, @ries9112 -- merkledag: not found means that those items were not found in the IPFS network at the time you tried to pin them... it's possible they are lost, but it's also possible that existing nodes caching those assets were just offline/unreachable at the time you tried to pin the items. Like you found with the edited update, sometimes just retrying works out.

The 500 error you got above is from the HEN lambdas--and usually that stems from one of the underlying APIs they use throwing an unhandled error. I'll have to take a look later and add some retry logic to the call out to get the metadata--though now that there are so many other APIs, maybe there's a more reliable API to use :)

Thanks so much to you both for the responses and the awesome work on this tool!! @atomantic perhaps one thing worth looking into is the GraphQL endpoint a user created. It's crazy fast and really reliable based on my experience with it over the past 1-2 weeks: https://hicdex.com/. Also worth mentioning is the fact that up until a couple of hours ago my profile on H=N was not loading because of an issue with the API and me owning more than 1,000 objkts. Now that issue seems fixed, but the script throws the same 500 error.

Hi, thanks for the tutorial, I'm pinning my works and my collection.
I have one question though, I used nodejs to pin the files from my address and in the status tab of the desktop app it says I'm hosting 1.7 GB of data, but when I open the Files tab I can see nothing in there. Is there a reason why files pinned with node are not showing up in there?
Thank a lot.

I have the same issue as @ries9112

Has anyone found a way to use a different endpoint to get the list of tokens to pin? I just can't seem to pin anything anymore using these tools, it crashes every time.

the issue seems to be with the AWS link in "get.tz.js":

/**
 * fetch collected tz info from hicetnunc API
 * NOTE: someday this might stop working
 */
const fetchJSON = require("./fetch.json");
module.exports = async (tz) => {
  const response = await fetchJSON(
    `https://51rknuvw76.execute-api.us-east-1.amazonaws.com/dev/tz`,
    {
      body: {
        tz,
      },
      method: "POST",
    }
  );
  return response.result;
};

500 {
  hostname: '51rknuvw76.execute-api.us-east-1.amazonaws.com',
  port: 443,
  path: '/dev/tz',
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'Content-Length': 45 },
  body: { tz: 'tz1ZBMhTa7gxSpaeXoqyc6bTCrxEHfZYSpPt' }
}
(node:8560) UnhandledPromiseRejectionWarning: Error: Failed to load page, status code: 500
    at ClientRequest.<anonymous> (C:\Users\andre\Documents\hicetnunc_tools-main\lib\fetch.json.js:33:11)
    at Object.onceWrapper (events.js:483:26)
    at ClientRequest.emit (events.js:376:20)
    at HTTPParser.parserOnIncomingClient (_http_client.js:647:27)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
    at TLSSocket.socketOnData (_http_client.js:515:22)
    at TLSSocket.emit (events.js:376:20)
    at addChunk (internal/streams/readable.js:309:12)
    at readableAddChunk (internal/streams/readable.js:284:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:223:10)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:8560) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:8560) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.