Download migrations over IPFS #86
Conversation
There was a problem hiding this comment.
We really need this. The current state is too brittle for use in Desktop / Brave.
Breaking updates are long running frustratiin to our Desktop users, and we should avoid exposing Brave users to the same problem.
| 2. Implement step 6. Unless nodes keep around a copy of the migration, this feature isn't going to be all that useful. | ||
| 3. Eventually, switch over to using IPFS by default instead of the gateway as the first option. | ||
|
|
||
| NOTE: the only reason we're proposing using the gateway first is that the migration blocks startup so it needs to be fast and reliable. |
There was a problem hiding this comment.
What if we set up peering to the relevant gateway or cluster nodes so we defaulted to IPFS and dogfood Bitswap? On the surface this seems easy, is it worthwhile?
There was a problem hiding this comment.
I'd like to start with HTTPs by default so we can get this in ASAP. Then, maybe in the next release, switch to IPFS by default so we have some time to test.
But yeah, peering by default sounds like a great idea.
There was a problem hiding this comment.
@aschmahmann could you add that to the proposal?
| _Why might this project be lower impact than expected? How could this project fail to complete, or fail to be successful?_ | ||
|
|
||
| 1. We may not be able to download/find the migration on IPFS. Trying the gateway first (or racing) should help mitigate this. | ||
| 2. The "temporary" node may be missing important parts of the configuration. For example, it may need alternative bootstrap nodes as our nodes may not be reachable. We'll need to think about this carefully and may need to "reach" into the old repo's config file a bit. |
There was a problem hiding this comment.
Could totally be a problem with a few possible ways to fix it, it might not be so bad to wait for the user reports to come in before deciding how to resolve this.
One perhaps simple way to deal with this would be allowing users to pass a flag ipfs daemon --migration-config=/path/to/config and then load that config file into the temporary node. It's a bit of a hassle for the user, but would certainly handle the job.
There was a problem hiding this comment.
I think we should read peering from existing config, instead of, or in addition to, bootstrapping config.
@aschmahmann Before allowing them to point migration at any config, I would prefer to see if it is sufficient to look at existing config, and maybe allow an ipfs peer or http gateway specified on cli.
| ##### Bundle the migrations | ||
|
|
||
| We could bundle all migrations with go-ipfs. But that would add 100s of megabytes to the go-ipfs distribution so we really don't want to do this. | ||
|
|
There was a problem hiding this comment.
Another alternative, which we'd like to do but is probably more work, is to have a system we trust enough to distribute binaries that are auto-executed by the user.
This would enable us to do auto-updating of go-ipfs and cover migrations at the same time. The downside here is that it's a) more work b) requires a bit (although not necessarily a ton) of design on how we to do this securely.
This is listed as a future opportunity, but technically we could do it instead of this.
mikeal
added
confidence:low
BigLep
added
the
Steward Priority
|
I'm not sure the right assignee for this since we don't have our project teams modeled in the Protocol organization, but this is getting owned by the "Data Systems" team. |
| _Why might this project be lower impact than expected? How could this project fail to complete, or fail to be successful?_ | ||
|
|
||
| 1. We may not be able to download/find the migration on IPFS. Trying the gateway first (or racing) should help mitigate this. | ||
| 2. The "temporary" node may be missing important parts of the configuration. For example, it may need alternative bootstrap nodes as our nodes may not be reachable. We'll need to think about this carefully and may need to "reach" into the old repo's config file a bit. |
There was a problem hiding this comment.
I think we should read peering from existing config, instead of, or in addition to, bootstrapping config.
@aschmahmann Before allowing them to point migration at any config, I would prefer to see if it is sufficient to look at existing config, and maybe allow an ipfs peer or http gateway specified on cli.
| // Whether or not to keep the migration after downloading it. | ||
| "Keep": "pin"|"cache"|"discard" | ||
| } | ||
| } |
There was a problem hiding this comment.
Should this be read from the existing config file, or something that a user puts in a separate migration config file? The existing config file may not be readable until after migration is performed.
What about having this specified by optional CLI flags?
--download-policy="ipfs,custom-gateway1.io,http"
--keep-migration="discard" | "pin" | "keep"
There was a problem hiding this comment.
GAHHHHHHH. This is so, very, annoying.
What if we just decoded this part of the config? As long as this part of the config remains somewhat consistent across upgrades (and we correctly handle missing fields), we should be fine.
Basically:
- Get the path to the repo.
- Read the config file into a
struct { Migrations config.MigrationConfig }type. - Move on.
This is significantly safer than, say, reading the swarm config, because it only deals with migrations.
Thoughts?
There was a problem hiding this comment.
That is fine. Will go with config file.
There was a problem hiding this comment.
I think it should also include:
// Peers lists the nodes to attempt to connect with when downloading
// migrations from IPFS.
Peers []peer.AddrInfo 
No description provided.