feat: cloning remote repository #420

gbaranski · 2021-05-04T10:11:15Z

closes #419

Cloning remote repository from HTTP URL
Caching repositories
Progress bars(git clone may take some time)
~~Improve performance by fetching with depth = 1~~ I can't find option in git2 library

gbaranski · 2021-05-04T12:25:54Z

onefetch-clone-repo.mp4

That's how it looks like with progress bars

spenserblack · 2021-05-04T12:43:59Z

src/onefetch/cli.rs

+        .arg(
+            Arg::with_name("remote")
+            .long("remote")
+            .help("Clone repository from remote"),
+        )


I think it would be useful if this arg either

Had another arg that required this arg

Took a second value

So that the user could specify whether the cloned repo should be deleted or kept.

e.g. onefetch --remote <my URL> --keep to keep the repo after cloning it.

I think it would be useful if this arg either

* Had another arg that [required](https://docs.rs/clap/2.33.3/clap/struct.Arg.html#method.requires) this arg * Took a second value

So that the user could specify whether the cloned repo should be deleted or kept.

e.g. onefetch --remote <my URL> --keep to keep the repo after cloning it.

Where would you like to keep this repository?

You could probably just clone it to the current directory if it's being kept.

You could probably just clone it to the current directory if it's being kept.

I think it does not make much sense, because you could just do git clone and then run onefetch there.

Purpose of this feature is to quickly look up information about repository while not caring at all about the repository itself, but just about statistics.

Purpose of this feature is to quickly look up information about repository while not caring at all about the repository itself, but just about statistics.

Yes, that makes sense, but, since it is not doing a shallow clone, you're getting the whole repository anyway. For example, if this feature was used on the Linux repo, it would take a long time. Worse if the user uses this feature and then decides to clone it to do some work on it. It is hardly "quickly looking up information."

For example, somebody could use this feature, and then decide it's a repository that they would want to work on based on the stats. If this feature doesn't support keeping the repository, they would have to clone it a second time.

Although I agree that it is unnecessary to add a --keep option if the repo is cloned with a shallow depth, since onefetch's clone would take less time, and a shallow clone isn't that useful for actual work. 🙂

spenserblack · 2021-05-04T12:59:01Z

🤔 I'm a bit worried about clone depth not being set. Perhaps at least a warning could be documented somewhere, like in the arg's help?

spenserblack · 2021-05-04T16:56:19Z

☹️ I'm having trouble using this feature right now. [onefetch error]: there is no TLS stream available; class=Ssl (16)
Seems like an issue with git2 failing to link with OpenSSL when I compile.

While I look into this, could someone run

time cargo run --release -- --remote https://github.com/torvalds/linux.git
time git clone --depth=1 https://github.com/torvalds/linux.git and time cargo run -- ./linux/

I'm curious what the time difference is between onefetch cloning a repository and the user manually performing a shallow clone.

o2sh · 2021-05-05T19:18:02Z

Well done @gbaranski 👍 , I took the liberty with bd3abbf to remove the default-feature = false from the git2 dependency in order to fix the issue reported by @spenserblack

[onefetch error]: there is no TLS stream available; class=Ssl (16)

As @spenserblack mentionned, the --remote CLI arg should have a required value and not rely on the INPUT which is optional.
What about some regex pattern matching to make sure the user has provided a valid URL? I'm not sure how good the error handling done by the RepoBuilder APIs 🤔 is.

Can we handle SSH? Or do we stick to HTTPS, in which case the the "ssh" and "ssh_key_from_memory" features in git2-rs should be disabled.

gbaranski · 2021-05-05T19:33:59Z

What about some regex pattern matching to make sure the user has provided a valid URL?

Yeah I also thought about that, we could check if it starts by http(s)://, but what about SSH? check if starts with git@?

Or maybe just use INPUT and check if folder exists, if it does not, then try to clone it from remote git repo.

Can we handle SSH? Or do we stick to HTTPS, in which case the the "ssh" and "ssh_key_from_memory" features in git2-rs should be disabled.

I think we could add --ssh-key <path> argument, so people could also alias onefetch to onefetch --ssh-key ~/.ssh/id_ed25519

o2sh · 2021-05-05T20:34:38Z

For the URL validation, you can make use of the url crate and do smth in the line of: Url::parse(&url)is_ok().

Or maybe just use INPUT and check if folder exists, if it does not, then try to clone it from remote git repo.

I would prefer to keep the default behaviour of the tool as is, which is to run locally and be 100% offline.

I think we could add --ssh-key argument, so people could also alias onefetch to onefetch --ssh-key ~/.ssh/id_ed25519

Or we can simply try with $HOME/.ssh/id_rsa like in this example. But yeah, maybe handling SSH isn't so important for a first iteration...

gbaranski · 2021-05-06T09:29:31Z

For the URL validation, you can make use of the url crate and do smth in the line of: Url::parse(&url)is_ok().

Or maybe just use INPUT and check if folder exists, if it does not, then try to clone it from remote git repo.

I would prefer to keep the default behaviour of the tool as is, which is to run locally and be 100% offline.

I think we could add --ssh-key argument, so people could also alias onefetch to onefetch --ssh-key ~/.ssh/id_ed25519

Or we can simply try with $HOME/.ssh/id_rsa like in this example. But yeah, maybe handling SSH isn't so important for a first iteration...

Yeah but some people might not want to use their SSH Key for that. And what if someone has multiple keys and wants to use specific one?

spenserblack · 2021-05-06T12:37:35Z

what if someone has multiple keys and wants to use specific one?

I've mostly used default behavior when using SSH to clone, and I know little about SSH beyond that, so I may be mistaken. But wouldn't ~/.ssh/config be used to control which key is used for the github.com (or wherever they're cloning from) domain? Would it be sufficient to expect the user to set up the SSH configuration so that they use the key they want to use?

Somewhat related:

Git users can use these to override the SSH key used (e.g. git config core.sshCommand "ssh -i my-key <other args>"),
but it's more typical (at least according to Git's documentation) to use ~/.ssh/config.

gbaranski · 2021-05-07T07:43:45Z

what if someone has multiple keys and wants to use specific one?

I've mostly used default behavior when using SSH to clone, and I know little about SSH beyond that, so I may be mistaken. But wouldn't ~/.ssh/config be used to control which key is used for the github.com (or wherever they're cloning from) domain? Would it be sufficient to expect the user to set up the SSH configuration so that they use the key they want to use?

Somewhat related:
* [`GIT_SSH`](https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables#_miscellaneous)

* [`GIT_SSH` AND `GIT_SSH_COMMAND`](https://git-scm.com/docs/git#Documentation/git.txt-codeGITSSHcode)

* `core.sshCommand`
Git users can use these to override the SSH key used (e.g. git config core.sshCommand "ssh -i my-key <other args>"),
but it's more typical (at least according to Git's documentation) to use ~/.ssh/config.

I think most people don't even have ~/.ssh/config file, because that's not necessary

spenserblack · 2021-05-07T11:59:48Z

I think most people don't even have ~/.ssh/config file

In that case, an additional CLI arg for SSH key path would probably be best, as you suggested earlier. I think the arg should require the --remote arg, and use <home dir>/.ssh/id_rsa as a default. There are a few crates to help get the home directory of each OS: dirs, dirs-next, directories, etc.
There could also be a validator to reject the --ssh-key arg if the remote is detected as HTTP. Or at least a warning that the key will not be used.

spenserblack · 2021-05-07T12:29:46Z

Still not sure about the necessity of this feature, though. It is very easy to accomplish this with a small script. See this bash script, which is run with ./onefetch-remote <repo URL>.

I appreciate the work you've done on this PR, but are we sure that this feature should be added?

o2sh · 2021-05-07T14:53:32Z

So far I've been kinda neutral about this feature overall - as it doesn't really "harm" the project to have it -, but the bash script does seem to be the best option for this 😕.

spenserblack · 2021-05-27T15:01:24Z

🤔 This PR has been in stasis for a while. Perhaps we should ask another collaborator or frequent contributor to offer their opinion?

o2sh · 2021-05-27T15:24:04Z

@HallerPatrick @yoichi @ebroto @erikgaal @CephalonRho @Luke-zhang-04

Luke-zhang-04 · 2021-05-31T19:23:50Z

so... this is kinda like https://github.com/spenserblack/repofetch?

o2sh · 2021-05-31T20:55:35Z

Not really, repofetch relies on the GH APIs to fetch information; it doesn't clone the repository locally. See #419 for more context.

Luke-zhang-04 · 2021-05-31T23:32:47Z

Hm, ok interesting. I'm just gonna say, I personally don't see myself using this feature, but it's cool nonetheless.

gbaranski added 2 commits May 4, 2021 12:07

feat: fetching from remote repository via HTTP URL

ad97e2e

feat: progress bars when cloning git remote repo

bc86842

fix: remove '/' in path, windows doesnt understand

dfd0472

gbaranski marked this pull request as ready for review May 4, 2021 12:32

spenserblack reviewed May 4, 2021

View reviewed changes

pull ssl/ssh features from git2 + code cleanup

bd3abbf

o2sh closed this Jun 2, 2021

feat: cloning remote repository #420

feat: cloning remote repository #420

Uh oh!

Conversation

gbaranski commented May 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gbaranski commented May 4, 2021

Uh oh!

spenserblack May 4, 2021

Choose a reason for hiding this comment

Uh oh!

gbaranski May 4, 2021

Choose a reason for hiding this comment

Uh oh!

spenserblack May 4, 2021

Choose a reason for hiding this comment

Uh oh!

gbaranski May 4, 2021

Choose a reason for hiding this comment

Uh oh!

spenserblack May 4, 2021

Choose a reason for hiding this comment

Uh oh!

spenserblack commented May 4, 2021

Uh oh!

spenserblack commented May 4, 2021

Uh oh!

o2sh commented May 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gbaranski commented May 5, 2021

Uh oh!

o2sh commented May 5, 2021

Uh oh!

gbaranski commented May 6, 2021

Uh oh!

spenserblack commented May 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gbaranski commented May 7, 2021

Uh oh!

spenserblack commented May 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spenserblack commented May 7, 2021

Uh oh!

o2sh commented May 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spenserblack commented May 27, 2021

Uh oh!

o2sh commented May 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Luke-zhang-04 commented May 31, 2021

Uh oh!

o2sh commented May 31, 2021

Uh oh!

Luke-zhang-04 commented May 31, 2021

Uh oh!

Uh oh!

gbaranski commented May 4, 2021 •

edited

Loading

o2sh commented May 5, 2021 •

edited

Loading

spenserblack commented May 6, 2021 •

edited

Loading

spenserblack commented May 7, 2021 •

edited

Loading

o2sh commented May 7, 2021 •

edited

Loading

o2sh commented May 27, 2021 •

edited

Loading