Using github and hugging face to manage projects
Published:
I often switch between multiple machines/rented servers for DL projects, so here is a quick personal note: how I use GitHub CLI (gh) to manage code and Hugging Face to manage models and datasets. Below is my usual command list and quick setup notes.
1. Preparation and networking
What gh is for: gh can create/manage GitHub repos, issues, and PRs directly from your local machine; plain Git itself does not “create a repo on GitHub”.
- Use GitHub for code: create/clone repos with gh, manage branches with Git.
Use Hugging Face to host models/data: use the hf/CLI to create repos, upload, and distribute.
- Network proxy (optional): If uploads/downloads are unstable, I temporarily set a proxy in Windows PowerShell:
$env:http_proxy = "http://127.0.0.1:7890"
$env:https_proxy = "http://127.0.0.1:7890"
On macOS, paste the proxy command provided by your client into the terminal, or just use the system VPN.
2. Configure SSH keys (GitHub)
# Generate an SSH key (if you don't have one)
ssh-keygen -t ed25519 -C "your-email@gmail.com"
# Show the public key content
cat ~/.ssh/id_ed25519.pub
Copy the public key to your GitHub account under SSH Keys. After that, you can pull/push over SSH. (Tip: If you already have a working SSH key, you can skip this step.)
3. Install and sign in to GitHub CLI (gh) (I like creating repos from the terminal)
# Install gh on Windows (via package manager)
winget install GitHub.cli
# Linux
sudo apt update
sudo apt install gh
# Sign in to GitHub (pick GitHub.com, SSH, existing key, browser login as prompted)
gh auth login
# Verify auth status
gh auth status
4. Clone and basic Git operations
# Clone via SSH
git clone git@github.com:username/repository-name.git
# Or clone with gh
gh repo clone username/repository-name
cd repository-name
git status
Common Git commands (assuming main as the default branch):
# View/configure remotes
git remote -v
# Pull latest changes
git pull origin main
# Branch management
git branch -a # list branches
git checkout branch-name # switch to existing branch
git checkout -b new-branch # create and switch
git branch -d branch-name # delete local branch
# Commit and push
git add .
git commit -m "Describe this change"
git push
5. Hugging Face: install and log in
# Install the command line tools
pip install -U "huggingface_hub[cli]"
# First create an Access Token on the website (Write permission), then log into the CLI
huggingface-cli login
Paste the token and press Enter as prompted.
# Verify identity
huggingface-cli whoami
6. Host models/datasets on Hugging Face
My usual workflow is to first create a remote repo (optionally private), then upload the current directory:
# Create a private repo (example name for a dataset)
huggingface-cli repo create --private my_datasets
# Upload (hf upload: left side is remote namespace/repo, right side is the local dir)
hf upload username/my_datasets .
To download into a specific local directory, remember to include –local-dir:
# Download the dataset to the data directory
huggingface-cli download username/my_datasets --local-dir data
# If the repo root contains a README, it will be placed directly in the target directory rather than nested
hf download username/project --local-dir output
# Result: output/README.md