Details
-
Type:
Task
-
Status: Closed (View Workflow)
-
Priority:
Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:None
-
Story Points:2
-
Epic Link:
-
Sprint:Spring 2, Spring 3
Description
We can reach a wider audience of developers and scientists if we mirror all our repositories on github.
For this task:
- Investigate ways to automate mirroring and updating lorainelab bitbucket repositories on github
- Investigate if there are ansible playbooks / modules written that could help with this
Attachments
Issue Links
- is blocked by
-
-
- Closed
-
Activity
Attached is the bitbucket-pipeline. This Bitbucket Pipeline YAML automates the synchronization of Bitbucket public repositories with GitHub, ensuring a 1:1 mirror between both platforms.
Capabilities of the Pipeline:
- Auto-Creates GitHub Repositories – If a repository doesn’t exist on GitHub, it is automatically created.
- Full Repository Mirroring – Uses git push --mirror to sync all branches, tags, commits, and deletions from Bitbucket to GitHub.
- Continuous Updates – Ensures that new commits in Bitbucket are automatically reflected in GitHub.
- Authentication Handling – Securely integrates Bitbucket App Passwords and GitHub PAT Tokens for seamless automation.
- Error Handling & Debugging – Includes logging to track sync issues and authentication failures.
Use Cases Covered:
- One-Time Migration – Easily transfer repositories from Bitbucket to GitHub.
- Ongoing Sync – Ensures repositories in GitHub stay continuously updated with new Bitbucket changes.
- Branch & Tag Management – Keeps all branches and tags identical between platforms.
- Repo Cleanup Enforcement – Automatically deletes branches/tags in GitHub if removed from Bitbucket.
Variables to Add in Bitbucket (Repository Variables) -
These variables should be created under Bitbucket -> Repository Settings -> Repository Variables
BITBUCKET_USERNAME (Bitbucket) Used for API authentication.
BITBUCKET_APP_PASSWORD (Bitbucket) Enables repository access. (Repository: Read, Repository: Write, Account: Read)
BITBUCKET_WORKSPACE (Bitbucket) Identifies the workspace.
GITHUB_USERNAME (GitHub) Used for API authentication.
GITHUB_TOKEN (GitHub) Grants API access for repo creation & pushing.
Link to repository - https://bitbucket.org/pranavbhatia1999/bitbucket-to-github-sync/src/main/
I am not able to view the contents of the bitbucket repository mentioned in the previous comments. A request: Could you paste the bitbucket pipeline yaml file into a comment? You can wrap the a comment in code tags, like this:
Here is some code.
This way I can easily look at the yaml and go from there.
image: debian:latest pipelines: default: - step: name: "Sync Public Bitbucket Repositories to GitHub" script: # Install necessary tools - apt-get update && apt-get install -y curl jq git - curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg - echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null - apt update && apt install gh -y # Fetch only public repositories from Bitbucket - | PUBLIC_REPOS=$(curl -s -u "${BITBUCKET_USERNAME}:${BITBUCKET_APP_PASSWORD}" \ "https://api.bitbucket.org/2.0/repositories/${BITBUCKET_WORKSPACE}?pagelen=100" | \ jq -r '.values[] | select(.is_private == false) | .slug') echo "Public repositories found: $PUBLIC_REPOS" # Loop through public repositories and sync them for REPO_NAME in $PUBLIC_REPOS; do echo "Processing public repository: $REPO_NAME" # Check if the repository exists on GitHub if GITHUB_TOKEN=${GITHUB_TOKEN} gh repo view "$GITHUB_USERNAME/$REPO_NAME" > /dev/null 2>&1; then echo "GitHub repository $REPO_NAME already exists." else echo "GitHub repository $REPO_NAME does NOT exist. Creating now..." GITHUB_TOKEN=${GITHUB_TOKEN} gh repo create "$GITHUB_USERNAME/$REPO_NAME" --private --confirm fi # Clone the Bitbucket repository echo "Cloning Bitbucket repository: $REPO_NAME" git clone --bare https://${BITBUCKET_USERNAME}:${BITBUCKET_APP_PASSWORD}@bitbucket.org/${BITBUCKET_WORKSPACE}/${REPO_NAME}.git || { echo "Cloning failed"; exit 1; } # Navigate into the cloned repo cd "$REPO_NAME.git" # Remove Bitbucket origin and add GitHub as new remote echo "Switching remote to GitHub..." git remote remove origin git remote add origin https://${GITHUB_USERNAME}:${GITHUB_TOKEN}@github.com/${GITHUB_USERNAME}/${REPO_NAME}.git # Push all branches and tags to GitHub echo "Pushing repository: $REPO_NAME to GitHub..." git push --mirror origin || { echo "Push failed"; exit 1; } # Cleanup cd .. rm -rf "$REPO_NAME.git" echo "Repository $REPO_NAME successfully mirrored to GitHub." done
Interesting!!!
I think the way this can work (in deployment) is that I can make a single "mirror-maker" repository, configure that repository with repository variables as indicated above, and then, BOOM, all the loraine lab repositories will have mirrors created in a github account.
Thus, it looks like I don't need to make any changes at all to my individual repositories in bitbucket?
Investigating github:
(1)
https://github.com/integrated-genome-browser
https://github.com/orgs/integrated-genome-browser/repositories
- this is an "org" (not sure what that means)
- it has only one repository, currently
- that repository appears to be a github.io web site repository
(2)
https://github.com/LoraineLabResearch
- this is also an "org"
- it had some repositories, a couple of forks and one almost empty "gittest"
- deleted all the existing repositories because they are not needed
I think I will use "LoraineLabResearch" as the target for mirroring the LoraineLab group's bitbucket-hosted repositories
A question for Pranav Bhatia about the github access token:
- Does it need to be a "classic" token?
Update:
- Made a new lorainelab workspace public repository https://bitbucket.org/lorainelab/bitbucket-to-github-sync
- added all the repository variables except the bitbucket app password (BITBUCKET_APP_PASSWORD) and the github token (GITHUB_TOKEN) because I have not made those yet
Customization:
- I don't want the pipeline to run every time I push to the repository; considering renaming pipeline "default" to something else
- To make the pipeline not the default, add it under category "custom" with pipeline name "mirror_to_github" (integrated genome browser pipeline has an example)
Update:
- My first commit of an initial default "starter" bitbucket pipeline YML configuration file printed some useful stuff, including a list of default repository variables, including:
Default variables:
BITBUCKET_BRANCH
BITBUCKET_BUILD_NUMBER
BITBUCKET_CLONE_DIR
BITBUCKET_COMMIT
BITBUCKET_GIT_HTTP_ORIGIN
BITBUCKET_GIT_SSH_ORIGIN
BITBUCKET_PARALLEL_STEP
BITBUCKET_PARALLEL_STEP_COUNT
BITBUCKET_PIPELINE_UUID
BITBUCKET_PROJECT_KEY
BITBUCKET_PROJECT_UUID
BITBUCKET_REPO_FULL_NAME
BITBUCKET_REPO_IS_PRIVATE
BITBUCKET_REPO_OWNER
BITBUCKET_REPO_OWNER_UUID
BITBUCKET_REPO_SLUG
BITBUCKET_REPO_UUID
BITBUCKET_SSH_KEY_FILE
BITBUCKET_STEP_RUN_NUMBER
BITBUCKET_STEP_TRIGGERER_UUID
BITBUCKET_STEP_UUID
BITBUCKET_WORKSPACE
- I guess I did not need to manually add BITBUCKET_WORKSPACE variable, since this repository is in the same workspace as the repositories I want to mirrow.
Ann Loraine Yes, we need the classic personal access token.
You can follow the steps below -
Go to GitHub - Settings - Developer Settings - Personal Access Tokens.
Click Generate new token (classic).
Select repo scope (for creating repositories).
Thanks Pranav Bhatia!
Update:
- I think I might change an aspect of how the cloning is happening. Instead of re-aliasing "origin" with the github repository, I might add the github repository as a new remote, calling it "downstream" or something like that.
- I also am considering dividing the workflow into two different pipelines or steps: one for repository creation and another one for repository updating...not sure about that yet, however.
Also, I think I should create a new ticket to cover the work I am doing now to deploy the new functionality to our lorainelab workspace.
Thank you Pranav Bhatia for your work! I am looking forward to getting this up and running on the entire lorainelab workspace!
git has a "mirror" option: