[IGBF-3469] Investigate: Mirror Loraine Lab repositories on GitHub - JIRA UNCC

Details

Type: Task
Status: Closed (View Workflow)
Priority: Major
Resolution: Done
Affects Version/s: None
Fix Version/s: None
Labels:
None

Story Points:
2
Epic Link:
Deploy and maintain infrastructure
Sprint:
Spring 2, Spring 3

Description

We can reach a wider audience of developers and scientists if we mirror all our repositories on github.

For this task:

Investigate ways to automate mirroring and updating lorainelab bitbucket repositories on github
Investigate if there are ansible playbooks / modules written that could help with this

Attachments

Issue Links

is blocked by

IGBF-4080 Investigate Parallel Use of GitHub with Existing Bitbucket Flows

Closed

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Ann Loraine added a comment - 14/Oct/23 4:38 PM

git has a "mirror" option:

https://gist.github.com/lsloan/ce704da0d62ce3808dbc12e5a37ba8fc

Show

Ann Loraine added a comment - 14/Oct/23 4:38 PM git has a "mirror" option: https://gist.github.com/lsloan/ce704da0d62ce3808dbc12e5a37ba8fc

Hide

Permalink

Pranav Bhatia (Inactive) added a comment - 13/Feb/25 2:30 PM - edited

Attached is the bitbucket-pipeline. This Bitbucket Pipeline YAML automates the synchronization of Bitbucket public repositories with GitHub, ensuring a 1:1 mirror between both platforms.

Capabilities of the Pipeline:

Auto-Creates GitHub Repositories – If a repository doesn’t exist on GitHub, it is automatically created.
Full Repository Mirroring – Uses git push --mirror to sync all branches, tags, commits, and deletions from Bitbucket to GitHub.
Continuous Updates – Ensures that new commits in Bitbucket are automatically reflected in GitHub.
Authentication Handling – Securely integrates Bitbucket App Passwords and GitHub PAT Tokens for seamless automation.
Error Handling & Debugging – Includes logging to track sync issues and authentication failures.

Use Cases Covered:

One-Time Migration – Easily transfer repositories from Bitbucket to GitHub.
Ongoing Sync – Ensures repositories in GitHub stay continuously updated with new Bitbucket changes.
Branch & Tag Management – Keeps all branches and tags identical between platforms.
Repo Cleanup Enforcement – Automatically deletes branches/tags in GitHub if removed from Bitbucket.

Variables to Add in Bitbucket (Repository Variables) -
These variables should be created under Bitbucket -> Repository Settings -> Repository Variables

BITBUCKET_USERNAME (Bitbucket) Used for API authentication.
BITBUCKET_APP_PASSWORD (Bitbucket) Enables repository access. (Repository: Read, Repository: Write, Account: Read)
BITBUCKET_WORKSPACE (Bitbucket) Identifies the workspace.
GITHUB_USERNAME (GitHub) Used for API authentication.
GITHUB_TOKEN (GitHub) Grants API access for repo creation & pushing.

Link to repository - https://bitbucket.org/pranavbhatia1999/bitbucket-to-github-sync/src/main/

Show

Pranav Bhatia (Inactive) added a comment - 13/Feb/25 2:30 PM - edited Attached is the bitbucket-pipeline. This Bitbucket Pipeline YAML automates the synchronization of Bitbucket public repositories with GitHub, ensuring a 1:1 mirror between both platforms. Capabilities of the Pipeline: Auto-Creates GitHub Repositories – If a repository doesn’t exist on GitHub, it is automatically created. Full Repository Mirroring – Uses git push --mirror to sync all branches, tags, commits, and deletions from Bitbucket to GitHub. Continuous Updates – Ensures that new commits in Bitbucket are automatically reflected in GitHub. Authentication Handling – Securely integrates Bitbucket App Passwords and GitHub PAT Tokens for seamless automation. Error Handling & Debugging – Includes logging to track sync issues and authentication failures. Use Cases Covered: One-Time Migration – Easily transfer repositories from Bitbucket to GitHub. Ongoing Sync – Ensures repositories in GitHub stay continuously updated with new Bitbucket changes. Branch & Tag Management – Keeps all branches and tags identical between platforms. Repo Cleanup Enforcement – Automatically deletes branches/tags in GitHub if removed from Bitbucket. Variables to Add in Bitbucket (Repository Variables) - These variables should be created under Bitbucket -> Repository Settings -> Repository Variables BITBUCKET_USERNAME (Bitbucket) Used for API authentication. BITBUCKET_APP_PASSWORD (Bitbucket) Enables repository access. (Repository: Read, Repository: Write, Account: Read) BITBUCKET_WORKSPACE (Bitbucket) Identifies the workspace. GITHUB_USERNAME (GitHub) Used for API authentication. GITHUB_TOKEN (GitHub) Grants API access for repo creation & pushing. Link to repository - https://bitbucket.org/pranavbhatia1999/bitbucket-to-github-sync/src/main/

Hide

Permalink

Ann Loraine added a comment - 14/Feb/25 5:55 PM

I am not able to view the contents of the bitbucket repository mentioned in the previous comments. A request: Could you paste the bitbucket pipeline yaml file into a comment? You can wrap the a comment in code tags, like this:

Here is some code.

This way I can easily look at the yaml and go from there.

Show

Ann Loraine added a comment - 14/Feb/25 5:55 PM I am not able to view the contents of the bitbucket repository mentioned in the previous comments. A request: Could you paste the bitbucket pipeline yaml file into a comment? You can wrap the a comment in code tags, like this: Here is some code. This way I can easily look at the yaml and go from there.

Hide

Permalink

Pranav Bhatia (Inactive) added a comment - 14/Feb/25 6:19 PM

image: debian:latest

pipelines:
  default:
    - step:
        name: "Sync Public Bitbucket Repositories to GitHub"
        script:
          # Install necessary tools
          - apt-get update && apt-get install -y curl jq git
          - curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
          - echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null
          - apt update && apt install gh -y

          # Fetch only public repositories from Bitbucket
          - |
            PUBLIC_REPOS=$(curl -s -u "${BITBUCKET_USERNAME}:${BITBUCKET_APP_PASSWORD}" \
              "https://api.bitbucket.org/2.0/repositories/${BITBUCKET_WORKSPACE}?pagelen=100" | \
              jq -r '.values[] | select(.is_private == false) | .slug')

            echo "Public repositories found: $PUBLIC_REPOS"

            # Loop through public repositories and sync them
            for REPO_NAME in $PUBLIC_REPOS; do
              echo "Processing public repository: $REPO_NAME"

              # Check if the repository exists on GitHub
              if GITHUB_TOKEN=${GITHUB_TOKEN} gh repo view "$GITHUB_USERNAME/$REPO_NAME" > /dev/null 2>&1; then
                echo "GitHub repository $REPO_NAME already exists."
              else
                echo "GitHub repository $REPO_NAME does NOT exist. Creating now..."
                GITHUB_TOKEN=${GITHUB_TOKEN} gh repo create "$GITHUB_USERNAME/$REPO_NAME" --private --confirm
              fi

              # Clone the Bitbucket repository
              echo "Cloning Bitbucket repository: $REPO_NAME"
              git clone --bare https://${BITBUCKET_USERNAME}:${BITBUCKET_APP_PASSWORD}@bitbucket.org/${BITBUCKET_WORKSPACE}/${REPO_NAME}.git || { echo "Cloning failed"; exit 1; }

              # Navigate into the cloned repo
              cd "$REPO_NAME.git"

              # Remove Bitbucket origin and add GitHub as new remote
              echo "Switching remote to GitHub..."
              git remote remove origin
              git remote add origin https://${GITHUB_USERNAME}:${GITHUB_TOKEN}@github.com/${GITHUB_USERNAME}/${REPO_NAME}.git

              # Push all branches and tags to GitHub
              echo "Pushing repository: $REPO_NAME to GitHub..."
              git push --mirror origin || { echo "Push failed"; exit 1; }

              # Cleanup
              cd ..
              rm -rf "$REPO_NAME.git"
              echo "Repository $REPO_NAME successfully mirrored to GitHub."
            done

Show

Pranav Bhatia (Inactive) added a comment - 14/Feb/25 6:19 PM image: debian:latest pipelines: default : - step: name: "Sync Public Bitbucket Repositories to GitHub" script: # Install necessary tools - apt-get update && apt-get install -y curl jq git - curl -fsSL https: //cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg - echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https: //cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/ null - apt update && apt install gh -y # Fetch only public repositories from Bitbucket - | PUBLIC_REPOS=$(curl -s -u "${BITBUCKET_USERNAME}:${BITBUCKET_APP_PASSWORD}" \ "https: //api.bitbucket.org/2.0/repositories/${BITBUCKET_WORKSPACE}?pagelen=100" | \ jq -r '.values[] | select(.is_private == false ) | .slug') echo "Public repositories found: $PUBLIC_REPOS" # Loop through public repositories and sync them for REPO_NAME in $PUBLIC_REPOS; do echo "Processing public repository: $REPO_NAME" # Check if the repository exists on GitHub if GITHUB_TOKEN=${GITHUB_TOKEN} gh repo view "$GITHUB_USERNAME/$REPO_NAME" > /dev/ null 2>&1; then echo "GitHub repository $REPO_NAME already exists." else echo "GitHub repository $REPO_NAME does NOT exist. Creating now..." GITHUB_TOKEN=${GITHUB_TOKEN} gh repo create "$GITHUB_USERNAME/$REPO_NAME" -- private --confirm fi # Clone the Bitbucket repository echo "Cloning Bitbucket repository: $REPO_NAME" git clone --bare https: //${BITBUCKET_USERNAME}:${BITBUCKET_APP_PASSWORD}@bitbucket.org/${BITBUCKET_WORKSPACE}/${REPO_NAME}.git || { echo "Cloning failed" ; exit 1; } # Navigate into the cloned repo cd "$REPO_NAME.git" # Remove Bitbucket origin and add GitHub as new remote echo "Switching remote to GitHub..." git remote remove origin git remote add origin https: //${GITHUB_USERNAME}:${GITHUB_TOKEN}@github.com/${GITHUB_USERNAME}/${REPO_NAME}.git # Push all branches and tags to GitHub echo "Pushing repository: $REPO_NAME to GitHub..." git push --mirror origin || { echo "Push failed" ; exit 1; } # Cleanup cd .. rm -rf "$REPO_NAME.git" echo "Repository $REPO_NAME successfully mirrored to GitHub." done

Hide

Permalink

Ann Loraine added a comment - 26/Feb/25 12:22 PM

Interesting!!!

I think the way this can work (in deployment) is that I can make a single "mirror-maker" repository, configure that repository with repository variables as indicated above, and then, BOOM, all the loraine lab repositories will have mirrors created in a github account.

Thus, it looks like I don't need to make any changes at all to my individual repositories in bitbucket?

Show

Ann Loraine added a comment - 26/Feb/25 12:22 PM Interesting!!! I think the way this can work (in deployment) is that I can make a single "mirror-maker" repository, configure that repository with repository variables as indicated above, and then, BOOM, all the loraine lab repositories will have mirrors created in a github account. Thus, it looks like I don't need to make any changes at all to my individual repositories in bitbucket?

Hide

Permalink

Ann Loraine added a comment - 26/Feb/25 1:56 PM - edited

Investigating github:

(1)

https://github.com/integrated-genome-browser
https://github.com/orgs/integrated-genome-browser/repositories

this is an "org" (not sure what that means)
it has only one repository, currently
that repository appears to be a github.io web site repository

(2)

https://github.com/LoraineLabResearch

this is also an "org"
it had some repositories, a couple of forks and one almost empty "gittest"
deleted all the existing repositories because they are not needed

I think I will use "LoraineLabResearch" as the target for mirroring the LoraineLab group's bitbucket-hosted repositories

Show

Ann Loraine added a comment - 26/Feb/25 1:56 PM - edited Investigating github: (1) https://github.com/integrated-genome-browser https://github.com/orgs/integrated-genome-browser/repositories this is an "org" (not sure what that means) it has only one repository, currently that repository appears to be a github.io web site repository (2) https://github.com/LoraineLabResearch this is also an "org" it had some repositories, a couple of forks and one almost empty "gittest" deleted all the existing repositories because they are not needed I think I will use "LoraineLabResearch" as the target for mirroring the LoraineLab group's bitbucket-hosted repositories

Hide

Permalink

Ann Loraine added a comment - 26/Feb/25 2:46 PM

A question for Pranav Bhatia about the github access token:

Does it need to be a "classic" token?

Reference:
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token)

Show

Ann Loraine added a comment - 26/Feb/25 2:46 PM A question for Pranav Bhatia about the github access token: Does it need to be a "classic" token? Reference: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token )

Hide

Permalink

Ann Loraine added a comment - 26/Feb/25 2:56 PM - edited

Update:

Made a new lorainelab workspace public repository https://bitbucket.org/lorainelab/bitbucket-to-github-sync
added all the repository variables except the bitbucket app password (BITBUCKET_APP_PASSWORD) and the github token (GITHUB_TOKEN) because I have not made those yet

Customization:

I don't want the pipeline to run every time I push to the repository; considering renaming pipeline "default" to something else
To make the pipeline not the default, add it under category "custom" with pipeline name "mirror_to_github" (integrated genome browser pipeline has an example)

Show

Ann Loraine added a comment - 26/Feb/25 2:56 PM - edited Update: Made a new lorainelab workspace public repository https://bitbucket.org/lorainelab/bitbucket-to-github-sync added all the repository variables except the bitbucket app password (BITBUCKET_APP_PASSWORD) and the github token (GITHUB_TOKEN) because I have not made those yet Customization: I don't want the pipeline to run every time I push to the repository; considering renaming pipeline "default" to something else To make the pipeline not the default, add it under category "custom" with pipeline name "mirror_to_github" (integrated genome browser pipeline has an example)

Hide

Permalink

Ann Loraine added a comment - 26/Feb/25 3:03 PM - edited

Update:

My first commit of an initial default "starter" bitbucket pipeline YML configuration file printed some useful stuff, including a list of default repository variables, including:

Default variables:
BITBUCKET_BRANCH
BITBUCKET_BUILD_NUMBER
BITBUCKET_CLONE_DIR
BITBUCKET_COMMIT
BITBUCKET_GIT_HTTP_ORIGIN
BITBUCKET_GIT_SSH_ORIGIN
BITBUCKET_PARALLEL_STEP
BITBUCKET_PARALLEL_STEP_COUNT
BITBUCKET_PIPELINE_UUID
BITBUCKET_PROJECT_KEY
BITBUCKET_PROJECT_UUID
BITBUCKET_REPO_FULL_NAME
BITBUCKET_REPO_IS_PRIVATE
BITBUCKET_REPO_OWNER
BITBUCKET_REPO_OWNER_UUID
BITBUCKET_REPO_SLUG
BITBUCKET_REPO_UUID
BITBUCKET_SSH_KEY_FILE
BITBUCKET_STEP_RUN_NUMBER
BITBUCKET_STEP_TRIGGERER_UUID
BITBUCKET_STEP_UUID
BITBUCKET_WORKSPACE

I guess I did not need to manually add BITBUCKET_WORKSPACE variable, since this repository is in the same workspace as the repositories I want to mirrow.

Show

Ann Loraine added a comment - 26/Feb/25 3:03 PM - edited Update: My first commit of an initial default "starter" bitbucket pipeline YML configuration file printed some useful stuff, including a list of default repository variables, including: Default variables: BITBUCKET_BRANCH BITBUCKET_BUILD_NUMBER BITBUCKET_CLONE_DIR BITBUCKET_COMMIT BITBUCKET_GIT_HTTP_ORIGIN BITBUCKET_GIT_SSH_ORIGIN BITBUCKET_PARALLEL_STEP BITBUCKET_PARALLEL_STEP_COUNT BITBUCKET_PIPELINE_UUID BITBUCKET_PROJECT_KEY BITBUCKET_PROJECT_UUID BITBUCKET_REPO_FULL_NAME BITBUCKET_REPO_IS_PRIVATE BITBUCKET_REPO_OWNER BITBUCKET_REPO_OWNER_UUID BITBUCKET_REPO_SLUG BITBUCKET_REPO_UUID BITBUCKET_SSH_KEY_FILE BITBUCKET_STEP_RUN_NUMBER BITBUCKET_STEP_TRIGGERER_UUID BITBUCKET_STEP_UUID BITBUCKET_WORKSPACE I guess I did not need to manually add BITBUCKET_WORKSPACE variable, since this repository is in the same workspace as the repositories I want to mirrow.

Hide

Permalink

Pranav Bhatia (Inactive) added a comment - 27/Feb/25 1:12 AM

Ann Loraine Yes, we need the classic personal access token.

You can follow the steps below -

Go to GitHub - Settings - Developer Settings - Personal Access Tokens.
Click Generate new token (classic).
Select repo scope (for creating repositories).

Show

Pranav Bhatia (Inactive) added a comment - 27/Feb/25 1:12 AM Ann Loraine Yes, we need the classic personal access token. You can follow the steps below - Go to GitHub - Settings - Developer Settings - Personal Access Tokens. Click Generate new token (classic). Select repo scope (for creating repositories).

Hide

Permalink

Ann Loraine added a comment - 27/Feb/25 9:26 AM - edited

Thanks Pranav Bhatia!

Update:

I think I might change an aspect of how the cloning is happening. Instead of re-aliasing "origin" with the github repository, I might add the github repository as a new remote, calling it "downstream" or something like that.
I also am considering dividing the workflow into two different pipelines or steps: one for repository creation and another one for repository updating...not sure about that yet, however.

Show

Ann Loraine added a comment - 27/Feb/25 9:26 AM - edited Thanks Pranav Bhatia ! Update: I think I might change an aspect of how the cloning is happening. Instead of re-aliasing "origin" with the github repository, I might add the github repository as a new remote, calling it "downstream" or something like that. I also am considering dividing the workflow into two different pipelines or steps: one for repository creation and another one for repository updating...not sure about that yet, however.

Hide

Permalink

Ann Loraine added a comment - 27/Feb/25 9:28 AM

Also, I think I should create a new ticket to cover the work I am doing now to deploy the new functionality to our lorainelab workspace.

Thank you Pranav Bhatia for your work! I am looking forward to getting this up and running on the entire lorainelab workspace!

Show

Ann Loraine added a comment - 27/Feb/25 9:28 AM Also, I think I should create a new ticket to cover the work I am doing now to deploy the new functionality to our lorainelab workspace. Thank you Pranav Bhatia for your work! I am looking forward to getting this up and running on the entire lorainelab workspace!

People

Assignee:

Pranav Bhatia (Inactive)

Reporter:

Ann Loraine

Votes:

0 Vote for this issue

Watchers:

2 Start watching this issue

Dates

Created:

14/Oct/23 3:28 PM

Updated:

27/Feb/25 9:32 AM

Resolved:

27/Feb/25 9:32 AM