Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-1993

Customize settings.xml in igb-maven-install4j Docker image

    Details

    • Type: Improvement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:

      Description

      To release IGB and other project artifacts to our Nexus maven repository, the release engineer (Dr. Loraine) manually runs "mvn deploy", which builds the artifacts and uploads them Nexus.

      This works OK so far, but there are some problems:

      • The build environment that gets used depends on whatever JRE or maven version that the release engineer has installed on their computer.
      • Write credentials for the Nexus maven repository have to be stored locally in the user's mvn settings.xml file.
      • The release engineer has to remember to deploy new artifacts.
      • The current process is manual. Mistakes are easy to make.

      Solution:

      We are already building artifacts using bitbucket pipelines. Let's enhance these pipelines to include an optional "deploy" step that uploads to our nexus repository.

      To do this, we need to customize the Docker image we created for building Java projects. See:

      Following the suggestion of Robert Scholte (see above link), we will add variables for Nexus user name and password to the settings file.

      Once that is done, we'll configure the bitbucket repository with repository environment variables and reference these in the maven build command.

      As a first step, we need to update the mvn settings in the Docker image we use for building IGB (and other java projects).

      Links:

      repo with Docker file, which will need to be edited:

      Loraine Lab Docker hub image

      Requirements:

      • Each Docker image requires a tag. The latest image is tagged with "jre-1.8.0_212". Tag the new Docker image with "jre-1.8.0_212-mvn-deploy". This ensures that you can deploy the new image without disrupting existing build pipelines, which are all referencing the image tagged with jre-1.8.0_212.

      Questions:

      • Where should the settings.xml file reside in the Docker image? This can be a user-level or system-level file. Which should this be?
      • Are there any maven commands that allow modifying settings? If yes, this could simplify the customization process.

        Attachments

          Issue Links

            Activity

            Hide
            ann.loraine Ann Loraine added a comment -

            Note to implementer:

            • Sign up for a free account at Docker Hub
            • Ask Dr. Loraine to add your user to "installer" team within lorainelab organization
            Show
            ann.loraine Ann Loraine added a comment - Note to implementer: Sign up for a free account at Docker Hub Ask Dr. Loraine to add your user to "installer" team within lorainelab organization
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            Prof. [~aloraine], could you please add 'pkulka10' to "installer" team within lorainelab organization?
            Thanks.

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - Prof. [~aloraine] , could you please add 'pkulka10' to "installer" team within lorainelab organization? Thanks.
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment - - edited

            Updates:
            1) I have cloned and created a new branch for both genoviz-sdk & integrated-genome-browser-docker.
            2) I have currently packed settings.xml file in the docker image.
            3) Deployed the image on my docker-hub account repo.
            4) Changed the image name to new image name in the bitbucket configuration file for genoviz-sdk.
            5) Tried to build it normally just to check if there are any problems with the image which is newly built.
            6) The build was successful.
            7) Added environment variables for username and password in order to pass them as a command line argument.
            8) Tried to incorporate maven deploy command in the bitbucket pipeline configuration file.

            Results: It was failing to build the pipeline because of the maven deploy command.

            Course of action:
            1) I'll try to follow the steps which are mentioned on:
            https://stackoverflow.com/questions/24830610/why-am-i-getting-a-401-unauthorized-error-in-maven
            as the person was facing the same problem.
            2) I will also try to post any local file to the test nexus repository just to make sure that there is no problem from repository side like access rights so that if we missed out on something then we could add it accordingly.
            4) I will try to follow the same steps on local a project.

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - - edited Updates: 1) I have cloned and created a new branch for both genoviz-sdk & integrated-genome-browser-docker. 2) I have currently packed settings.xml file in the docker image. 3) Deployed the image on my docker-hub account repo. 4) Changed the image name to new image name in the bitbucket configuration file for genoviz-sdk. 5) Tried to build it normally just to check if there are any problems with the image which is newly built. 6) The build was successful. 7) Added environment variables for username and password in order to pass them as a command line argument. 8) Tried to incorporate maven deploy command in the bitbucket pipeline configuration file. Results: It was failing to build the pipeline because of the maven deploy command. Course of action: 1) I'll try to follow the steps which are mentioned on: https://stackoverflow.com/questions/24830610/why-am-i-getting-a-401-unauthorized-error-in-maven as the person was facing the same problem. 2) I will also try to post any local file to the test nexus repository just to make sure that there is no problem from repository side like access rights so that if we missed out on something then we could add it accordingly. 4) I will try to follow the same steps on local a project.
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Questions:

            • Are you able to use the credentials provided (separately) to log in to the nexus repository site?
            • If you are able to do the above, can you use the Nexus Web UI to upload an artifact to the test repository?

            If the answer to either of the above questions is "no" then it is possible the user created for your testing does not have the correct permissions set. If yes, please let me know and I will investigate.

            Show
            ann.loraine Ann Loraine added a comment - - edited Questions: Are you able to use the credentials provided (separately) to log in to the nexus repository site? If you are able to do the above, can you use the Nexus Web UI to upload an artifact to the test repository? If the answer to either of the above questions is "no" then it is possible the user created for your testing does not have the correct permissions set. If yes, please let me know and I will investigate.
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            Prof. [~aloraine], Yeah I was able to login into nexus repo.
            I will try to upload an artifact manually and will let you know if that works out.

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - Prof. [~aloraine] , Yeah I was able to login into nexus repo. I will try to upload an artifact manually and will let you know if that works out.
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            Prof. [~aloraine], there is nothing wrong with the credentials as I am able to login to the NEXUS repo.
            But, when I am trying to upload any artifact, it is not allowing me to do so.
            I have tried using the CURL command as well as mvn deploy command.
            Below are the commands which were used to upload an artifact.
            mvn deploy:deploy-file -DgeneratePom=false -DrepositoryId=nexus -Durl=https://nexus.bioviz.org/repository/test-maven2 -DpomFile=pom.xml -Dfile=target/genoviz-1.1.8.jar
            curl -v -u pruthakulkarni:ndorwaedleo5883 --upload-file pom.xml https://nexus.bioviz.org/repository/test-maven2
            Do you have any idea about it?

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - Prof. [~aloraine] , there is nothing wrong with the credentials as I am able to login to the NEXUS repo. But, when I am trying to upload any artifact, it is not allowing me to do so. I have tried using the CURL command as well as mvn deploy command. Below are the commands which were used to upload an artifact. mvn deploy:deploy-file -DgeneratePom=false -DrepositoryId=nexus -Durl= https://nexus.bioviz.org/repository/test-maven2 -DpomFile=pom.xml -Dfile=target/genoviz-1.1.8.jar curl -v -u pruthakulkarni:ndorwaedleo5883 --upload-file pom.xml https://nexus.bioviz.org/repository/test-maven2 Do you have any idea about it?
            Hide
            ann.loraine Ann Loraine added a comment -

            Probably the permissions are not properly set for your Nexus user. I would suggest setting this aside and helping with testing tasks for the rest of the day. (I won't be able to look at the permissions until later.)

            Show
            ann.loraine Ann Loraine added a comment - Probably the permissions are not properly set for your Nexus user. I would suggest setting this aside and helping with testing tasks for the rest of the day. (I won't be able to look at the permissions until later.)
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            Sure. I will go ahead and start testing a few tickets for today.

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - Sure. I will go ahead and start testing a few tickets for today.
            Hide
            ann.loraine Ann Loraine added a comment -

            Prutha Kulkarni you should now be able to upload artifacts to test-maven2 repository. (I reconfigured permissions.) To test this, log in. You should now see an "upload" link on the left side of the page in "Browser Server Contents" view.

            Show
            ann.loraine Ann Loraine added a comment - Prutha Kulkarni you should now be able to upload artifacts to test-maven2 repository. (I reconfigured permissions.) To test this, log in. You should now see an "upload" link on the left side of the page in "Browser Server Contents" view.
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment - - edited

            Updates:
            I tried to upload artifacts using my local setup and the attempt was successful. When I tried to do the same this on my clone in bitbucket, the build was failing. I figured that the build is not picking up our customised settings.xml file so, I changed the docker image file a bit again and this time I copied the settings.xml to "/root/.m2/" instead of "/usr/share/maven/ref" folder to make settings.xml file accessible to all maven commands that are available in the bitbucket pipeline file. And the attempt was successful. Now we can deploy the artifacts to nexus repository with the help of bitbucket pipelines.

            Steps taken:
            1) Cloned my branch of integrated-genome-browser-docker to the local system.
            2) Install docker application on system and started the docker daemon.
            3) Logged into the docker hub and created one repository and given access of bitbucket to the repository.
            4) Built the docker image using command :
            "docker build -t pkula10/jre-1.8.0_212-mvn-deploy ."
            5) Logged in to the docker hub with docker hub credentials using following command : "docker login iindex.docker.io"
            6) Pushed the docker image to my account repository using following command:
            "docker push pkula10/jre-1.8.0_212-mvn-deploy"
            7) customized "distributionManagement" section in the parent pom of "genoviz sdk" project with :
            id : test-maven2
            name: test-maven2
            url: https://nexus.bioviz.org/repository/test-maven2
            8) Updated docker image in the bitbucket pipeline file.
            9) Added custom pipeline for the mvn deploy command which refers our custom settings.xml.
            Also, for this added two environment variables for the credentials of nexus repo.

            The code changes can be found at:
            https://bitbucket.org/pkulka10/igb_docker/src/IGBF-1993/
            https://bitbucket.org/pkulka10/new_genoviz/src/IGBF-1945/

            For testing:
            1) Go to the Branches section of my fork and click on the "..." present at the end of IGBF-1945 branch row.
            2) Click on Run pipeline for a branch option and select custom pipeline "deployToNEXUSRepo".
            3) Click on Run and wait for the pipeline to build.
            4) After the build you can log into my https://nexus.bioviz.org/ account
            Please ask the credentials whenever you are testing it.
            5) Go to test-maven2 repository. you will find the uploaded artifact directory there.
            You can also compare the content of the directory of maven-release repository with the test-maven2 repository just to be sure if the artifact is uploaded properly.

            Note: The image value is changed in bitbucket pipeline for genoviz sdk and it is pointing to the latest docker image in my account. I have changed it for the code review and testing purpose. Let me know once the review is finished. I will change it back to the original one and will submit a pull request for the same.

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - - edited Updates: I tried to upload artifacts using my local setup and the attempt was successful. When I tried to do the same this on my clone in bitbucket, the build was failing. I figured that the build is not picking up our customised settings.xml file so, I changed the docker image file a bit again and this time I copied the settings.xml to "/root/.m2/" instead of "/usr/share/maven/ref" folder to make settings.xml file accessible to all maven commands that are available in the bitbucket pipeline file. And the attempt was successful. Now we can deploy the artifacts to nexus repository with the help of bitbucket pipelines. Steps taken: 1) Cloned my branch of integrated-genome-browser-docker to the local system. 2) Install docker application on system and started the docker daemon. 3) Logged into the docker hub and created one repository and given access of bitbucket to the repository. 4) Built the docker image using command : "docker build -t pkula10/jre-1.8.0_212-mvn-deploy ." 5) Logged in to the docker hub with docker hub credentials using following command : "docker login iindex.docker.io" 6) Pushed the docker image to my account repository using following command: "docker push pkula10/jre-1.8.0_212-mvn-deploy" 7) customized "distributionManagement" section in the parent pom of "genoviz sdk" project with : id : test-maven2 name: test-maven2 url: https://nexus.bioviz.org/repository/test-maven2 8) Updated docker image in the bitbucket pipeline file. 9) Added custom pipeline for the mvn deploy command which refers our custom settings.xml. Also, for this added two environment variables for the credentials of nexus repo. The code changes can be found at: https://bitbucket.org/pkulka10/igb_docker/src/IGBF-1993/ https://bitbucket.org/pkulka10/new_genoviz/src/IGBF-1945/ For testing: 1) Go to the Branches section of my fork and click on the "..." present at the end of IGBF-1945 branch row. 2) Click on Run pipeline for a branch option and select custom pipeline "deployToNEXUSRepo". 3) Click on Run and wait for the pipeline to build. 4) After the build you can log into my https://nexus.bioviz.org/ account Please ask the credentials whenever you are testing it. 5) Go to test-maven2 repository. you will find the uploaded artifact directory there. You can also compare the content of the directory of maven-release repository with the test-maven2 repository just to be sure if the artifact is uploaded properly. Note: The image value is changed in bitbucket pipeline for genoviz sdk and it is pointing to the latest docker image in my account. I have changed it for the code review and testing purpose. Let me know once the review is finished. I will change it back to the original one and will submit a pull request for the same.
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            Prof. [~aloraine], I have made the changed that were suggested by you.
            Please check the latest commits for the same.
            Links are given below:
            https://bitbucket.org/pkulka10/igb_docker/commits/9564f6e05be82e895c80b59f0a5c88b9c65d739b
            https://bitbucket.org/pkulka10/new_genoviz/commits/4d2ad88ea59022ed65b29dc221d8323190beefc6

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - Prof. [~aloraine] , I have made the changed that were suggested by you. Please check the latest commits for the same. Links are given below: https://bitbucket.org/pkulka10/igb_docker/commits/9564f6e05be82e895c80b59f0a5c88b9c65d739b https://bitbucket.org/pkulka10/new_genoviz/commits/4d2ad88ea59022ed65b29dc221d8323190beefc6
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Thank you Prutha Kulkarni!

            When you get a moment, please push the newest Docker image to the LoraineLab Team Docker Hub account.

            HOWEVER - please be careful! The Loraine Lab production image needs two coordinates: image name AND tag. In your Docker Hub account (pkula10), you are not using using tags. For production, however, we need BOTH*.

            To publish the Docker image to the production Docker Hub account, you need to push to

            using this command (from above):

            • docker push lorainelab/igb-maven-install4j:tagname

            With "tagname" as "jre-1.8.0-212-mvn-deploy"

            Note that in the above docker repository, every image has the same name with DIFFERENT tags. This allows us to easily track down JRE-specific errors and other such problems, while also retaining our ability to build older release branches as needed.

            Show
            ann.loraine Ann Loraine added a comment - - edited Thank you Prutha Kulkarni ! When you get a moment, please push the newest Docker image to the LoraineLab Team Docker Hub account. HOWEVER - please be careful! The Loraine Lab production image needs two coordinates: image name AND tag. In your Docker Hub account (pkula10), you are not using using tags. For production, however, we need BOTH *. To publish the Docker image to the production Docker Hub account, you need to push to https://cloud.docker.com/u/lorainelab/repository/docker/lorainelab/igb-maven-install4j using this command (from above): docker push lorainelab/igb-maven-install4j:tagname With "tagname" as "jre-1.8.0-212-mvn-deploy" Note that in the above docker repository, every image has the same name with DIFFERENT tags. This allows us to easily track down JRE-specific errors and other such problems, while also retaining our ability to build older release branches as needed.
            Hide
            ann.loraine Ann Loraine added a comment -

            This can wait until next week – don't worry about doing this today if you are not in Kannapolis. Take your time and don't rush.

            Show
            ann.loraine Ann Loraine added a comment - This can wait until next week – don't worry about doing this today if you are not in Kannapolis. Take your time and don't rush.
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            Prof. [~aloraine], I have noted the things mentioned above. Whenever I am in Kannapolis next week, I will push the changes to production docker hub account carefully.
            Thanks.

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - Prof. [~aloraine] , I have noted the things mentioned above. Whenever I am in Kannapolis next week, I will push the changes to production docker hub account carefully. Thanks.
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            Prof. [~aloraine], I have deployed the latest docker image to lorainlab account with name "lorainelab/igb-maven-install4j:jre-1.8.0-212-mvn-deploy".
            Also, I have tested it on my genoviz sdk fork. Since, I was using the changed bitbucket configuration of genoviz sdk to test the changes in the docker image. So I wanted to know if I can submit a pull request for genoviz repository also (with changed docker image name as well as new bitbucket configuration)?
            I am confused because we were using its pipeline configuration for testing even though there is a different ticket related to genoviz sdk which is IGBF-1945.
            Please let me know the same.

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - Prof. [~aloraine] , I have deployed the latest docker image to lorainlab account with name "lorainelab/igb-maven-install4j:jre-1.8.0-212-mvn-deploy". Also, I have tested it on my genoviz sdk fork. Since, I was using the changed bitbucket configuration of genoviz sdk to test the changes in the docker image. So I wanted to know if I can submit a pull request for genoviz repository also (with changed docker image name as well as new bitbucket configuration)? I am confused because we were using its pipeline configuration for testing even though there is a different ticket related to genoviz sdk which is IGBF-1945 . Please let me know the same.
            Hide
            ann.loraine Ann Loraine added a comment -

            Thank you Prutha Kulkarni

            Please submit 2 PRs:

            • to genoviz sdk repository (to master)
            • to igb docker repository (if not already merged - new settings.xml file and Docker file as required)
            Show
            ann.loraine Ann Loraine added a comment - Thank you Prutha Kulkarni Please submit 2 PRs: to genoviz sdk repository (to master) to igb docker repository (if not already merged - new settings.xml file and Docker file as required)
            Hide
            ann.loraine Ann Loraine added a comment -

            Update:

            • igb docker PR merged
            Show
            ann.loraine Ann Loraine added a comment - Update: igb docker PR merged
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            Prof. [~aloraine], I have also submitted a PR for genoviz sdk.
            code changes are present at: https://bitbucket.org/lorainelab/genoviz-sdk/pull-requests/17/igbf-1945/diff

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - Prof. [~aloraine] , I have also submitted a PR for genoviz sdk. code changes are present at: https://bitbucket.org/lorainelab/genoviz-sdk/pull-requests/17/igbf-1945/diff
            Hide
            ann.loraine Ann Loraine added a comment -

            I am able to build Genoviz SDK using the new Docker and configured as in above. The new Docker image appears to be working great. Marking this as "closed".

            Show
            ann.loraine Ann Loraine added a comment - I am able to build Genoviz SDK using the new Docker and configured as in above. The new Docker image appears to be working great. Marking this as "closed".

              People

              • Assignee:
                prutha Prutha Kulkarni (Inactive)
                Reporter:
                ann.loraine Ann Loraine
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: