Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-1760

Determine how to build OBR XML from bundle jar

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      The App Store will allow developers to upload their bundles for review. IGB team will then review and deploy the bundles to an OSGi bundle repository (OBR) for IGB to access.

      We need to generate text to add to IGB's OBR once a user has done this.

      A bundle's manifest file contains all the information needed to generate this text for repository.xml.

      For this task, investigate how we (the site maintainers) can generate the required text for the OBR XML.

      See:

      https://felix.apache.org/documentation/subprojects/apache-felix-osgi-bundle-repository.html

      Karaf (which uses felix) has some functions for working with OBRs:
      https://svn.apache.org/repos/asf/karaf/site/production/manual/latest/obr.html

      Felix and Karaf both have CLIs, but the Karaf CLI is better.

        Attachments

          Issue Links

            Activity

            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            There is a maven karaf plugin like felix which can be used to do this.
            Here are some useful links on how to use the plugin.
            https://svn.apache.org/repos/asf/karaf/site/production/manual/latest/karaf-maven-plugin.html
            https://gist.github.com/rocketraman/914c55ab8577c95d3408

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - There is a maven karaf plugin like felix which can be used to do this. Here are some useful links on how to use the plugin. https://svn.apache.org/repos/asf/karaf/site/production/manual/latest/karaf-maven-plugin.html https://gist.github.com/rocketraman/914c55ab8577c95d3408
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            This also may be useful:

            Above StackOverflow record points to this tool:

            According to the documentation, it looks like we can run Aries to generate a repository.xml using some existing bundles. I'm not sure, but this seems like exactly what we need. I have not tried using it, however.

            Show
            ann.loraine Ann Loraine added a comment - - edited This also may be useful: OSGi Bundle Repositories (How to host them on a Server)? - https://stackoverflow.com/questions/20337818/osgi-bundle-repositories-how-to-host-them-on-a-server Above StackOverflow record points to this tool: Repository Generation Tool from the Aries project - http://aries.apache.org/documentation/tools/repositoryGenerator.html According to the documentation, it looks like we can run Aries to generate a repository.xml using some existing bundles. I'm not sure, but this seems like exactly what we need. I have not tried using it, however.
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            I checked out the tool which is Apache Aries. It can be used to build OBR XML from the bundle jar.
            I went through the documentation in order to install the tool and generate the OBR XML. The tool which is available online uses the latest version of dependencies because of which there is a problem building OBR XML. If we could find a version of the tool which is compatible with our bundle version then it is possible to build the XML using only one command which is (this command is already mentioned in the repositoryGenerator.html webpage link which is shared previously):
            java -jar org.apache.aries.application.tooling.repository.generator-xxx.jar [repository xml location] url1 [url2...]
            Eg: java -jar org.apache.aries.application.tooling.repository.generator-0.1.0-SNAPSHOT.jar c:\temp\test\repo.xml file:///C:/temp/test/jars http://aaa.com/public/Aries/obr/test.b1/test.b1_1.0.0.jar

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - I checked out the tool which is Apache Aries. It can be used to build OBR XML from the bundle jar. I went through the documentation in order to install the tool and generate the OBR XML. The tool which is available online uses the latest version of dependencies because of which there is a problem building OBR XML. If we could find a version of the tool which is compatible with our bundle version then it is possible to build the XML using only one command which is (this command is already mentioned in the repositoryGenerator.html webpage link which is shared previously): java -jar org.apache.aries.application.tooling.repository.generator-xxx.jar [repository xml location] url1 [url2...] Eg: java -jar org.apache.aries.application.tooling.repository.generator-0.1.0-SNAPSHOT.jar c:\temp\test\repo.xml file:///C:/temp/test/jars http://aaa.com/public/Aries/obr/test.b1/test.b1_1.0.0.jar
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            This is the command which I used to build repository.xml
            java -jar org.apache.aries.application.tooling.repository.generator-1.0.1-SNAPSHOT.jar Users\pruthakulkarni\repository.xml Users\pruthakulkarni\IGB\igb_prutha\optionalPlugins\protannot\target\protannot-9.1.0.jar

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - This is the command which I used to build repository.xml java -jar org.apache.aries.application.tooling.repository.generator-1.0.1-SNAPSHOT.jar Users\pruthakulkarni\repository.xml Users\pruthakulkarni\IGB\igb_prutha\optionalPlugins\protannot\target\protannot-9.1.0.jar
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            Using following commands we were able to generate a repository.xml.
            But, most of the details are missing. We are now investigating on how to use the manifest file(which has values to those missing details) to generate the xml using maven obr plugin commands.
            Commands:
            mvn org.apache.felix:maven-obr-plugin:1.2.0:install-file -Dfile=target/protannot-9.1.0.jar -DgroupId=1 -DartifactId=68898 -Dversion=9.1.0 -Dpackaging=jar -DobrRepository=Users\pruthakulkarni\repository.xml
            if we want to skip -DgroupId, -DartifactId, -Dpackaging these parameters then we can simply use
            mvn org.apache.felix:maven-obr-plugin:1.2.0:install-file -DpomFile=pom.xml -Dfile=target/protannot-9.1.0.jar -Dversion=9.1.0 -DobrRepository=Users\pruthakulkarni\repository.xml

            After running these commands, the path where the xml is generated will be shown.

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - Using following commands we were able to generate a repository.xml. But, most of the details are missing. We are now investigating on how to use the manifest file(which has values to those missing details) to generate the xml using maven obr plugin commands. Commands: mvn org.apache.felix:maven-obr-plugin:1.2.0:install-file -Dfile=target/protannot-9.1.0.jar -DgroupId=1 -DartifactId=68898 -Dversion=9.1.0 -Dpackaging=jar -DobrRepository=Users\pruthakulkarni\repository.xml if we want to skip -DgroupId, -DartifactId, -Dpackaging these parameters then we can simply use mvn org.apache.felix:maven-obr-plugin:1.2.0:install-file -DpomFile=pom.xml -Dfile=target/protannot-9.1.0.jar -Dversion=9.1.0 -DobrRepository=Users\pruthakulkarni\repository.xml After running these commands, the path where the xml is generated will be shown.
            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment -

            Prutha KulkarniSameer Shanbhag

            Just to let you know.
            Cytoscape is using below XML file.

            http://chianti.ucsd.edu/cyto_web/plugins/plugins.xml

            search for this link in the Appstore code. It might be helpful in creating rest endpoint.

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - Prutha Kulkarni Sameer Shanbhag Just to let you know. Cytoscape is using below XML file. http://chianti.ucsd.edu/cyto_web/plugins/plugins.xml search for this link in the Appstore code. It might be helpful in creating rest endpoint.
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            To generate index using bnd command:

            • bnd index file.jar

            Creates index (minus description) from bundle jar file.

            Ref:

            Show
            ann.loraine Ann Loraine added a comment - - edited To generate index using bnd command: bnd index file.jar Creates index (minus description) from bundle jar file. Ref: https://bnd.bndtools.org/chapters/400-commands.html
            Hide
            ann.loraine Ann Loraine added a comment -
            Show
            ann.loraine Ann Loraine added a comment - Install bnd on ubuntu: https://www.howtoinstall.co/en/ubuntu/trusty/bnd
            Hide
            sameer Sameer Shanbhag (Inactive) added a comment - - edited

            Dr. [~aloraine]

            By using BND Tools we were able to generate the Repository.xml file using the Bundle Jar, there were a few problems associated with the Repository file generated, few of which are listed below:
            1. The generated XML is using a different schema which is not recognized by the IGB App
            2. We tried to extract the requirements from the file, but the requirements have more packages than mentioned in Manifest or/ required for running the bundle
            3. We also tried the downgrading the version of the bnd tools but anything below version 4.0.0 does not support Indexing using JAR file and hence we should use the convention method to generate the XML using the Manifest File, and the best option we have currently.

            I have pushed the code to the following branch which does the task.

            https://bitbucket.org/sshanbh1/sameer-appstorefork/commits/b0d11f247c0bbadda81148149a1ffd9fa87f90af

            Show
            sameer Sameer Shanbhag (Inactive) added a comment - - edited Dr. [~aloraine] By using BND Tools we were able to generate the Repository.xml file using the Bundle Jar, there were a few problems associated with the Repository file generated, few of which are listed below: 1. The generated XML is using a different schema which is not recognized by the IGB App 2. We tried to extract the requirements from the file, but the requirements have more packages than mentioned in Manifest or/ required for running the bundle 3. We also tried the downgrading the version of the bnd tools but anything below version 4.0.0 does not support Indexing using JAR file and hence we should use the convention method to generate the XML using the Manifest File, and the best option we have currently. I have pushed the code to the following branch which does the task. https://bitbucket.org/sshanbh1/sameer-appstorefork/commits/b0d11f247c0bbadda81148149a1ffd9fa87f90af
            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment -

            IGBF-1794 changes are merged into Sameer's branch for ease of development.
            [~aloraine]

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - IGBF-1794 changes are merged into Sameer's branch for ease of development. [~aloraine]
            Hide
            sameer Sameer Shanbhag (Inactive) added a comment - - edited

            [~aloraine] Sai Charan Reddy Vallapureddy

            The Latest Code is up and running and can be tested using my branch. I am updating the status to Ready for the Pull request.

            https://bitbucket.org/lorainelab/appstore/pull-requests/49/igbf-1760/diff

            Show
            sameer Sameer Shanbhag (Inactive) added a comment - - edited [~aloraine] Sai Charan Reddy Vallapureddy The Latest Code is up and running and can be tested using my branch. I am updating the status to Ready for the Pull request. https://bitbucket.org/lorainelab/appstore/pull-requests/49/igbf-1760/diff
            Hide
            sameer Sameer Shanbhag (Inactive) added a comment -

            [~aloraine]

            https://bitbucket.org/lorainelab/appstore/pull-requests/49/igbf-1760/diff

            Note: I have updated the settings file and updated the pull request. Can be merged if everything is perfect.

            Show
            sameer Sameer Shanbhag (Inactive) added a comment - [~aloraine] https://bitbucket.org/lorainelab/appstore/pull-requests/49/igbf-1760/diff Note: I have updated the settings file and updated the pull request. Can be merged if everything is perfect.
            Hide
            ann.loraine Ann Loraine added a comment -

            Added some comments to pull request.

            Show
            ann.loraine Ann Loraine added a comment - Added some comments to pull request.
            Hide
            ann.loraine Ann Loraine added a comment -

            Sai Charan Reddy Vallapureddy and Sameer Shanbhag please look at comments on PR. Use your best judgement on how to proceed - your choice!

            Show
            ann.loraine Ann Loraine added a comment - Sai Charan Reddy Vallapureddy and Sameer Shanbhag please look at comments on PR. Use your best judgement on how to proceed - your choice!
            Hide
            sameer Sameer Shanbhag (Inactive) added a comment -

            Updated the Repository with the changes mentioned in the Comments. Pushed new changes and the Pull Request is submitted and can be reviewed at :

            https://bitbucket.org/lorainelab/appstore/pull-requests/49/igbf-1760/diff

            Show
            sameer Sameer Shanbhag (Inactive) added a comment - Updated the Repository with the changes mentioned in the Comments. Pushed new changes and the Pull Request is submitted and can be reviewed at : https://bitbucket.org/lorainelab/appstore/pull-requests/49/igbf-1760/diff
            Hide
            ann.loraine Ann Loraine added a comment - - edited
            • Shut down apache server.
            • Copied appstore/settings.py to appstore/settings.py.bak1
            • did "git checkout settings.py" to discard local changes to settings.py (in appstore)
            • Ran "git pull origin master" to update all code

            Sai Charan Reddy Vallapureddy would you be able to update the site?

            My apologies for any errors I may have introduced!

            Show
            ann.loraine Ann Loraine added a comment - - edited Shut down apache server. Copied appstore/settings.py to appstore/settings.py.bak1 did "git checkout settings.py" to discard local changes to settings.py (in appstore) Ran "git pull origin master" to update all code Sai Charan Reddy Vallapureddy would you be able to update the site? My apologies for any errors I may have introduced!
            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment -

            [~aloraine] Sure mam, I will do that.

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - [~aloraine] Sure mam, I will do that.
            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment -

            [~aloraine]

            Old local changes in settings.py have latest configurations. Reverted back to old settings.py.

            Before doing make migrations and migrate follow below steps.
            Since we migrated from sqlite3 to mysql. we need to delete mysql database;
            1. Login to mysql DB instance
            2. USE testappstore;
            3. DROP database testappstore;
            4. CREATE DATABASE testappstore CHARACTER SET UTF8;

            I have updated testappstore. Now the site is up to date with the latest code.
            Sameer Shanbhag Please test your changes.

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - [~aloraine] Old local changes in settings.py have latest configurations. Reverted back to old settings.py. Before doing make migrations and migrate follow below steps. Since we migrated from sqlite3 to mysql. we need to delete mysql database; 1. Login to mysql DB instance 2. USE testappstore; 3. DROP database testappstore; 4. CREATE DATABASE testappstore CHARACTER SET UTF8; I have updated testappstore. Now the site is up to date with the latest code. Sameer Shanbhag Please test your changes.
            Hide
            ann.loraine Ann Loraine added a comment -
            Show
            ann.loraine Ann Loraine added a comment - Sameer Shanbhag or Sai Charan Reddy Vallapureddy : Please add URLs for repository.xml's for released and pending apps in the comments. I tried: https://testappstore.bioviz.org/obr/released/repository.xml https://testappstore.bioviz.org/obr/repository.xml https://testappstore.bioviz.org/obr/pending/repository.xml but got 404 error.
            Hide
            sameer Sameer Shanbhag (Inactive) added a comment -

            [~aloraine]

            Here is the Link for Pending Releases:
            https://testappstore.bioviz.org/obr/pending_releases/repository.xml

            & Approved Releases:
            https://testappstore.bioviz.org/obr/releases/repository.xml

            The Approved Releases will not work as there are no default values set for the apps that are already there in the database.

            For testing:
            You can
            1. Submit an app
            2. Hit the Pending Apps URL

            To Released application URL to work, we will need to change the appsDB.json to populate the field during migration (Which is not required currently) or we can just delete all the apps currently present and repopulate all the applications from the UI (which will be required as I deleted the Repository.xml that we had for testing)

            Show
            sameer Sameer Shanbhag (Inactive) added a comment - [~aloraine] Here is the Link for Pending Releases: https://testappstore.bioviz.org/obr/pending_releases/repository.xml & Approved Releases: https://testappstore.bioviz.org/obr/releases/repository.xml The Approved Releases will not work as there are no default values set for the apps that are already there in the database. For testing: You can 1. Submit an app 2. Hit the Pending Apps URL To Released application URL to work, we will need to change the appsDB.json to populate the field during migration (Which is not required currently) or we can just delete all the apps currently present and repopulate all the applications from the UI (which will be required as I deleted the Repository.xml that we had for testing)
            Hide
            ann.loraine Ann Loraine added a comment -

            The link: https://testappstore.bioviz.org/obr/releases/repository.xml gives a 500 Internal Server error.

            Sai Charan Reddy Vallapureddy is this because you are working on configuring the site?

            Show
            ann.loraine Ann Loraine added a comment - The link: https://testappstore.bioviz.org/obr/releases/repository.xml gives a 500 Internal Server error. Sai Charan Reddy Vallapureddy is this because you are working on configuring the site?
            Hide
            sameer Sameer Shanbhag (Inactive) added a comment - - edited

            [~aloraine]

            As mentioned above we populated the database with the dummy data required for repository generation. I have fixed it now and you can try hitting that URL.

            Installing in the IGB app won't work for now as the URI mentioned in the repository.xml file is not a valid URL.

            Released Apps: https://testappstore.bioviz.org/obr/releases/repository.xml
            Pending Apps: https://testappstore.bioviz.org/obr/pending_releases/repository.xml

            Show
            sameer Sameer Shanbhag (Inactive) added a comment - - edited [~aloraine] As mentioned above we populated the database with the dummy data required for repository generation. I have fixed it now and you can try hitting that URL. Installing in the IGB app won't work for now as the URI mentioned in the repository.xml file is not a valid URL. Released Apps: https://testappstore.bioviz.org/obr/releases/repository.xml Pending Apps: https://testappstore.bioviz.org/obr/pending_releases/repository.xml
            Hide
            ann.loraine Ann Loraine added a comment -

            Looking at:

            A change request:

            • Please store "description" text in the XML as base64-encoded to work properly with IGB.
            Show
            ann.loraine Ann Loraine added a comment - Looking at: https://testappstore.bioviz.org/obr/releases/repository.xml A change request: Please store "description" text in the XML as base64-encoded to work properly with IGB.
            Hide
            ann.loraine Ann Loraine added a comment -

            Should we move this back to "Open" or "In Progress" since there are some changes we need to make, or should we create new tickets for the changes?

            Show
            ann.loraine Ann Loraine added a comment - Should we move this back to "Open" or "In Progress" since there are some changes we need to make, or should we create new tickets for the changes?
            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment -

            [~aloraine]

            It is better to create new tickets for the below problems. These problems are not related to OBR XML generation.

            Now, we have two problems.

            1. appsDB.json is outdated. We need to update it with new fields and values.
            2. description is in normal text format. We need to convert normal text to base64 and put it in repository.xml file.

            Creation of OBR XML is done.

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - [~aloraine] It is better to create new tickets for the below problems. These problems are not related to OBR XML generation. Now, we have two problems. 1. appsDB.json is outdated. We need to update it with new fields and values. 2. description is in normal text format. We need to convert normal text to base64 and put it in repository.xml file. Creation of OBR XML is done.
            Hide
            sameer Sameer Shanbhag (Inactive) added a comment -

            I will create a subtask for the issues mentioned by Sai Charan Reddy Vallapureddy and use this as a parent task.

            Show
            sameer Sameer Shanbhag (Inactive) added a comment - I will create a subtask for the issues mentioned by Sai Charan Reddy Vallapureddy and use this as a parent task.
            Hide
            ann.loraine Ann Loraine added a comment -

            Actually, if you don't have a strong preference either way, please do not make sub-tasks but instead make all new tickets linking to this one.

            Show
            ann.loraine Ann Loraine added a comment - Actually, if you don't have a strong preference either way, please do not make sub-tasks but instead make all new tickets linking to this one.
            Hide
            sameer Sameer Shanbhag (Inactive) added a comment -

            Sure, I'll create two different issues and link it to this one.

            Show
            sameer Sameer Shanbhag (Inactive) added a comment - Sure, I'll create two different issues and link it to this one.

              People

              • Assignee:
                sameer Sameer Shanbhag (Inactive)
                Reporter:
                ann.loraine Ann Loraine
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: