Details
-
Type:
Task
-
Status: Closed (View Workflow)
-
Priority:
Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:
-
Story Points:5
-
Epic Link:
-
Sprint:Summer 2019 Sprint 11, Summer 2019 Sprint 12, Fall 2019 Sprint 1, Fall 2019 Sprint 2, Fall 2019 Sprint 3, Fall 4 : 30 Sep to 11 Oct, Fall 5 : 14 Oct to 25 Oct, Fall 6 : 28 Oct to 8 Nov, Fall 7 : 11 Nov to 22 Nov, Fall 8 : 25 Nov to 6 Dec
Description
There are ten Apps called "developmentPlugins" in a directory of the same name.
These were created many years ago to demonstrate various aspects of the IGB API.
They are useful as demonstrations of what is possible, but are not being used currently. For release in IGB App Store they would need some additional work.
To prepare for that, let's make a new, single repository in bitbucket for all of them. The repo should be configured to trigger building of OBR index file (repository.xml) and jar files for each App, similar to what we did for ProtAnnot and the other so-called "optional" plugins. (See related issue.)
For this new repository:
- Create an aggregate "parent" POM with each development plug-in App as a module.
- Write a maven profile that will allow a user to build all of the development Apps and create a single OBR index file (repository.xml) that includes all of them.
- Also enable a developer to build OBR index files for individual Apps.
Attachments
Issue Links
- relates to
-
IGBF-1773 Create new repos for optionalPlugins
-
- Closed
-
Activity
Feel free to change the specification and plan to match the goals below:
Use case #1:
I would like for a developer to be able to work on just one App at a time without having to build them all just to build a repository.xml for their own App. For the next step in developing these Apps, individual developers will need to be able to fork and clone the repository and work on just one of the Apps – e.g., the "Transcript Variant" App – without needing to build the others. They would do this on their local.
For this to work, the developer would need to be able to build a repository.xml file that would include the App that they are working on so that they could install it into IGB for testing. For that, it seems like the best approach would be to have the default profile in the child POMs build an OBR index file (repository.xml) for just that one App. Again, they would do this on their local.
Use case #2:
In addition, I would like for the bitbucket repository to be able to build all the Apps and also build a single repository XML that includes all of them. This is to support a different use case – the use case of a casual user or developer who is exploring the system and wants to experiment with the Apps in IGB. What would happen is that they would add the Downloads link to their list of App repositories and then – *boom* – all the development plugins will be available to install in IGB.
Note that this means that the Downloads folder can only accommodate one OBR index file - which includes all the Apps. (This is due to IGB limits.)
So ... I think probably the best way to do this is to use a maven profile that can be triggered by the bitbucket pipeline YML file...or...by a developer seeking to replicate same on their local. We could also take advantage of the way the bitbucket pipelines YML files allow you to create different pipelines for individual branches.
There are a lot of moving parts so my notion about how to implement may not work exactly right. If you need to do something different, that is fine ... use your best judgement! The key here is to make it super-easy for developers to contribute and super-easy for users to quickly try out one of the Apps in IGB. Whatever you think will work and will be easy to maintain and extend would be fine. Probably the first thing you create will get revised as you get feedback from testers and users.
Also take a look at the new bitbucket pipeline YML file for the main IGB project. It may show some useful things - including something you suggested not long ago. (Thanks for that.)
We can talk about it person when I get back as well.
Prof. [~aloraine], I have created plugin specific maven profiles as well a parent maven profile.
Whenever a developer commits any change to the repository, normal build process will be triggered.
I have created bitbucket-pipeline file such that it consists of custom configuration for
1) building all plugins:
This uses parent pom maven profile to create a consolidated OBR index file and it transfers it to the downloads section along with all plugin JARs as and when triggered manually.
Manual trigger:
- Go to "Branches" section of the repository.
- click on the three dots "..." present at the end of the branch row.
- Select "Run pipeline for a branch" option.
- Select desired custom pipeline configuration from the dropdown menu and click on "Run" button.
2) Separate custom pipeline configuration for each and every plugin:
This will use maven profile present in the plugin's pom and will build a plugin specific OBR index file. It will also transfer this file along with the plugin JAR to the downloads section of the repository.
Also, I wanted to highlight that currently there are only 9 plugins available under developmentPlugins repository. 10th folder i.e. "TableView" doesn't contain any kind of source code. It only has a Jar file which is without any specific version. I am not sure what exactly it does.
Please let me know if this is as per the requirements or not so that I'll make required changes(if any) and will transfer the ownership of the repository to lorainlab.
You can check my fork for the same and can also trigger the pipeline configuration manually to check if it is working as expected.
https://bitbucket.org/pkulka10/prutha_developmentplugins/src/master/
Hi Prutha,
Thank you for all your work on this - it is a big task!
I realized we actually do not need to transfer the ownership of the repository. Instead, you should keep it in your account and lorainelab can fork it.
I have a some questions and suggestions that might make the build system a little easier for developers to understand and use.
- I think each sub-project (module) ought to specify its IGB platform dependencies separately from the others, since they are all independent from each other. Within each child POM, could you define a custom property that specifies the least version the IGB platform required to compile and also to run the Apps?
- Likewise, the child project POMs don't need to use the same version as the parent project POM. That is, they should not have to strictly match the IGB platform version. Is there a way you can configure the bitbucket pipeline curl command to not assume that the parent and child POMs all are using the same version? Maybe there's a way you can use file globbing in the curl command so that you don't have to know what the version string in each jar is or even the names of the jars that need to be copied?
- Take a look at the tasks in each of the child POMs. Since each POM inherits from the parent POM, you could potentially make the child POMs simpler by moving common plugins configurations into the parent POM.
This may be helpful: https://maven.apache.org/plugins/maven-help-plugin/usage.html
It explains how you can use the maven help pluginto investigate the "effective POM" for child POMs in multi-module projects like this one. There are a lot of great tools you can use.
I have simplified the child POMs by adding common dependencies to the parent POM.
Also, I have removed the version extraction part from the bitbucket pipeline configuration. (I have tried using file globbing functionality to remove version number dependency. But the thing is, the globbing is not supported for --form parameters. It is only supported for other URLs. So, I changed the code in such a way that it uses a grep command to find the jar file irrespective of the version number. It searches for the file depending upon the name of it. Its output is then given to the curl command to post this file to the downloads section.)
I am not so sure about the first suggestion in the previous comment on how can we specify the least version of the IGB platform which is required to compile/run the apps. Prof. [~aloraine], can we please discuss this in person?
Sure. I am working downstairs today. Feel free to stop by any time.
Changes are made for every plugin specifying lowest version of IGB platform required to compile as well as run the apps.
Code changes are available at :
https://bitbucket.org/pkulka10/prutha_developmentplugins/src/master/
I have added the README.md file to the repository and have mentioned the build instruction for the same.
Main repository is present at:
https://bitbucket.org/pkulka10/developmentplugins
The link which was mentioned previously is a link to my development fork.
Prof. [~aloraine], let me know if I should transfer the ownership to lorainlab.
Thanks.
- please also explain how to run/test an App for the two different workflows described.
- please also explain (briefly) how to set up pipeline - e.g., environment variables that need to be added
Question:
- Does OBR index file get build during "install" phase?
Prof. [~aloraine], I have modified the README file and included the steps for pipeline setup as well as for running the plugins.
Also, Yes. The OBR file gets generated during "install" phase of the plugin.
But, the default pipeline configuration is just to build the JARs and not an OBR index file. It is generated on each and every custom pipeline build trigger except default configuration.
Consolidated OBR is generated on "BuildAllPlugins" configuration trigger and an individual OBR file is generated for each plugin when the corresponding plugin's custom configuration is triggered.
Sorry, I meant: Explain this stuff in the README file for developers.
They need to know how to work on these Apps within their IDE and test Apps there, in addition to how to configure their bitbucket repo to copy App jars and OBR index file to the Downloads folder.
Also, please add a new comment with the link to the fork you are using for development. It's confusing because the lorainelab repository is actually just a fork of one of yours. But it looks like maybe you are not longer using that upstream fork and instead are now working on a fork of the lorainelab fork? Please just clarify which fork you are now using and we'll be good to go.
Sorry about that Prof. [~aloraine]
Here is the link to main developmentPlugins repository:
https://bitbucket.org/pkulka10/developmentplugins/src/master/
Also, I would like to mention that I have added each configuration step in the README file so that a person who is completely new to these plugin can also contribute in the development, can test plugins in IGB and can configure bitbucket pipeline setup without taking external help.
Thanks.
Question about Bitbucket pipelines section:
- Since there is already a bitbucket pipelines YML file in the repository, are steps 1 to 3 are necessary. Maybe they should be removed?
Question about: "mvn clean install -P bitbucket-pipeline-profile -DobrRepository=<project_build_directory_path>"
- Is -DobrRepository strictly required?
I ask about this because I believe the default configuration for the maven plugin is to write the OBR index file to $project.build.directory. For building a single module, that would be the modules "target" directory. So I'm not sure if the option -DobrRepository is strictly necessary?
Prof. [~aloraine], That is right. We do not have to specify steps 1-3 since we already have a yml file.
I have removed those steps from the README file now. Also, for a multi-module maven bundles project, we don't strictly need to specify -DobrRepository parameter. We can simply default it to the local maven repository.
I have made the changes for that as well. Could you please have a look at the README and yml file for the same?
Here is the location for main developmentPlugins repo:
https://bitbucket.org/pkulka10/developmentplugins/src/master/
[~aloraine] - As the changes suggested by you are reviewed and was merged into this new repository, I have transferred the ownership to "lorainelab" now.
Loraine Lab already owns this repository: https://bitbucket.org/lorainelab/development-apps/src/master/
Please submit PRs to this repository with your recommended changes.
Prof [~aloraine], I have now submitted a PR to the repository mentioned above.
The code changes are present at :
https://bitbucket.org/lorainelab/development-apps/pull-requests/2/igbf-1907/diff
Could you please merge them into master?
Merged and ready for testing.
The repository for development plugins looks fine. Followed the instructions of README.md to install the plugins but they were not visible to install in App Manager. Moving the ticket to To-do. Please check.
Make sure you start with a fresh system - no previous version of the app installed - delete the bundle cache in ~/.igb to be on the safe side
I was able to fix the bug for all the plugins except SearchModeLucene. For this plugin everything is same and I am not sure why it is happening. Still working on changing the pom for SearchModeLucene so that its repository.xml can be detected in IGB.
Shamika Gajanan Kulkarni, I have made the changes to development plugins pom files so that IGB could detect a plugin whenever we add a new repository related to development plugins. Could you please review it once again? The code changes can be found on branch:
https://bitbucket.org/pkulka10/developmentplugins/src/IGBF-1907/
Also, as discussed with the professor today morning, I have removed the SearchModeLucene development plugin from the development plugin codebase.
Shamika Gajanan Kulkarni, sorry about the previous link. The changes were made on an old fork.
Please refer https://bitbucket.org/pkulka10/development-apps/commits/dbb1310e87209aac5055dd501fc094dbeb5131a7 link for testing the same.
Prutha Kulkarni I was able to view all the apps from the downloads link which contains all jars.
One small request. Can you please provide me the write access to your repository so that I can check the different bitbucket configurations for individual apps too?
Thank you.
Shamika Gajanan Kulkarni, I have given you the write access to repo now. Could you please check it once?
Prutha Kulkarni Reviewed the ticket along with custom build for each development plugin, default and build all plugins configurations. Working as expected. Kindly submit a pull request for the same.
Prof. [~aloraine], do we really have to create a maven profile specifically to build consolidated OBR index file?
As the consolidated index file is already generated in the default local maven repository, we just have to specify one command line parameter while building the developmentPlugins project to get the consolidated OBR index file to the build directory which is "DobrRepository".
I have made the changes to my fork of developmentPlugins repo. Could you please have a look at it?
Link to my fork is:
https://bitbucket.org/pkulka10/prutha_developmentplugins/src/master/
Currently it is building separate OBR index files along with JARs in the target folder of the corresponding plugin. I have copied these JAR files along with a consolidated OBR index file to the download section of the repo with the help of pipeline configuration.
Thanks.