[IGBF-3660] Include schema of UCSC data in the properties field of the dataset. - JIRA UNCC

Details

Type: Task
Status: Closed (View Workflow)
Priority: Blocker
Resolution: Done
Affects Version/s: None
Fix Version/s: 10.1.0
Labels:
None

Story Points:
5
Epic Link:
Improve IGB for users
Sprint:
Spring 6, Spring 7, Spring 8, Spring 9, Spring 10

Description

Pass the schema of the UCSC data in the properties field while constructing the dataset. This enables developers to extract the schema/metadata of the data easily. This metadata can be used to provide rich set of Filter by/Color by options to IGB users. Structure the metadata in the same way as actual data, for example, [attribute 1, attribute 2, ..., props].

Attachments

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Attachments

Screenshot_1.png
211 kB
27/Mar/24 3:34 PM
Screenshot_2.png
64 kB
27/Mar/24 3:36 PM
Screenshot_3.png
118 kB
27/Mar/24 3:36 PM
Screenshot_4.png
72 kB
27/Mar/24 3:36 PM
Screenshot 2024-04-30 at 12.26.15 AM.png
197 kB
03/May/24 11:53 AM

Issue Links

is blocked by

IGBF-3640 Expand color by and Filter by to include the props

Closed

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Jaya Sravani Sirigineedi added a comment - 19/Mar/24 5:10 PM - edited

As per discussion with Kaushik Gopu, will add the asked columns in the featureProps variable in DataSet objects created while getting the available datasets using the schema response from the schema API, example API: https://api.genome.ucsc.edu/list/schema?genome=galGal6;track=nestedRepeats. Below is an example showing how the featureProps look like:

Example schema response:

Unable to find source-code formatter for language: json. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml

// https://api.genome.ucsc.edu/list/schema?genome=galGal6;track=nestedRepeats
{
  "downloadTime": "2024:03:19T17:07:25Z",
  "downloadTimeStamp": 1710868045,
  "genome": "galGal6",
  "track": "nestedRepeats",
  "dataTime": "2018-10-11T14:52:43",
  "dataTimeStamp": 1539294763,
  "columnTypes": [
    {
      "name": "bin",
      "sqlType": "smallint(6)",
      "jsonType": "number",
      "description": "Indexing field to speed chromosome range queries"
    },
    {
      "name": "chrom",
      "sqlType": "varchar(255)",
      "jsonType": "string",
      "description": "Chromosome (or contig, scaffold, etc.)"
    },
    {
      "name": "chromStart",
      "sqlType": "int(10) unsigned",
      "jsonType": "number",
      "description": "Start position in chromosome"
    },
    {
      "name": "chromEnd",
      "sqlType": "int(10) unsigned",
      "jsonType": "number",
      "description": "End position in chromosome"
    },
    {
      "name": "name",
      "sqlType": "varchar(255)",
      "jsonType": "string",
      "description": "Name of item"
    },
    {
      "name": "score",
      "sqlType": "int(10) unsigned",
      "jsonType": "number",
      "description": "Average of fragment identity scores, transformed into 0..1000 range for shading"
    },
    {
      "name": "strand",
      "sqlType": "char(1)",
      "jsonType": "string",
      "description": "+, -, or . for mixed (some fragments +, some -)"
    },
    {
      "name": "thickStart",
      "sqlType": "int(10) unsigned",
      "jsonType": "number",
      "description": "for BED compatibility -- same as chromStart"
    },
    {
      "name": "thickEnd",
      "sqlType": "int(10) unsigned",
      "jsonType": "number",
      "description": "for BED compatibility -- same as chromEnd"
    },
    {
      "name": "reserved",
      "sqlType": "int(10) unsigned",
      "jsonType": "number",
      "description": "for BED compatibility"
    },
    {
      "name": "blockCount",
      "sqlType": "int(11)",
      "jsonType": "number",
      "description": "Number of blocks"
    },
    {
      "name": "blockSizes",
      "sqlType": "longblob",
      "jsonType": "string",
      "description": "Comma separated list of block (fragment) sizes"
    },
    {
      "name": "chromStarts",
      "sqlType": "longblob",
      "jsonType": "string",
      "description": "Start positions relative to chromStart"
    },
    {
      "name": "blockStrands",
      "sqlType": "longblob",
      "jsonType": "string",
      "description": "Strand of each fragment"
    },
    {
      "name": "id",
      "sqlType": "int(10) unsigned",
      "jsonType": "number",
      "description": "RepeatMasker-assigned ID used to join fragments"
    },
    {
      "name": "repClass",
      "sqlType": "varchar(255)",
      "jsonType": "string",
      "description": "Class of repeat"
    },
    {
      "name": "repFamily",
      "sqlType": "varchar(255)",
      "jsonType": "string",
      "description": "Family of repeat"
    }
  ],
  "shortLabel": "Interrupted Rpts",
  "type": "bed 12 +",
  "longLabel": "Fragments of Interrupted Repeats Joined by RepeatMasker ID",
  "itemCount": 29762,
  "visibility": "hide",
  "group": "varRep",
  "exonNumbers": "off",
  "useScore": "1"
}

featureProps for the above schema:

Unable to find source-code formatter for language: json. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml

{ 
  "genome": "galGal6",
  "track": "nestedRepeats",
  "props": {
       "id": "number",
       "repClass": "string",
       "repFamily": "string"
  }
}

Show

Jaya Sravani Sirigineedi added a comment - 19/Mar/24 5:10 PM - edited As per discussion with Kaushik Gopu , will add the asked columns in the featureProps variable in DataSet objects created while getting the available datasets using the schema response from the schema API, example API: https://api.genome.ucsc.edu/list/schema?genome=galGal6;track=nestedRepeats . Below is an example showing how the featureProps look like: Example schema response: Unable to find source-code formatter for language: json. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml // https://api.genome.ucsc.edu/list/schema?genome=galGal6;track=nestedRepeats { "downloadTime" : "2024:03:19T17:07:25Z" , "downloadTimeStamp" : 1710868045, "genome" : "galGal6" , "track" : "nestedRepeats" , "dataTime" : "2018-10-11T14:52:43" , "dataTimeStamp" : 1539294763, "columnTypes" : [ { "name" : "bin" , "sqlType" : "smallint(6)" , "jsonType" : "number" , "description" : "Indexing field to speed chromosome range queries" }, { "name" : "chrom" , "sqlType" : "varchar(255)" , "jsonType" : "string" , "description" : "Chromosome (or contig, scaffold, etc.)" }, { "name" : "chromStart" , "sqlType" : " int (10) unsigned" , "jsonType" : "number" , "description" : "Start position in chromosome" }, { "name" : "chromEnd" , "sqlType" : " int (10) unsigned" , "jsonType" : "number" , "description" : "End position in chromosome" }, { "name" : "name" , "sqlType" : "varchar(255)" , "jsonType" : "string" , "description" : "Name of item" }, { "name" : "score" , "sqlType" : " int (10) unsigned" , "jsonType" : "number" , "description" : "Average of fragment identity scores, transformed into 0..1000 range for shading" }, { "name" : "strand" , "sqlType" : " char (1)" , "jsonType" : "string" , "description" : "+, -, or . for mixed (some fragments +, some -)" }, { "name" : "thickStart" , "sqlType" : " int (10) unsigned" , "jsonType" : "number" , "description" : " for BED compatibility -- same as chromStart" }, { "name" : "thickEnd" , "sqlType" : " int (10) unsigned" , "jsonType" : "number" , "description" : " for BED compatibility -- same as chromEnd" }, { "name" : "reserved" , "sqlType" : " int (10) unsigned" , "jsonType" : "number" , "description" : " for BED compatibility" }, { "name" : "blockCount" , "sqlType" : " int (11)" , "jsonType" : "number" , "description" : " Number of blocks" }, { "name" : "blockSizes" , "sqlType" : "longblob" , "jsonType" : "string" , "description" : "Comma separated list of block (fragment) sizes" }, { "name" : "chromStarts" , "sqlType" : "longblob" , "jsonType" : "string" , "description" : "Start positions relative to chromStart" }, { "name" : "blockStrands" , "sqlType" : "longblob" , "jsonType" : "string" , "description" : "Strand of each fragment" }, { "name" : "id" , "sqlType" : " int (10) unsigned" , "jsonType" : "number" , "description" : "RepeatMasker-assigned ID used to join fragments" }, { "name" : "repClass" , "sqlType" : "varchar(255)" , "jsonType" : "string" , "description" : " Class of repeat" }, { "name" : "repFamily" , "sqlType" : "varchar(255)" , "jsonType" : "string" , "description" : "Family of repeat" } ], "shortLabel" : "Interrupted Rpts" , "type" : "bed 12 +" , "longLabel" : "Fragments of Interrupted Repeats Joined by RepeatMasker ID" , "itemCount" : 29762, "visibility" : "hide" , "group" : "varRep" , "exonNumbers" : "off" , "useScore" : "1" } featureProps for the above schema: Unable to find source-code formatter for language: json. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml { "genome" : "galGal6" , "track" : "nestedRepeats" , "props" : { "id" : "number" , "repClass" : "string" , "repFamily" : "string" } }

21 older comments

Hide

Permalink

Nowlan Freese added a comment - 08/May/24 1:39 PM

Built Sravani's branch locally on my Mac.

Schema is now fetched only when the user selects a specific data set from the UCSC REST in the Available Data panel. Schema is passed to props and appears correctly when using Color By or Filter By.

Ready for pull request.

Show

Nowlan Freese added a comment - 08/May/24 1:39 PM Built Sravani's branch locally on my Mac. Schema is now fetched only when the user selects a specific data set from the UCSC REST in the Available Data panel. Schema is passed to props and appears correctly when using Color By or Filter By. Ready for pull request.

Hide

Permalink

Jaya Sravani Sirigineedi added a comment - 10/May/24 10:35 AM

Created a single Pull request for both tickets, https://jira.bioviz.org/browse/IGBF-3660 and https://jira.bioviz.org/browse/IGBF-3640. Nowlan Freese already reviewed it, Kaushik Gopu Please review it and let me know if find any issues.

Show

Jaya Sravani Sirigineedi added a comment - 10/May/24 10:35 AM Created a single Pull request for both tickets, https://jira.bioviz.org/browse/IGBF-3660 and https://jira.bioviz.org/browse/IGBF-3640 . Nowlan Freese already reviewed it, Kaushik Gopu Please review it and let me know if find any issues.

Hide

Permalink

Kaushik Gopu added a comment - 10/May/24 12:10 PM

Suggested surrounding code with try-catch block to avoid unexpected failures while retrieving props data. Other than that, everything looks great to me.

Show

Kaushik Gopu added a comment - 10/May/24 12:10 PM Suggested surrounding code with try-catch block to avoid unexpected failures while retrieving props data. Other than that, everything looks great to me.

Hide

Permalink

Ann Loraine added a comment - 13/May/24 10:55 AM - edited

PR is merged and new installers are built and deployed to bioviz.org "early access" section.

Show

Ann Loraine added a comment - 13/May/24 10:55 AM - edited PR is merged and new installers are built and deployed to bioviz.org "early access" section.

Hide

Permalink

Nowlan Freese added a comment - 15/May/24 11:58 AM

Testing main branch installer on Mac.

Schema is now fetched only when the user selects a specific data set from UCSC REST. The schema props are being passed to Color by... and Filter... (see ~~IGBF-3640~~ testing).

Closing ticket.

Show

Nowlan Freese added a comment - 15/May/24 11:58 AM Testing main branch installer on Mac. Schema is now fetched only when the user selects a specific data set from UCSC REST. The schema props are being passed to Color by... and Filter... (see IGBF-3640 testing). Closing ticket.

Include schema of UCSC data in the properties field of the dataset.

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates