Tag: Metadata

Retrieve genome data by BioProject using the Datasets command-line tool

You can now retrieve genome data using the NCBI Datasets command-line tool and API by simply providing a BioProject accession. You can go directly from a BioProject accession to genome data even when the BioProject accession is the parent of multiple BioProjects (Figure 1).

Figure 1. Command-lines using BioProject accessions with the datasets command-line tool and sample metadata. Top panel: command-line for downloading genome metadata for the Sanger 25 Genomes Project (PRJEB33226). Middle panel: a portion of the metadata in JSON format for the 25 Genomes Project. Bottom panel: command-line for downloading sequence data and annotation metadata for a component BioProject for the king scallop (PRJEB35331). Continue reading “Retrieve genome data by BioProject using the Datasets command-line tool” →

Find SRA datasets in the cloud using BigQuery Taxonomy Analysis tables!

Now that the Sequence Read Archive (SRA) is publicly available on the cloud, you can harness the power of high-performance cloud computing to analyze all the data you wish without having to download a single byte. To help you programmatically find datasets of interest to you, we’ve loaded BigQuery with the SRA Metadata Table, which contains the descriptive information supplied at the time of sequence submission. Searches of the SRA Metadata Table are dependent on the quality and consistency of the metadata as submitted which means it can sometimes be a challenge to identify a complete and relevant set of suitable sequences. However, the Taxonomy Analysis Table can be a useful tool to overcome this challenge. Here’s why.

NCBI indexes SRA runs with one or more taxonomy terms when species-specific sequence k-mers are matched in the submitted sequences. The Taxonomy Analysis Table (tax_analysis) thus becomes a catalog of all taxonomic IDs detected in every run, based on the specificity and accuracy characteristics of these unique hashes sampled from reference genomes. We have now added the Taxonomy Analysis Table to BigQuery so you can filter hundreds of thousands of runs by this calculated taxonomic content to gather target datasets. Use this in conjunction with the BigQuery Taxonomy Table (which connects scientific names to taxonomic IDs) and link back to the BigQuery Metadata Table.

Explore/link to these four new tables in BigQuery:

tax_analysis_info: a summary table for the results of the STAT tool
tax_analysis: use the taxonomy analysis table to locate any number of runs based on kmer hits to a particular organism or branch in a taxonomic tree.
taxonomy: NCBI Taxonomy database where you can locate the taxid based on organism names.
kmer: contains kmers mapped to a particular organism and allows you to continue exploring organismal content further. You can leverage kmer tables in your downstream analysis by building custom kmer libraries.

Big_Query_example
Figure 1. SRA runs found using the taxonomy tables and BigQuery for taxid:694002, Betacoronavirus.

Check out our helpful summary information for additional information on taxonomic analysis.

We are actively working on new tools and ways to help you use the cloud to access and compute on SRA data. We are piloting this new feature in BigQuery, and plan to add this information to Amazon Cloud’s (AWS) Athena soon.

If you need help getting started, refer to our  tutorials  and  how-to video playlist  on YouTube!

SRA cloud sequences hold the promise of additional discoveries related to COVID-19

We recently announced that we made all of the Sequence Read Archive (SRA) publicly available on two cloud platforms. This archive of genetic sequences is a treasure trove of information and the cloud environments provide high-performance computing capabilities via a GCP or AWS account – right from your own device. High-throughput sequencing has made generating data extremely fast and inexpensive, which has fueled the rapid growth of SRA. Putting it on the cloud makes it possible to analyze “the high-throughput, unassembled sequence data, across all such sequences”.

So, what are some of the potential discoveries that await? To investigate some of the possibilities, we have held a series of codeathons to see if known and unknown viruses could be found lurking within SRA cloud datasets. Spoiler alert – they are! And just recently, a team from Stanford reported that they were able to identify a 2019-nCoV-like Coronavirus in pangolins by examining data sets identified via a meta-metagenomic search of SRA and downloaded using the SRA Toolkit. One challenge this team faced was downloading the datasets: 2.5TB corresponding to approximately 10¹³ bases took over 48 hours to gather. How might cloud-based SRA tools have made this task easier/faster? Here’s how:

BigQuery: allows native cloud programmatic access to and search based on SRA metadata in the cloud. SRA Toolkit enables retrieval and reading of sequencing files from the SRA datasets in the cloud and writing files into the same format, respectively.
Coming soon to the cloud are tools for large scale BLAST processing for a Read Alignment and Annotation Pipeline Tool (RAPT). These tools allow the data to be analyzed directly in the cloud, eliminating the need for download to local storage for analysis.
Also in the works is a mechanism to provide better access to taxonomic content of SRA runs as calculated by NCBI tools.

We are continually adding new functionality to better support your cloud workflows and are happy to help! Contact us at [email protected] if you have questions or need help getting started. If you need assistance setting up GCP or AWS, please follow the steps in our how-to videos on YouTube.

Pangenome metadata scopeathon in San Diego, January 16-17

We will help run a scopeathon (January 16-17, 2020). This event focuses on planning and designing software to extract value from organismal and genera-level graph genomes by dynamically labeling with metadata. We’re seeking people who are interested in describing community level genomes as graphs, or solving problems involving complex phenotypic interactions with specific genomes. If this describes you, please apply! We also encourage people who will be in San Diego for the International Plant & Animal Genome XXVIII conference to apply. The event is open to anyone selected and willing to travel to San Diego. We will work with data from the following organism groups:

Viruses
Microbes (Bacteria and Archaea)
Human
Plants (corn/wheat, others)
Animals

Continue reading “Pangenome metadata scopeathon in San Diego, January 16-17” →

Suggest a project for the next NIH hackathon!

The NCBI will host a collaborative biodata science hackathon on the NIH Campus in Bethesda, Maryland February 20-22.

We are now collecting project proposals focusing on building tools and pipelines for advanced analysis of biomedical datasets including text, images, next generation sequencing data, proteomics, and metadata. Proposals for tutorial pipelines and educational tools for advanced analysis are also welcome.

Submit your project proposal here! Submissions are due January 7, 2019.

Continue reading “Suggest a project for the next NIH hackathon!” →

Applications for July 2018 post-ISMB Chicago Bioinformatics Hackathon at Northwestern University Feinberg School of Medicine are open

From July 11-13, 2018, NCBI will help with a data science hackathon at the Northwestern Feinberg School of Medicine campus in downtown Chicago. This hackathon focuses on genomics and general data science analyses including text, image, and sequence processing. The event is for researchers, including students and postdocs, who already use large datasets or develop pipelines for analyses from high-throughput experiments. Some projects are available to other non-scientific developers, mathematicians or librarians. The hackathon is open to anyone selected for the hackathon and willing to travel to Chicago.

Continue reading “Applications for July 2018 post-ISMB Chicago Bioinformatics Hackathon at Northwestern University Feinberg School of Medicine are open” →

NCBI to help with Rocky Mountain Genomics HackCon at University of Colorado BioFrontiers Institute June 18 – 23, 2018

The NCBI will assist with a plenary session and a hackathon at the BioFrontiers Institute in Boulder Colorado June 18-23rd.

The hackathon will focus on tools for advanced analysis of biomedical datasets including text, images, next generation sequencing data, proteomics, and metadata. Many individuals who attend these events have already engaged in the use of large datasets or in the development of informatics tools, code, or pipelines; however, researchers who are in the earlier stages of their data science journey, including students and postdocs are also encouraged to apply. Some projects are available to other non-scientific developers, mathematicians, or librarians.

The event is open to anyone selected for the hackathon and willing to travel to Boulder, Colorado.

Before the hackathon, there will be a one-day Plenary Session, featuring technical speakers in precision medicine, metagenomics and advanced RNA-Seq analysis, as well as an exhibitor and poster session. Please see https://hackathon.colorado.edu for more details.

There will also be an advanced nascent RNA-Seq workshop, as well as an advanced pipelining workshop, run solely by the University of Colorado, although NCBI personnel may be present. Please see https://hackathon.colorado.edu for more details.

Continue reading “NCBI to help with Rocky Mountain Genomics HackCon at University of Colorado BioFrontiers Institute June 18 – 23, 2018” →

Introducing the new Virus Sequence Search Interface

BLAST is a powerful search tool, but often a search is just the beginning of the journey. We put ourselves in the shoes of a researcher who has just sequenced a handful of samples from the latest viral outbreak and tried to understand what information would be most useful. We also reached out to researchers in the field and asked: a) what questions do they really want to answer? and b) how can NCBI best provide the answers? Based on insights from those questions and answers, we developed the new Virus Sequence Search Interface (Fig. 1). The Search Interface is an NCBI Labs project, which means it is an experimental project, and we may modify the resource based on your feedback and experiences.

Figure 1. The Virus Sequence Selection Interface. — **Figure 1.** The Virus Sequence Selection Interface. The Virus Sequence Selection Interface accepts as input nucleotide and protein accessions, as well as FASTA and plain-text formatted sequences. The user selects either “Nucleotide” or “Protein,” depending on the sequence type, and selects the virus type from the pull-down menu below the text entry field.

Continue reading “Introducing the new Virus Sequence Search Interface” →

November 15 NCBI Minute: Advanced Use of NCBI APIs for High-Throughput Analysis of Genomic Data and Metadata

On Wednesday, November 15, 2017, at 12:00 PM EST, NCBI will present a webinar on advanced applications of the NCBI APIs we previously introduced in our general API webinar in September. This webinar is intended for bioinformaticians, computational biologists and others who are already comfortable with writing scripts to access, download and analyze data.

Continue reading “November 15 NCBI Minute: Advanced Use of NCBI APIs for High-Throughput Analysis of Genomic Data and Metadata” →