Subject level clustering using a negative binomial model for small transcriptomic studies

Qian Li, Janelle R. Noel-MacDonnell, Devin C. Koestler, Ellen L Goode, Brooke L. Fridley

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Background: Unsupervised clustering represents one of the most widely applied methods in analysis of high-throughput 'omics data. A variety of unsupervised model-based or parametric clustering methods and non-parametric clustering methods have been proposed for RNA-seq count data, most of which perform well for large samples, e.g. N ≥ 500. A common issue when analyzing limited samples of RNA-seq count data is that the data follows an over-dispersed distribution, and thus a Negative Binomial likelihood model is often used. Thus, we have developed a Negative Binomial model-based (NBMB) clustering approach for application to RNA-seq studies. Results: We have developed a Negative Binomial Model-Based (NBMB) method to cluster samples using a stochastic version of the expectation-maximization algorithm. A simulation study involving various scenarios was completed to compare the performance of NBMB to Gaussian model-based or Gaussian mixture modeling (GMM). NBMB was also applied for the clustering of two RNA-seq studies; type 2 diabetes study (N = 96) and TCGA study of ovarian cancer (N = 295). Simulation results showed that NBMB outperforms GMM applied with different transformations in majority of scenarios with limited sample size. Additionally, we found that NBMB outperformed GMM for small clusters distance regardless of sample size. Increasing total number of genes with fixed proportion of differentially expressed genes does not change the outperformance of NBMB, but improves the overall performance of GMM. Analysis of type 2 diabetes and ovarian cancer tumor data with NBMB found good agreement with the reported disease subtypes and the gene expression patterns. This method is available in an R package on CRAN named NB.MClust. Conclusion: Use of Negative Binomial model based clustering is advisable when clustering over dispersed RNA-seq count data.

Original languageEnglish (US)
Article number474
JournalBMC Bioinformatics
Volume19
Issue number1
DOIs
StatePublished - Dec 12 2018

Fingerprint

Negative Binomial Model
Statistical Models
Cluster Analysis
Clustering
Model-based
Mixture Modeling
RNA
Gaussian Mixture
Count Data
Ovarian Cancer
Model-based Clustering
Diabetes
Medical problems
Clustering Methods
Sample Size
Ovarian Neoplasms
Type 2 Diabetes Mellitus
Genes
Gene
Unsupervised Clustering

Keywords

  • Clustering
  • EM algorithm
  • Gaussian mixture model
  • Model-based
  • Negative binomial
  • RNA-seq

ASJC Scopus subject areas

  • Structural Biology
  • Biochemistry
  • Molecular Biology
  • Computer Science Applications
  • Applied Mathematics

Cite this

Subject level clustering using a negative binomial model for small transcriptomic studies. / Li, Qian; Noel-MacDonnell, Janelle R.; Koestler, Devin C.; Goode, Ellen L; Fridley, Brooke L.

In: BMC Bioinformatics, Vol. 19, No. 1, 474, 12.12.2018.

Research output: Contribution to journalArticle

Li, Qian ; Noel-MacDonnell, Janelle R. ; Koestler, Devin C. ; Goode, Ellen L ; Fridley, Brooke L. / Subject level clustering using a negative binomial model for small transcriptomic studies. In: BMC Bioinformatics. 2018 ; Vol. 19, No. 1.
@article{4d0b8effc2a648b7a0c95cc79ee5995f,
title = "Subject level clustering using a negative binomial model for small transcriptomic studies",
abstract = "Background: Unsupervised clustering represents one of the most widely applied methods in analysis of high-throughput 'omics data. A variety of unsupervised model-based or parametric clustering methods and non-parametric clustering methods have been proposed for RNA-seq count data, most of which perform well for large samples, e.g. N ≥ 500. A common issue when analyzing limited samples of RNA-seq count data is that the data follows an over-dispersed distribution, and thus a Negative Binomial likelihood model is often used. Thus, we have developed a Negative Binomial model-based (NBMB) clustering approach for application to RNA-seq studies. Results: We have developed a Negative Binomial Model-Based (NBMB) method to cluster samples using a stochastic version of the expectation-maximization algorithm. A simulation study involving various scenarios was completed to compare the performance of NBMB to Gaussian model-based or Gaussian mixture modeling (GMM). NBMB was also applied for the clustering of two RNA-seq studies; type 2 diabetes study (N = 96) and TCGA study of ovarian cancer (N = 295). Simulation results showed that NBMB outperforms GMM applied with different transformations in majority of scenarios with limited sample size. Additionally, we found that NBMB outperformed GMM for small clusters distance regardless of sample size. Increasing total number of genes with fixed proportion of differentially expressed genes does not change the outperformance of NBMB, but improves the overall performance of GMM. Analysis of type 2 diabetes and ovarian cancer tumor data with NBMB found good agreement with the reported disease subtypes and the gene expression patterns. This method is available in an R package on CRAN named NB.MClust. Conclusion: Use of Negative Binomial model based clustering is advisable when clustering over dispersed RNA-seq count data.",
keywords = "Clustering, EM algorithm, Gaussian mixture model, Model-based, Negative binomial, RNA-seq",
author = "Qian Li and Noel-MacDonnell, {Janelle R.} and Koestler, {Devin C.} and Goode, {Ellen L} and Fridley, {Brooke L.}",
year = "2018",
month = "12",
day = "12",
doi = "10.1186/s12859-018-2556-9",
language = "English (US)",
volume = "19",
journal = "BMC Bioinformatics",
issn = "1471-2105",
publisher = "BioMed Central",
number = "1",

}

TY - JOUR

T1 - Subject level clustering using a negative binomial model for small transcriptomic studies

AU - Li, Qian

AU - Noel-MacDonnell, Janelle R.

AU - Koestler, Devin C.

AU - Goode, Ellen L

AU - Fridley, Brooke L.

PY - 2018/12/12

Y1 - 2018/12/12

N2 - Background: Unsupervised clustering represents one of the most widely applied methods in analysis of high-throughput 'omics data. A variety of unsupervised model-based or parametric clustering methods and non-parametric clustering methods have been proposed for RNA-seq count data, most of which perform well for large samples, e.g. N ≥ 500. A common issue when analyzing limited samples of RNA-seq count data is that the data follows an over-dispersed distribution, and thus a Negative Binomial likelihood model is often used. Thus, we have developed a Negative Binomial model-based (NBMB) clustering approach for application to RNA-seq studies. Results: We have developed a Negative Binomial Model-Based (NBMB) method to cluster samples using a stochastic version of the expectation-maximization algorithm. A simulation study involving various scenarios was completed to compare the performance of NBMB to Gaussian model-based or Gaussian mixture modeling (GMM). NBMB was also applied for the clustering of two RNA-seq studies; type 2 diabetes study (N = 96) and TCGA study of ovarian cancer (N = 295). Simulation results showed that NBMB outperforms GMM applied with different transformations in majority of scenarios with limited sample size. Additionally, we found that NBMB outperformed GMM for small clusters distance regardless of sample size. Increasing total number of genes with fixed proportion of differentially expressed genes does not change the outperformance of NBMB, but improves the overall performance of GMM. Analysis of type 2 diabetes and ovarian cancer tumor data with NBMB found good agreement with the reported disease subtypes and the gene expression patterns. This method is available in an R package on CRAN named NB.MClust. Conclusion: Use of Negative Binomial model based clustering is advisable when clustering over dispersed RNA-seq count data.

AB - Background: Unsupervised clustering represents one of the most widely applied methods in analysis of high-throughput 'omics data. A variety of unsupervised model-based or parametric clustering methods and non-parametric clustering methods have been proposed for RNA-seq count data, most of which perform well for large samples, e.g. N ≥ 500. A common issue when analyzing limited samples of RNA-seq count data is that the data follows an over-dispersed distribution, and thus a Negative Binomial likelihood model is often used. Thus, we have developed a Negative Binomial model-based (NBMB) clustering approach for application to RNA-seq studies. Results: We have developed a Negative Binomial Model-Based (NBMB) method to cluster samples using a stochastic version of the expectation-maximization algorithm. A simulation study involving various scenarios was completed to compare the performance of NBMB to Gaussian model-based or Gaussian mixture modeling (GMM). NBMB was also applied for the clustering of two RNA-seq studies; type 2 diabetes study (N = 96) and TCGA study of ovarian cancer (N = 295). Simulation results showed that NBMB outperforms GMM applied with different transformations in majority of scenarios with limited sample size. Additionally, we found that NBMB outperformed GMM for small clusters distance regardless of sample size. Increasing total number of genes with fixed proportion of differentially expressed genes does not change the outperformance of NBMB, but improves the overall performance of GMM. Analysis of type 2 diabetes and ovarian cancer tumor data with NBMB found good agreement with the reported disease subtypes and the gene expression patterns. This method is available in an R package on CRAN named NB.MClust. Conclusion: Use of Negative Binomial model based clustering is advisable when clustering over dispersed RNA-seq count data.

KW - Clustering

KW - EM algorithm

KW - Gaussian mixture model

KW - Model-based

KW - Negative binomial

KW - RNA-seq

UR - http://www.scopus.com/inward/record.url?scp=85058477189&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85058477189&partnerID=8YFLogxK

U2 - 10.1186/s12859-018-2556-9

DO - 10.1186/s12859-018-2556-9

M3 - Article

C2 - 30541426

AN - SCOPUS:85058477189

VL - 19

JO - BMC Bioinformatics

JF - BMC Bioinformatics

SN - 1471-2105

IS - 1

M1 - 474

ER -