Standardize and append a batch of data

Here, we’ll learn

• how to standardize a less well curated collection

• how to append it to the growing versioned collection

import lamindb as ln
import bionty as bt

ln.settings.verbosity = "hint"
bt.settings.auto_save_parents = False
ln.settings.transform.stem_uid = "ManDYgmftZ8C"
ln.settings.transform.version = "1"
ln.track()

💡 connected lamindb: testuser1/test-scrna

💡 Assuming editor is Jupyter Lab.

💡 notebook imports: bionty==0.42.7 lamindb==0.69.9

💡 saved: Transform(uid='ManDYgmftZ8C5zKv', name='Standardize and append a batch of data', key='scrna2', version='1', type='notebook', updated_at=2024-04-10 17:52:44 UTC, created_by_id=1)

💡 saved: Run(uid='DvJwXOir8J9LgGl9PYAt', transform_id=2, created_by_id=1)

💡 tracked pip freeze > /home/runner/.cache/lamindb/run_env_pip_DvJwXOir8J9LgGl9PYAt.txt

Let’s now consider a less-well curated dataset:

adata = ln.core.datasets.anndata_pbmc68k_reduced()
adata

Show code cell output Hide code cell output

AnnData object with n_obs × n_vars = 70 × 765
    obs: 'cell_type', 'n_genes', 'percent_mito', 'louvain'
    var: 'n_counts', 'highly_variable'
    uns: 'louvain', 'louvain_colors', 'neighbors', 'pca'
    obsm: 'X_pca', 'X_umap'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'

We are still working with human data, and can globally set an organism:

bt.settings.organism = "human"

annotate = ln.Annotate.from_anndata(adata, var_index=bt.Gene.symbol, categoricals={"cell_type": bt.CellType.name})

❗ 3 non-validated categories are not saved in Feature.name: ['percent_mito', 'louvain', 'n_genes']!
      → to lookup categories, use lookup().columns
      → to save, run add_new_from_columns

✅ added 5 records from public with Gene.symbol for var_index: ['GPX1', 'SOD2', 'RN7SL1', 'SNORD3B-2', 'IGLL5']

❗ 11 non-validated categories are not saved in Gene.symbol: ['RP11-782C8.1', 'RP11-277L2.3', 'RP11-156E8.1', 'RP3-467N11.1', 'RP11-390E23.6', 'RP11-489E7.4', 'RP11-291B21.2', 'RP11-620J15.3', 'TMBIM4-1', 'AC084018.1', 'CTD-3138B18.5']!
      → to lookup categories, use lookup().var_index
      → to save, run add_new_from_var_index

Standardize & validate genes

Let’s convert Gene symbols to Ensembl ids via standardize(). Note that this is a non-unique mapping and the first match is kept because the keep parameter in .standardize() defaults to "first":

adata.var["ensembl_gene_id"] = bt.Gene.standardize(
    adata.var.index,
    field=bt.Gene.symbol,
    return_field=bt.Gene.ensembl_gene_id,
)
# use ensembl_gene_id as the index
adata.var.index.name = "symbol"
adata.var = adata.var.reset_index().set_index("ensembl_gene_id")

# we only want to save data with validated genes
validated = bt.Gene.validate(adata.var.index, bt.Gene.ensembl_gene_id, mute=True)
adata_validated = adata[:, validated].copy()

💡 standardized 754/765 terms

Here, we’ll use .raw:

adata_validated.raw = adata.raw[:, validated].to_adata()
adata_validated.raw.var.index = adata_validated.var.index

annotate = ln.Annotate.from_anndata(adata_validated, var_index=bt.Gene.ensembl_gene_id, categoricals={"cell_type": bt.CellType.name})

❗ 3 non-validated categories are not saved in Feature.name: ['percent_mito', 'louvain', 'n_genes']!
      → to lookup categories, use lookup().columns
      → to save, run add_new_from_columns

annotate.validate()

✅ var_index is validated against Gene.ensembl_gene_id

💡 mapping cell_type on CellType.name

❗    9 terms are not validated: 'Dendritic cells', 'CD19+ B', 'CD4+/CD45RO+ Memory', 'CD8+ Cytotoxic T', 'CD4+/CD25 T Reg', 'CD14+ Monocytes', 'CD56+ NK', 'CD8+/CD45RA+ Naive Cytotoxic', 'CD34+'
      → save terms via .add_new_from('cell_type')

False

Standardize & validate cell types

Since none of the cell types are validate, let us search the cell type names from the public ontology, and add the name found in the AnnData object as a synonym to the top match found in the public ontology.

bionty = bt.CellType.public()  # access the public ontology through bionty
name_mapper = {}
for name in adata_validated.obs.cell_type.unique():
    # search the public ontology and use the ontology id of the top match
    ontology_id = bionty.search(name).iloc[0].ontology_id
    # create a record by loading the top match from bionty
    record = bt.CellType.from_public(ontology_id=ontology_id)
    name_mapper[name] = record.name  # map the original name to standardized name
    record.save()
    record.add_synonym(name)

Show code cell output Hide code cell output

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0001087'

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000910'

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000919'

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0002057'

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0002101'

We can now standardize cell type names using the search-based mapper:

adata_validated.obs.cell_type = adata_validated.obs.cell_type.map(name_mapper)

Now, all cell types are validated:

annotate.validate()

✅ var_index is validated against Gene.ensembl_gene_id

✅ cell_type is validated against CellType.name

True

Register

artifact = annotate.save_artifact(description="10x reference adata")

💡 path content will be copied to default storage upon `save()` with key `None` ('.lamindb/ODkxta4ERH5EpGXMIetM.h5ad')

✅ storing artifact 'ODkxta4ERH5EpGXMIetM' at '/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna/.lamindb/ODkxta4ERH5EpGXMIetM.h5ad'

💡 parsing feature names of X stored in slot 'var'

✅    754 terms (100.00%) are validated for ensembl_gene_id

✅    linked: FeatureSet(uid='CXggu91TFAXkH3lbOZIe', n=754, type='number', registry='bionty.Gene', hash='j8QkIeLBgJwsscY4vVPx', created_by_id=1)

💡 parsing feature names of slot 'obs'

✅    1 term (25.00%) is validated for name

❗    3 terms (75.00%) are not validated for name: n_genes, percent_mito, louvain

✅    linked: FeatureSet(uid='vNB6W95UYPYDqZsJwbYw', n=1, registry='core.Feature', hash='EIJcRo5vSqJ707xTTMtW', created_by_id=1)

✅ saved 2 feature sets for slots: 'var','obs'

artifact.view_lineage()

Append the shard to the collection

Query the previous collection:

collection_v1 = ln.Collection.filter(
    name="My versioned scRNA-seq collection", version="1"
).one()

Create a new version of the collection by sharding it across the new artifact and the artifact underlying version 1 of the collection:

collection_v2 = ln.Collection(
    [artifact, collection_v1.artifacts[0]],
    is_new_version_of=collection_v1,
)
collection_v2.save()
collection_v2.labels.add_from(artifact)
collection_v2.labels.add_from(collection_v1)

Show code cell output Hide code cell output

✅ loaded: FeatureSet(uid='Ca5PrYB3AOm1piIwUIES', n=4, registry='core.Feature', hash='8gDo0ILt6wrOfQFp6okl', updated_at=2024-04-10 17:52:37 UTC, created_by_id=1)

💡 adding collection [1] as input for run 2, adding parent transform 1

💡 adding artifact [1] as input for run 2, adding parent transform 1

✅ saved 1 feature set for slot: 'var'

💡 transferring cell_type

💡 transferring donor

💡 transferring tissue

💡 transferring cell_type

💡 transferring assay

Version 2 of the collection covers significantly more conditions.

collection_v2.describe()

Collection(uid='pmqiGV7fNOJTWI04vw29', name='My versioned scRNA-seq collection', version='2', hash='HNR3VFV60_yqRnUka11E', visibility=1, updated_at=2024-04-10 17:53:03 UTC)

Provenance:
  📎 transform: Transform(uid='ManDYgmftZ8C5zKv', name='Standardize and append a batch of data', key='scrna2', version='1', type='notebook', updated_at=2024-04-10 17:52:44 UTC, created_by_id=1)
  📎 run: Run(uid='DvJwXOir8J9LgGl9PYAt', started_at=2024-04-10 17:52:44 UTC, is_consecutive=True, transform_id=2, created_by_id=1)
  📎 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at=2024-04-10 17:50:52 UTC)
Features:
  var: FeatureSet(uid='CPlkkc63hVnZSKRWgRe0', n=36508, type='number', registry='bionty.Gene', hash='b5NMddLHEyZqn-vSYvBI', updated_at=2024-04-10 17:53:01 UTC, created_by_id=1)
    'MIR1302-2HG', 'FAM138A', 'OR4F5', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'OR4F29', 'None', 'OR4F16', 'None', 'LINC01409', 'FAM87B', 'LINC01128', 'LINC00115', 'FAM41C', 'None', ...
  obs: FeatureSet(uid='Ca5PrYB3AOm1piIwUIES', n=4, registry='core.Feature', hash='8gDo0ILt6wrOfQFp6okl', updated_at=2024-04-10 17:52:37 UTC, created_by_id=1)
    🔗 donor (12, core.ULabel): 'D496', '621B', 'A29', 'A36', 'A35', '637C', 'A52', 'A37', 'D503', '640C', ...
    🔗 tissue (17, bionty.Tissue): 'blood', 'thoracic lymph node', 'spleen', 'lung', 'mesenteric lymph node', 'lamina propria', 'liver', 'jejunal epithelium', 'omentum', 'bone marrow', ...
    🔗 cell_type (40, bionty.CellType): 'dendritic cell', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated', 'cytotoxic T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'CD14-positive, CD16-negative classical monocyte', 'CD38-positive naive B cell', 'B cell, CD19-positive', 'CD4-positive, alpha-beta T cell', 'classical monocyte', 'T follicular helper cell', ...
    🔗 assay (3, bionty.ExperimentalFactor): '10x 3' v3', '10x 5' v2', '10x 5' v1'
Labels:
  📎 tissues (17, bionty.Tissue): 'blood', 'thoracic lymph node', 'spleen', 'lung', 'mesenteric lymph node', 'lamina propria', 'liver', 'jejunal epithelium', 'omentum', 'bone marrow', ...
  📎 cell_types (40, bionty.CellType): 'dendritic cell', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated', 'cytotoxic T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'CD14-positive, CD16-negative classical monocyte', 'CD38-positive naive B cell', 'B cell, CD19-positive', 'CD4-positive, alpha-beta T cell', 'classical monocyte', 'T follicular helper cell', ...
  📎 experimental_factors (3, bionty.ExperimentalFactor): '10x 3' v3', '10x 5' v2', '10x 5' v1'
  📎 ulabels (12, core.ULabel): 'D496', '621B', 'A29', 'A36', 'A35', '637C', 'A52', 'A37', 'D503', '640C', ...

View data lineage:

collection_v2.view_lineage()