DIR Logo
Model nameCell type prediction model
Model architecture
  • Mask2Former
Training dataset
  • TCGA with pathologist’s (J.V.) annotation [1] (117,358 patches from 469 WSIs)
Model predictionsTumor cells, Lymphocytes, Eosinophils, Neutrophils, Plasma cells
Model nameTumor compartment segmentation model
Model architecture
  • SegFormer
Training dataset
  • TCGA with pathologist’s (J.V.) annotation [1] (99,871 patches from 469 WSIs)
  • Lizard dataset [2] (8,686 patches from 190 images)
Model predictionsTumor epithelium compartment, Tumor stroma compartment
Model nameTumor region identification model
Model architecture
  • CONCH + Logistic Regression
Training dataset
  • HunCRC [3] (402,904 patches from 200 WSIs)
Model predictionsTumor region, non-tumor region
Model nameMSI prediction model
Model architecture
  • UNI2-h for feature extraction, followed by Logistic Regression on extracted features
Training dataset
  • SurGen CRC cohort [4] (851 slides total; 94 dMMR slides, 757 pMMR slides)
  • CPTAC CRC cohort [5] (221 slides total; 53 dMMR slides and 168 pMMR slides)
  • TCGA CRC cohort [6] (557 slides total; 76 dMMR slides and 481 pMMR slides)
  • ORION CRC cohort [7] (37 slides total; 7 dMMR slides and 30 pMMR slides)
Model predictionsSlide-level binary classification of MSI status (pMMR / dMMR)

References

  1. Väyrynen, J. P. et al. Prognostic significance of immune cell populations identified by machine learning in colorectal cancer using routine hematoxylin and Eosin–Stained sections. Clinical Cancer Research 26, 4326–4338 (2020). https://doi.org/10.1158/1078-0432.CCR-20-0071
  2. Graham, S. et al. Lizard: a large-scale dataset for colonic nuclear instance segmentation and classification. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 684-693) (2021).
  3. Pataki, B. Á. et al. HunCRC: annotated pathological slides to enhance deep learning applications in colorectal cancer screening. Scientific Data 9, 370 (2022). https://doi.org/10.1038/s41597-022-01450-y
  4. Myles, C., Um, I. H., Marshall, C., Harris-Birtill, D., & Harrison, D. J. (2025). SurGen: 1020 H&E-stained whole-slide images with survival and genetic markers. GigaScience, 14, Article giaf086. https://doi.org/10.1093/gigascience/giaf086
  5. Vasaikar, S., Huang, C., Wang, X., Petyuk, V. A., Savage, S. R., Wen, B., Dou, Y., Zhang, Y., Shi, Z., Arshad, O. A., Gritsenko, M. A., Zimmerman, L. J., McDermott, J. E., Clauss, T. R., Moore, R. J., Zhao, R., Monroe, M. E., Wang, Y.-T., Chambers, M. C., … Zhang, B. (2019). Proteogenomic analysis of human colon cancer reveals new therapeutic opportunities. Cell, 177(4), 1035–1049.e19. https://doi.org/10.1016/j.cell.2019.03.030
  6. The Cancer Genome Atlas Network. (2012). Comprehensive molecular characterization of human colon and rectal cancer. Nature, 487, 330–337. https://doi.org/10.1038/nature11252
  7. Lin, J.-R., Chen, Y.-A., Campton, D., Cooper, J., Coy, S., Yapp, C., Tefft, J. B., McCarty, E., Ligon, K. L., Rodig, S. J., Reese, S., George, T., Santagata, S., & Sorger, P. K. (2023). High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers. Nature Cancer, 4(7), 1036–1052. https://doi.org/10.1038/s43018-023-00576-1

Immunoscore quantifies a patient's immune response by measuring the density of CD3+ and CD8+ T-cell populations within the core tumor (CT) and invasive margin (IM) region [1]. Immunoscore ranges from 0 to 4, which is the sum of points awarded individually for each of the 4 parameters (CD3+ in CT, CD3+ in IM, CD8+ in CT, CD8+ in IM) that exceeds a predefined density threshold. High immunoscore indicates robust immune response and a significantly lower risk of recurrence.


Traditionally scored by pathologists using immunohistochemistry stainings, we adapted the method described by [2], termed as image feature model 1 (IFM1), for an automated scoring approach using multiplexed-protein images. Cytokeratin (CK) is used as the tumor marker to delineate CT and IM from whole-slide images. In this demo, we applied the adapted IFM1 scoring on CRC04 specimen of Orion dataset [2].


References

  1. Galon, J. et al. Towards the introduction of the ‘Immunoscore’ in the classification of malignant tumours. The Journal of Pathology 232, 199–209 (2013).
  2. Lin, J.-R. et al. High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers. Nature Cancer 4, 1036–1052 (2023).
Model nameCell type segmentation model
Model architecture
  • CellViT-SAMH [1]
Training dataset
  • Paired same-section H&E and COMET-stained mIF (4 in-house WSIs from JY lab)
Model predictionsEpithelial cells (CK), Endothelial cells (CD34), Stromal cells (SMA), Immune cells (CD45)
Model nameTissue category segmentation model
Model architecture
  • CONCH [2]
Training dataset
  • H&E images with unsupervised clustering and pathologist’s (J.V.) annotation (4 in-house WSIs from JY lab)
Model predictionsTumor tissue region, Normal tissue region (non-neoplastic liver parenchyma)

References

  1. Hörst, F. et al. CellViT: Vision Transformers for precise cell segmentation and classification. Medical Image Analysis 94, 103143 (2024). https://doi.org/10.1016/j.media.2024.103143
  2. Lu, M.Y., Chen, B., Williamson, D.F.K. et al. A visual-language foundation model for computational pathology. Nat Med 30, 863–874 (2024). https://doi.org/10.1038/s41591-024-02856-4
Model nameEstrogen receptor-positive (ER+) tumour cell prediction model
Model architecture
  • ResNet152
Training dataset
  • Paired same-section H&E and mIF (7 in-house WSIs from JY lab (93,436 ER+ and 160,223 ER- image patches); mIF images stained with Leica Bond Max autostainer and imaged with Zeiss Axioscan 7 slide scanner)
Model predictionsER-positive tumour cells (ER+,CK/EPCAM+)

Models developed and provided by JY Lab, Institute of Molecular and Cell Biology (IMCB).

Model nameCK+ Tumour Cell Prediction Model
Model architecture
  • Pix2Pix generative adversarial network (GAN) [1]
Training dataset
  • Paired same-section H&E and multiplex immunohistochemistry (pan-CK) from lung carcinoma tissue sections [3] (24 WSIs total; 20 WSIs used for balanced 80:20 train/validation split, 4 WSIs held out for test)
Model predictionsCK+ cells, CK- cells
Model nameCK+ PD-L1+ Tumour Cell Prediction Model
Model architecture
  • Vision Transformer - Base, 16x16 patch size (ViT-B/16) [2]
Training dataset
  • Paired same-section H&E and multiplex immunohistochemistry (PD-L1) from lung carcinoma tissue sections [3] (24 WSIs total; 20 WSIs used for balanced 80:20 train/validation split, 4 WSIs held out for test; PD-L1 model was trained on 112 x 112-pixel single-cell patches from CK-positive cells)
Model predictionsCK+PD-L1- cells, CK+PD-L1+ cells

References

  1. P. Isola, J. Y. Zhu, T. Zhou and A. A. Efros, Image-to-Image Translation with Conditional Adversarial Networks, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 5967-5976, doi: 10.1109/CVPR.2017.632.
  2. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N., An image is worth 16x16 words: Transformers for image recognition at scale. (2021). International Conference on Learning Representations (ICLR). doi: 10.48550/arXiv.2010.11929
  3. Fincham RE, Joseph C, Wee F, Li R, Ye J, Nie L, et al. 1089 H&E 2.0: Deep Learning Prediction of Lung Cancer Biomarkers Pan-Cytokeratin and PD-L1 using a Dual Model Framework. Journal for ImmunoTherapy of Cancer. 2025;13:. https://doi.org/10.1136/jitc-2025-SITC2025.1089

GigaTIME generates 21 virtual multiplex marker overlays from an H&E whole-slide image. For inference, the slide is processed at 0.25 mpp, and the final stitched overlays are resized to align with the original uploaded slide for visualization.

Model nameGigaTIME (Whole-slide multi-marker virtual staining model)
Model architecture
  • Patch-based encoder-decoder architecture built on NestedUNet (UNet++) [1]
Training dataset
  • Proprietary paired H&E and multiplex immunofluorescence (mIF) training dataset with 21 protein channels; ~40 million cells from 21 lung adenocarcinoma patients [1]
Model predictionsDAPI, PD-1, CD14, CD4, T-bet, CD34, CD68, CD16, CD11c, CD138, CD20, CD3, CD8, PD-L1, CK, Ki67, Tryptase, Actin, Caspase3, PHH3, Transgelin

References

  1. Valanarasu, J. M. J., Xu, H., Usuyama, N., Kim, C., Wong, C., Argaw, P., Ben Shimol, R., Crabtree, A., Matlock, K., Bartlett, A. Q., Bagga, J., Gu, Y., Zhang, S., Naumann, T., Fox, B. A., Wright, B., Robicsek, A., Piening, B., Bifulco, C., Wang, S., … Poon, H. (2026). Multimodal AI generates virtual population for tumor microenvironment modeling. Cell, 189(2), 386–400.e19. https://doi.org/10.1016/j.cell.2025.11.016
v1.0.0