基于人工智能技术的面相照片标志点自动定位的评价

doi:10.13591/j.cnki.kqyx.2023.12.001

Abstract

Abstract:

Objective To develop an automatic system to simplify the progress based on convolutional neural networks and build its accuracy evaluation system. Methods A total of 467 lateral and frontal views (age range: 6-55 years) were collected. Forty-five landmarks were detected in the front view and so as 31 in the profile. An automatic locating system based on CNN was developed, consisting of a global detection module and a local correction module. An accuracy evaluation system based on aesthetic considerations was proposed, which consisted of the standardized average error (NME) of the detected points, success rate (SDR) of landmark locating within the unit distance and the successful classification rate (SCR). Results The NME of our test set was 0.079±0.221 in profile and 0.025±0.021 in frontage.The SDR of 0.02, 0.04, 0.06, 0.08 and 0.10 units were respectively 54.17%, 85.71%, 93.94%, 96.69%, and 97.37% in profile, 58.54%、87.59%、95.64%、98.03%、99.00% in frontage. Most of their SCR of our test set were 100%. Conclusion In this study, we successfully proposed an automatic landmark locating system based on CNN. The system can detect 76 landmarks with high detection accuracy within 20 seconds. Moreover, we constructed an evaluation metric of the automatic landmark locating system which focused on the facial aesthetics. Both the location and evaluation system can highly simplify the photogrammetric analysis.

Key words: photogrammetric analysis, convolutional neural networks, facial aesthetics

CLC Number:

R783.5

QIU Tao,HE Tao,ZHANG Qiang,XIAO Yuxuan,GUO Weihua. Automated photogrammetric analysis from a 2D photograph by convolutional neural network[J]. Stomatology, 2023, 43(12): 1057-1064.

Figures/Tables 8

Fig.1

Tab.1

Tab.2

Landmark definitions of profile subset"

标志点	缩写	定义中线标志点
发际点	Tr	在前额中部的发际线上
软组织额点	G'	软组织额部之最前点
软组织鼻根点	N'	软组织侧面上相对的鼻根点
鼻顶(尖)点	Prn	鼻部软组织之最突点
鼻小柱点	Cm	小柱顶部之间的中点,与相应鼻孔的顶部水平
鼻下(底)点	Sn	鼻小柱与上唇之连接点,代表上唇基底部的位置/鼻中隔下缘与上唇交界处的中点
上唇凹点	A'	鼻下点与上唇突点弧线连线之最凹点
上唇突点	UI	上唇之最突点
上唇下点/上口点	Stms	上唇(下)红缘之最(低)下点
下唇上点/下口点	Stmi	下唇(上)红缘之最高(上)点。在正常面形者中,此点常与上口点Stms/Stoms重合在一起
下唇突点	LL	下唇之最突点/下红线的中点
下唇凹点/颏唇沟点	B'	颏唇沟之最凹点。代表下唇之基底部/下唇与颏部之间的最凹点
软组织颏前点	Pog'	颏部软组织的最前点。代表软组织颏部的位置
软组织颏顶点	Gn'	Sn-Pog'与C-Me'延长线的交点/软组织颏前点与软组织颏下点之曲线中点/蝶鞍点、颏顶点间连线(S—N)延长线与颏部软组织外形轮廓之交点
软组织颏下点	Me'	软组织颏部最低点
颏颈角点	C	软组织颏部与颈部的交界点
对称分布标志点
外眦点	Ex	眼裂的外连合
眶上缘点	Os	眉毛下缘的最高点
眶下缘点	Or	眼眶下缘的最低点
颊点	Chk	在Camper平面与连接外眼角和唇连合的连线的交叉处
鼻翼点	Al	鼻翼等高线上的最侧点
鼻翼外侧点	Ac	鼻翼弯曲根部的最外侧点
鼻孔轴下点	Itn	鼻孔轴下点
鼻孔轴上点	Stn	鼻孔轴线上点
唇峰点	Cph	在人中的每个隆起边缘上恰好在朱红线上方
口裂点	Ch	唇连合
耳屏上缘点	T	在耳屏的上缘
耳廓最高点	Sa	耳廓游离缘的最高点
耳垂最后点	Pa	耳垂游离缘的最后点
耳垂最下点	Sba	耳垂游离缘的最低点
下颌角	Go	下颌角最外侧点

Tab.2

Fig.2

Fig.3

Fig.4

Fig.5

Tab.3

Definitions of angles and proportions and SCR in the evaluation and test set"

检测指标上面-面高比	定义 $N' - S t m s N' - M e'$ '×100	验证集SCR81.08	测试SCR 100.00
下面-面高比	$S n - M e' N' - M e'$ ×100	89.19	100.00
颏-面高比	$B' - M e' N' - M e'$ ×100	100.00	100.00
下颌-面高比	$S t m i - M e' N' - M e'$ ×100	89.19	100.00
下颌-上面高比	$S t m i - M e' N' - S t m s$ ×100	89.19	100.00
下颌-下面高比	$S t m i - M e' S n - M e'$ ×100	81.08	100.00
上唇-上面高比	$S n - S t m s N' - S t m s$ ×100	81.08	100.00
上唇-下颌高比	$S n - S t m s S t m i - M e'$ ×100	81.08	100.00
上唇-鼻高比	$S n - S t m s N' - S n$ ×100	89.19	100.00
下唇-面高比	$S t m i - B' S n - M e'$ ×100	70.27	100.00
下唇-下颌高比	$S t m i - B' S t m i - M e'$ ×100	75.68	100.00
下唇-颏高比	$S t m i - B' B' - M e'$ ×100	94.59	100.00
颏颈角	∠C'-Me'/G'-Pog'	94.59	100.00
面突角	∠G'-Sn-Pog'	89.19	100.00
全面突角	∠G'-Prn-Pog'	100.00	100.00
正中第三面角	∠N'-T-Sn	94.59	100.00
下第三面角	∠Sn-T-Me'	81.08	100.00
鼻唇角	∠C'-Sn-Ul	97.30	97.30
鼻面角	∠G'-Pog'/N'-Prn	91.89	81.30
鼻颏角	∠N'-Prn-Pog'	100.00	100.00
颏唇沟角	∠Ll-B'-Pog'	81.08	16.00
面高-面宽比	$N' - M e' Z y l - Z y r$ ×100	100.00	100.00
下颌-面宽比	$G o l - G o r Z y l - Z y r$ ×100	100.00	100.00
下颌宽-面高比	$G o l - G o r N' - M e'$ ×100	97.30	92.00
下颌高-下颌宽比	$S t m i - M e' G o l - G o r$ ×100	97.30	92.00
上唇高-唇宽比	$S n - S t m s C h l - C h r$ ×100	91.89	88.00
口宽-轮廓比	$C h l - C h r C h l - S t m s + C h r - S t m s$ ×100	97.59	98.67

Tab.3

References 32

[1]	Peck H, Peck S. A concept of facial esthetics[J]. Angle Orthod, 1970, 40(4):284-318. doi: 10.1043/0003-3219(1970)040<0284:ACOFE>2.0.CO;2 pmid: 4917993
[2]	Bashour M. History and current concepts in the analysis of facial attractiveness[J]. Plast Reconstr Surg, 2006, 118(3):741-756. doi: 10.1097/01.prs.0000233051.61512.65 pmid: 16932186
[3]	Malkoç S, Demir A, Uysal T, et al. Angular photogrammetric analysis of the soft tissue facial profile of Turkish adults[J]. Eur J Orthod, 2009, 31(2):174-179. doi: 10.1093/ejo/cjn082
[4]	He ZJ, Jian XC, Wu XS, et al. Anthropometric measurement and analysis of the external nasal soft tissue in 119 young Han Chinese adults[J]. J Craniofac Surg, 2009, 20(5):1347-1351. doi: 10.1097/SCS.0b013e3181ae41cf
[5]	Chu G, Zhao JM, Han MQ, et al. Three-dimensional prediction of nose morphology in Chinese young adults: A pilot study combining cone-beam computed tomography and 3dMD photogrammetry system[J]. Int J Legal Med, 2020, 134(5):1803-1816. doi: 10.1007/s00414-020-02351-8 pmid: 32647961
[6]	Bishara SE, Cummins DM, Jorgensen GJ, et al. A computer assisted photogrammetric analysis of soft tissue changes after orthodontic treatment. Part I: Methodology and reliability[J]. Am J Orthod Dentofacial Orthop, 1995, 107(6):633-639. doi: 10.1016/S0889-5406(95)70107-9
[7]	Moshkelgosha V, Fathinejad S, Pakizeh Z, et al. Photographic facial soft tissue analysis by means of linear and angular measurements in an adolescent Persian population[J]. Open Dent J, 2015, 9: 346-356. doi: 10.2174/1874210601509010346 pmid: 26464606
[8]	Cummins DM, Bishara SE, Jakobsen JR. A computer assisted photogrammetric analysis of soft tissue changes after orthodontic treatment. Part Ⅱ: Results[J]. Am J Orthod Dentofacial Orthop, 1995, 108(1):38-47. doi: 10.1016/S0889-5406(95)70064-1
[9]	Arık SÖ, Ibragimov B, Xing L. Fully automated quantitative cephalometry using convolutional neural networks[J]. J Med Imaging, 2017, 4(1):014501. doi: 10.1117/1.JMI.4.1.014501
[10]	Wang SM, Li HQ, Li JZ, et al. Automatic analysis of lateral cephalograms based on multiresolution decision tree regression voting[J]. J Healthc Eng, 2018, 2018: 1797502.
[11]	Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks[J]. Commun ACM, 2017, 60(6):84-90. doi: 10.1145/3065386
[12]	Aubreville M, Knipfer C, Oetter N, et al. Automatic classification of cancerous tissue in laseren domicroscopy images of the oral cavity using deep learning[J]. Sci Rep, 2017, 7(1):11979. doi: 10.1038/s41598-017-12320-8 pmid: 28931888
[13]	Yu MX, Yan H, Xia JB, et al. Deep convolutional neural networks for tongue squamous cell carcinoma classification using Raman spectroscopy[J]. Photodiagnosis Photodyn Ther, 2019, 26: 430-435. doi: S1572-1000(19)30162-0 pmid: 31082525
[14]	Zhang R, Huang L, Xia W, et al. Multiple supervised residual network for osteosarcoma segmentation in CT images[J]. ComputMed Imaging Graph, 2018, 63: 1-8.
[15]	Tuzoff DV, Tuzova LN, Bornstein MM, et al. Tooth detection and numbering in panoramic radiographs using convolutional neural networks[J]. Dentomaxillofac Radiol, 2019, 48(4):20180051. doi: 10.1259/dmfr.20180051
[16]	Hwang HW, Park JH, Moon JH, et al. Automated identification of cephalometric landmarks: Part 2-Might it be better than human?[J]. Angle Orthod, 2020, 90(1):69-76. doi: 10.2319/022019-129.1
[17]	Douglas TS. Image processing for craniofacial landmark identification and measurement: A review of photogrammetry and cephalometry[J]. Comput Med Imaging Graph, 2004, 28(7):401-409. doi: 10.1016/j.compmedimag.2004.06.002
[18]	YUE W, BIN Y. Application of machine learning in orthodontics[J]. Stomatology, 2022, 42(1):29-35.
[19]	Yao J, Zeng W, He T, et al. Automatic localization of cephalometric landmarks based on convolutional neural network[J]. Am J Orthod Dentofacial Orthop, 2022, 161(3):e250-e259.
[20]	Rhee SC, Dhong ES, Yoon ES. Photogrammetric facial analysis of attractive Korean entertainers[J]. Aesthetic Plast Surg, 2009, 33(2):167-174. doi: 10.1007/s00266-008-9257-0 pmid: 19034560
[21]	Wen YF, Wong HM, McGrath CP. A longitudinal study of facial growth of Southern Chinese in Hong Kong: Comprehensive photogrammetric analyses[J]. PLoS One, 2017, 12(10):e0186598. doi: 10.1371/journal.pone.0186598
[22]	El-Fegh I, Galhood M, Sid-Ahmed M, et al. Automated 2-D cephalometric analysis of X-ray by image registration approach based on least square approximator[J]. Annu Int Conf IEEE Eng Med Biol Soc, 2008, 2008: 3949-3952. doi: 10.1109/IEMBS.2008.4650074 pmid: 19163577
[23]	Vanden Homberg MJC, Gevaert CM, Georgiadou Y. The changing face of accountability in humanitarianism: Using artificial intelligence for anticipatory action[J]. PaG, 2020, 8(4):456-467. doi: 10.17645/pag.v8i4.3158
[24]	Gragnaniello D, Marra F, Verdoliva L. Detection of AI-generated synthetic faces[M]// Handbook of Digital Face Manipulation and Detection. Cham: Springer, 2022: 191-212.
[25]	Patcas R, Bornstein MM, Schätzle MA, et al. Artificial intelligence in medico-dental diagnostics of the face: A narrative review of opportunities and challenges[J]. Clin Oral Investig, 2022, 26(12):6871-6879. doi: 10.1007/s00784-022-04724-2
[26]	Li YS, Dang B, LiWC, et al. GLH-water: A large-scale dataset for global surface water detection in large-size very-high-resolution satellite imagery[EB/OL]. 2023: 2303.09310. https://arxiv.org/abs/2303.09310
[27]	Ahmed N, Bin Mahbub R, Rahman RM. Learning to extract buildings from ultra-high-resolution drone images and noisy labels[J]. Int J Remote Sens, 2020, 41(21):8216-8237. doi: 10.1080/01431161.2020.1763496
[28]	Degollada E, Amigó N, O’Callaghan S, et al. A novel technique for photo-identification of the fin whale, Balaenoptera physalus, as determined by drone aerial images[J]. Drones, 2023, 7(3):220. doi: 10.3390/drones7030220
[29]	Yao ZX, Liu QM, Yang JK, et al. PPUP-GAN: A GAN-based privacy-protecting method for aerial photography[J]. Future Gener Comput Syst, 2023, 145: 284-292. doi: 10.1016/j.future.2023.03.031
[30]	Tian CW, Xu Y, Zuo WM, et al. Coarse-to-fine CNN for image super-resolution[J]. IEEE Trans Multimed, 2021, 23: 1489-1502. doi: 10.1109/TMM.2020.2999182
[31]	Deng SQ, Xiong YJ, Wang M, et al. Harnessing unrecognizable faces for improving face recognition[C]// 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). J Waikoloa, HI, USA. IEEE, 2023: 3413-3422.
[32]	Kong WY, You ZS, Lv XB. 3D face recognition algorithm based on deep Laplacian pyramid under the normalization of epidemic control[J]. Comput Commun, 2023, 199: 30-41. doi: 10.1016/j.comcom.2022.12.011

Automated photogrammetric analysis from a 2D photograph by convolutional neural network

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 32

Related Articles 1

Recommended Articles

Metrics

Comments