Protecting world leaders against deep fakes using facial, gestural, and vocal mannerisms

Protecting world leaders against deep fakes using facial, gestural, and vocal mannerisms
Advertisements

Edited by William Press, The University of Texas at Austin, Austin, TX; received September 23, 2022; accepted October 3, 2022.

November 23, 2022

119 (48) e2216035119

Abstract

Since their emergence a few years ago, artificial intelligence (AI)-synthesized media—so-called deep fakes—have dramatically increased in quality, sophistication, and ease of generation. Deep fakes have been weaponized for use in nonconsensual pornography, large-scale fraud, and disinformation campaigns. Of particular concern is how deep fakes will be weaponized against world leaders during election cycles or times of armed conflict. We describe an identity-based approach for protecting world leaders from deep-fake imposters. Trained on several hours of authentic video, this approach captures distinct facial, gestural, and vocal mannerisms that we show can distinguish a world leader from an impersonator or deep-fake imposter.

Data, Materials, and Software Availability

Some study data available (The data associated with this manuscripts includes training videos which we will make available. We prefer not to make available the other data in the form of the trained behavioral models because we fear that this could be used by an adversary to evaluate the realism of fake videos. We will, however, upon request make our model available to researchers working in the general space of digital forensics) (18).

Acknowledgments

Author contributions

M.B. and H. F. designed research, performed research, contributed new reagents/analytic tools, analyzed data, and wrote the paper.

Competing interests

The authors declare no competing interest.

References

2

B. Chesney, D. Citron, Deep fakes: A looming challenge for privacy, democracy, and national security. Calif. Law Rev. 107, 1753 (2019).

4

M. Groh, Z. Epstein, C. Firestone, R. Picard, Deepfake detection by human crowds, machines, and machine-informed crowds. Proc. Natl. Acad. Sci. U.S.A. 119, e2110013119 (2022).

5

P. Zhou, X. Han, V. I. Morariu, L. S. Davis, Two-stream neural networks for tampered face detection., International Conference on Computer Vision and Pattern Recognition (2017).

7

Y. Li et al., In ictu oculi: Exposing AI created fake videos by detecting eye blinking., International Workshop on Information Forensics and Security pp. 1–7. (2018).

8

S. Agarwal, H. Farid, O. Fried, M. Agrawala, Detecting deep-fake videos from phoneme-viseme mismatches in CVPR Workshop on Media Forensics. pp. 660–661. (2020).

9

S. Agarwal et al., Protecting world leaders against deep fakes in CVPR Workshop on Media Forensics. 1. (2019).

10

D. Cozzolino, A. Rössler, J. Thies, M. Nießner, L. Verdoliva, ID-reveal: Identity-aware deepfake video detection in International Conference on Computer Vision. pp. 15108–15117. (2021).

11

A. Rössler et al., FaceForensics++: Learning to detect manipulated facial images in International Conference on Computer Vision and Pattern Recognition. (2019).

12

T. Baltrusaitis, A. Zadeh, Y. C. Lim, L. P. Morency, Openface 2.0: Facial behavior analysis toolkit in IEEE International Conference on Automatic Face& Gesture Recognition. pp. 59–66. (2018).

13

R. Gross, I. Matthews, J. Cohn, T. Kanade, S. Baker, Multi-pie in IEEE International Conference on Automatic Face& Gesture Recognition. pp. 1–8. (2008).

14

P. Ekman, W. V. Friesen, Measuring facial movement. Environ. Psychol. Nonverbal Behav. 1, 56–75 (1976).

17

M. Boháček, M. Hrúz, Sign pose-based transformer for word-level sign language recognition in IEEE/CVF Winter Conference on Applications of Computer Vision Workshops. pp. 182–191. (2022).

18

A. Bauer, The Use of Signing Space in a Shared Sign Language of Australia (De Gruyter, ed. 1, 2014).

19

S. Davis, P. Mermelstein, Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Trans. Acoust. Speech Signal Process. 28, 357–366 (1980).

20

F. Pedregosa et al., Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).

Information & Authors

Information

Published in

Proceedings of the National Academy of Sciences

Vol. 119 | No. 48
November 29, 2022

Classifications

Copyright

Data, Materials, and Software Availability

Some study data available (The data associated with this manuscripts includes training videos which we will make available. We prefer not to make available the other data in the form of the trained behavioral models because we fear that this could be used by an adversary to evaluate the realism of fake videos. We will, however, upon request make our model available to researchers working in the general space of digital forensics) (18).

Submission history

Received: September 23, 2022

Accepted: October 3, 2022

Published online: November 23, 2022

Published in issue: November 29, 2022

Keywords

  1. synthetic media
  2. deep fakes
  3. disinformation
  4. digital forensics

Acknowledgments

Author Contributions

M.B. and H. F. designed research, performed research, contributed new reagents/analytic tools, analyzed data, and wrote the paper.

Competing Interests

The authors declare no competing interest.

Notes

Two aspects of the default OpenFace2 configuration are changed to ensure more consistent facial tracking: 1) the Multi-PIE (10) model is always used and 2) the temporal smoothing of action units is disabled.

Authors

Affiliations

Matyáš Boháček1

Gymnasium of Johannes Kepler 169 00 Prague, Czech Republic

Department of Electrical and Computer Sciences, School of Information, University of California, Berkeley, CA 94708

Notes

1M.B. and H.F. contributed equally to this work.

Metrics & Citations

Metrics

Citation statements
Altmetrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

View Options

View options

PDF format

Download this article as a PDF file

DOWNLOAD PDF

Media

Figures

Tables

Other

Read More

Author:

Leave a Reply Cancel reply