Training a Segmentation-based Visual Anonymization Service for Street Scenes

Training a Segmentation-based Visual Anonymization Service for Street Scenes Martin Korb[0009−0000−2551−9763] and Werner Bailer[0000−0003−2442−4900] JOANNEUM RESEARCH – DIGITAL, Graz, Austria {firstname.lastname}@joanneum.at Abstract. In order to share image and video datasets for training and validation of machine learning, personal identifiable information (PII) of natural persons needs to be removed. Street scenes are widely used in computer vision research, and anonymization of street scenes typically addresses making faces/heads and license plates unidentifiable. Many existing anonymization approaches are based on bounding boxes of detections, which is problematic as more content than necessary is concealed. This in particular and issue for fisheye cameras, which are quite widely used in vehicle-based capture. We provide a dataset with additional annotations on existing image datasets in order to train a segmentation model for heads and license plates. The training pipeline includes data augmentation in order to handle images resulting from undistorted fisheye images. We train Mask2Former models on the dataset, report the performance of these baseline models, and provide a service implementation and web demo for testing these models. Keywords: anonymization, privacy, segmentation, dataset 1 Introduction Sharing image and video datasets for training and validation of machine learning methods does not only foster advances in the research community, but is also increasingly important to comply with transparency requirements, such as demanded by the EU AI Act [2]. However, in order to comply with data protection requirements (such as the EU General Data Protection Regulation [1]), personal identifiable information (PII) of natural persons needs to be removed before image and video data can be shared with other parties (unless a specific data processing agreement would be put in place). The removal of PII, referred to as anonymization, is thus of increasing importance. A range of anonymization methods is available (see for example overview in [13], ranging from blacking or blurring to advanced methods for faces, that insert generated faces preserving some attributes (e.g. [11]). A taxonomy of different types of information as well as a comprehensive review methods are provided in [15]. However, independent of the method to be applied, robust detection of the regions containing PII is required. Street scenes are widely used in computer vision research, including applications such as automated driving or the creation of digital twins of cityscapes. 2 Korb and Bailer Anonymization of street scenes typically addresses making faces or heads and license plates unidentifiable. Many existing anonymization approaches are based on bounding box detections, which is problematic as more content than necessary is concealed (e.g. blurred). This in particular and issue for fisheye cameras, which are quite widely used in vehicle-based capture. Due to the distortion, objects towards the image boundaries take large areas and are no longer aligned with the image axes, so that bounding boxes tend to cover large background regions. The contributions of this paper are: –We provide a dataset with annotations on top of existing street scene images for training and validation segmentation of human heads and license plates. –We describe a process for training a transformed-based segmenter, while iteratively extending the dataset with semi-automatically generated annotations. –We implement an anonymization service using the obtained segmentations, evaluate it across datasets and show that the resulting anonymization has no negative influence on common vision tasks. The rest of this paper is organized as follows. In Section 2 we give a brief overview of related works and instance segmentation. In Section 3 we describe the process to how we created our dataset for anonymization tasks. In Section 4 we describe our web anonymization service and Section 5 concludes the paper. 2 Related Work Although anonymization is a common task, only a limited number of methods and datasets are available. Many datasets for face detection exist (see e.g. Table 3 in [21]), however, most only have bounding box annotations. Also some datasets for license plate detection exist [16, 18], along with detection methods [14]. A practical issue is that face and license plate annotations are available on quite different types of image data, thus jointly training on them is challenging. It has also been studied whether anoymization affects subsequent image analysis tasks such as segmentation. However, also these works focus on bounding box annotations [20, 8]. A number of recent methods for (instance) segmentation are proposed in literature. BEiT [3] adopts the successful approach of BERT (Bidirectional Encoder Representations from Transformers) from natural language processing, using 16 ×16 image patches as visual tokens. The model is pretrained by masking patches and then fine-tuned to specific downstream tasks such as semantic segmentation and instance segmentation. InternImage [17] is a vision foundation model, which is based on a CNN architecture. Swin transformer [12] is a backbone for vision tasks, adding the concept of shifted windows in the attention block, implemented as an hierarchical architecture. The backbone serves a number of downstream tasks, including semantic and instance segmentation. Segmentation-based Visual Anonymization Service 3 OneFormer [9] is a transformer-based model aiming to unify semantic segmentation, instance segmentation and panoptic segmentation with a shared backbone. Mask2Former [4] addresses a similar aim of addressing different segmentation tasks with one architecture, which in this case introduces a transformer decoder with masked attention. Recently, approaches based on vision-language foundation models have been proposed, e.g. SAM [10]. However, for a task with few known classes of interested, these models are computationally very expensive compared to other methods. 3 Training and Dataset Creation Process Our approach is based on the Mask2Former [5] instance segmentation approach. We leverage the object part annotations of the ADE20K dataset [19], in particular the head part annotations of persons, and the license plate part annotations of cars, motorbikes and trucks, as the initial training data for the segmentation. To be more precise, 1672 images of the ADE20K dataset were used for training data and another 419 for validation. However, it has to be noted that the annotations in ADE20K are not complete, i.e., not all instances in an image may be annotated. We thus leverage an iterative process for expanding the dataset. 1. Manually correct annotations of ADE20K parts, also discriminating face views and back views of heads. CVAT1is used to import the annotations and perform the manual correction. 2. Modify a Mask2Former model pretrained on ADE20K to human head, number plate and background and train it further on the part annotations for these classes. For training the PyTorch multi-step learning rate scheduler2 was used, starting with a learning rate of 10−4and 0.1 as the multiplicative factor of learning rate decay. A maximum of 65 epochs was trained, as it was found that further training brought no relevant improvement on the validation set. We opted for ‘AdamW’3optimizer with a batch size of 6. 3. Use the model to label 426 images of the Cityscapes [6] dataset and manually correct the results in CVAT. As described in point 2, we trained a pretrained model further on our initial dataset. To do so, we use the object detection and segmentation framework mmdetection4. After we had completed the above points we created a dataset for object detection in COCO JSON format5and trained again. 170 annotated images of the Street category of the Places365 [19] dataset and another 216 annotated images from Cityscapes [6] were used to create a test 1https://github.com/cvat-ai/cvat 2https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler. MultiStepLR.html 3https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html#adamw 4https://github.com/open-mmlab/mmdetection 5https://cocodataset.org/#format-data 4 Korb and Bailer Table 1: Test results on Places365 and Cityscapes. IoU area mAP 0.50:0.05:0.95 all 0.319 0.50 all 0.589 0.75 all 0.327 0.50:0.95 small 0.276 0.50:0.95 medium 0.485 0.50:0.95 large 0.543 dataset. Note that the 216 Cityscapes images used for testing are from different cities than those used in training, in order to keep the subsets as independent as possible. To test the model, the COCO evaluation tools6are used to calculate average precisions (AP), average recalls (AR) and mean average precisions (mAP). The resulting model performs on the mentioned test dataset with the results shown in Table 1. However, on a dataset such as the vehicle-centric digital twin dataset (Vilanova 500) [7], where images are taken with fisheye lenses, and the undistorted images have scaled up objects near the image borders, the model performs less well. We have thus implemented a data augmentation approach that simulates the effects of undistortion from fisheye captured images, and apply it to all our training images generating some kind of “tunnel” effect onto them. The resulting transformed dataset is then added to the training dataset. The key idea behind this data augmentation was to use a coordinate-based transformation and spread this effect outward from the center to obtain a tunnel effect. In short, we did the following steps to achieve this: 1. take a coordinate pair (x, y) and normalize them relative to a center point (xc, ycto obtain (xn, yn), 2. calculate the polar coordinates (r, θ) of the normalized ones (xn, yn), 3. apply a non-linear transformation to the radius rto create a ”tunnel” effect, 4. convert the obtained polar coordinates (r′, θ) back to Cartesian coordinates. After some experimentation we concluded that the non-linear transformation r′= 1 + (s·(1 −r1/4)) simulated this tunnel effect in the best way, where the parameter sdescribes the strength in which the coordinates are spread outward. We used for our dataset the value s= 1.8. To ensure that the polygons of the segmentation masks are correctly transformed to their corresponding images, the inverse transformation was used on their coordinates. The resulting model, obtained after training with this data augmentation, provides the results reported in Table 2 on Vilanova 500. While still lower than on images without distortion (in particular for small objects), the results are significantly better than without augmentation. We have also verified that anonymization on this dataset does not degrade the performance for person and vehicle classes, when applying the segmentation model from [7]. 6https://cocodataset.org/#detection-eval Segmentation-based Visual Anonymization Service 5 Table 2: Test results on Vilanova 500 (undistorted fisheye images) with augmentation. IoU area mAP 0.50:0.05:0.95 all 0.165 0.50 all 0.365 0.75 all 0.103 0.50:0.95 small 0.133 0.50:0.95 medium 0.410 0.50:0.95 large 0.500 We release AnonCityDT, a dataset for anonymization for cityscape digital twins. We provide our updated and additional annotations for the ADE20K and Cityscapes datasets, the annotation on the Vilanove 500 dataset, as well as the code for data augmentation. The resources can be found at https://github. com/didymosxr/AnonCityDT. 4 Anonymization Service and Demo Application We provide a web demo which uses the model described in Section 3, implemented with the help of the open source Python library gradio7. We chose gradio because it is a fast and uncomplicated way for us to integrate our machine learning model into a web interface. One of the reasons why we chose instance segmentation was that we get segmentation masks when we apply the model to an input. Figure 1 provides an architecture diagram of the demo application and its backend services. An image or video is given by the user as input, which is preprocessed for inference by the neural network. After getting the results, the logic of the visualizer, in this case a ‘blurrer’, is applied to blur the detected masks onto the given data. The blurred data is then given back to the web application where, in both cases, one can download it via a link. In the case of an image, one can see immediately the blurred result as a visualization. The backend of the demo is deployed as a Docker container, which can be hosted on a local machine or in the cloud. The front end consists only of a web browser. As shown in Figure 2 our demo application is opened in a browser window. There are two tabs for image and video anonymization respectively. In both tabs one can adjust the intensity of the blur effect with a slider, which regularizes the kernel size of the Gaussian blur. The intensity is set in a certain range, such that a blur effect always occurs. In the video tab one is additionally able to toggle between an frame-per-frame process (processing every from independently) and the standard processing method (interpolating detections for smoothness over time). After uploading the desired data, clicking on the ‘submit’ button starts the process. The resulting anonymized media file can then be downloaded via a provided link in both cases. In the case of images, the result is also displayed in the application itself and using the ‘clear’ button sets the application on default. 7https://www.gradio.app/ 6 Korb and Bailer Image Video Gradio REST Mask2Former Inference Blurrer Visualization Link Fig. 1: System architecture diagram. Fig. 2: Demo application implemented with gradio (video tab). 5 Conclusion In this paper, we provide a dataset that has been specifically annotated to human heads and license plates of most vehicles. Furthermore this dataset is also provided with a data augmentation to simulate a tunnel effect, such that trained models can better detect distorted objects. We described the process how we created our dataset and with the resulted trained model we further provide an anonymization service, together with a demo in the form of an web application, to anonymize given media files. Acknowledgements. The research leading to these results has been funded partially by the European Union’s Horizon Europe programme under grant agreement n◦101092875 DIDYMOS-XR (https://didymos-xr.eu/). The authors thank Hermann F¨urntratt for support with the service and demo implementation and Tobias Jandl-Scherf for support with data annotation. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. Segmentation-based Visual Anonymization Service 7 References 1. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance). Tech. Rep. 2016/679 (2016) 2. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). Tech. Rep. 2024/1689 (2024) 3. Bao, H., Dong, L., Piao, S., Wei, F.: Beit: Bert pre-training of image transformers. In: International Conference on Learning Representations (2021) 4. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022) 5. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022) 6. Cordts, M., Omran, M., Ramos, S., Scharw¨achter, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset. In: CVPR Workshop on the Future of Datasets in Vision. vol. 2, p. 1 (2015) 7. F¨urntratt, H., Onsori-Wechtitsch, S., Bailer, W., Ventura, I.A., Navarro, C.S., Jevtic, A., Haidar, J.: Learning scene semantics from vehicle-centric data for cityscale digital twins. In: Proceedings of Conference on Content-based Multimedia Indexing (2024) 8. Hukkel˚as, H., Lindseth, F.: Does image anonymization impact computer vision training? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 140–150 (2023) 9. Jain, J., Li, J., Chiu, M.T., Hassani, A., Orlov, N., Shi, H.: Oneformer: One transformer to rule universal image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2989–2998 (2023) 10. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015–4026 (2023) 11. Li, M., Zuo, W., Zhang, D.: Convolutional network for attribute-driven and identity-preserving human face generation. arXiv preprint arXiv:1608.06434 (2016) 12. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021) 13. Padilla-L´opez, J.R., Chaaraoui, A.A., Fl´orez-Revuelta, F.: Visual privacy protection methods: A survey. Expert Systems with Applications 42(9), 4177–4195 (2015) 14. Peter, R., Grosselfinger, A.K., M¨unch, D., Arens, M.: Automated license plate detection for image anonymization. In: Bouma, H., Prabhu, R., Stokes, R.J., 8 Korb and Bailer Yitzhaky, Y. (eds.) Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies III. vol. 11166, p. 111660R. International Society for Optics and Photonics, SPIE (2019). https://doi.org/10.1117/12.2524023, https://doi.org/10. 1117/12.2524023 15. Ribaric, S., Ariyaeeinia, A., Pavesic, N.: De-identification for privacy protection in multimedia content: A survey. Signal Processing: Image Communication 47, 131–151 (2016) 16. Silva, S.M., Jung, C.R.: License plate detection and recognition in unconstrained scenarios. In: Proceedings of the European conference on computer vision (ECCV). pp. 580–596 (2018) 17. Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al.: Internimage: Exploring large-scale vision foundation models with deformable convolutions. arxiv 2022. arXiv preprint arXiv:2211.05778 2(5), 6 (2023) 18. Xu, Z., Yang, W., Meng, A., Lu, N., Huang, H., Ying, C., Huang, L.: Towards endto-end license plate detection and recognition: A large dataset and baseline. In: Proceedings of the European conference on computer vision (ECCV). pp. 255–271 (2018) 19. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 633–641 (2017) 20. Zhou, J., Beyerer, J.: Impacts of data anonymization on semantic segmentation. In: 2022 IEEE Intelligent Vehicles Symposium (IV). pp. 997–1004. IEEE (2022) 21. Zou, Z., Chen, K., Shi, Z., Guo, Y., Ye, J.: Object detection in 20 years: A survey. Proceedings of the IEEE 111(3), 257–276 (2023)