On Improving Privacy and Security
Through User-Informed Design.
vorgelegt von
MSc.
Marta Piekarska
geb. in Warschau
von der Fakultät IV – Elektrotechnik und Informatik
der Technischen Universität Berlin
zur Erlangung des akademischen Grades
Doktor der Ingenieurwissenschaften
-Dr.-Ing. –
genehmigte Dissertation
Promotionsasschuss:
Vorsitzende: Prof. Dr.-Ing. Adam Wolisz
Gutachter: Prof. Dr. Jean-Pierre Seifert
Gutachter: Prof. Dr. Ross Anderson, FRS, FREng
Gutachterin: Prof Dr. rer. nat. Melanie Volkamer
Tag der wissenschaftlichen Aussprache: 25. November 2016
Berlin 2018
i/xvii
To my Mom, for being the one to read every line of every paper and my
reference point as an everyday expert.
To my Dad, for never letting me rest.
To Jean-Pierre, for letting me run offfree and explore the Privacy World.
To everyone who believed that Piekarska, Ph.D. will not be a misnomer.
“On Improving Privacy and Security Through User-Informed Design.”
iii/xvii
Zusammenfassung
Was ist Privatsphäre? Gemäß der Definition des Oxford Dictionary of Modern
English [4] ist Privatsphäre “ein Zustand, in welchem jemand durch andere weder
beobachtet noch gestört wird”. Allerdings ist die Definition von “gestört werden“
bzw. „beobachtet werden" in der heutigen Zeit, schwierig zu bestimmen. Wenn
wir den Begriffdes “gestört werdens" einmal näher betrachten, dann variieren die
Reaktionen, ob wir ein Verhalten als ärgerlich oder lästig empfinden, von Mensch zu
Mensch. Übertragen wir nun diese Fragestellung in die virtuelle Welt, werden die
Grenzen immer unklarer.
Faktoren, die die Auffassung beeinflussen, was wir unter dem Begriff“störend”
in Bezug auf Privatsphäre verstehen, hängen stark vom Kontext sowie der Infor-
mation ab. In Bezug auf “beobachtet werden” geht es jetzt nicht mehr einfach
nur um die “anderen”. Die “anderen” beobachten uns im virtuellen Raum durch
ihre Computer oder mobilen Geräte, und diese Computer Geräte sind allgegenwär-
tig. Einige “Computer” Geräte werden von den Nutzern gar nicht als solche wahr
genommen, wie z.B. ein Auto oder so etwas profanes wie ein Kühlschrank. Daher
ist es von entscheidender Bedeutung, über Privatsphäre nicht nur im Kontext des
realen Lebens zu sprechen. Ich bin fest von der folgenden Aussage überzeugt, gemäß
des Kontextes der vorliegenden Thesis: Nutzer legen Wert auf ihre Privatsphäre.
Jedoch wissen sie nicht, wie sie sie im virtuellen Raum schützen können.
In meiner Arbeit postuliere ich folgende Aussage: Erstens, aus Sicht der En-
twickler ist es wichtig, die Nutzer nicht als ihren Feind zu sehen. Entwickler müssen
die Nutzer als Verbündete sehen, mit deren Kollaborationen man Werkzeuge en-
twickeln kann, die der Nutzer braucht und die er intuitiv bedienen kann. Zweitens,
die Nutzer sorgen sich natürlich um ihre persönlichen Daten und Information (PII).
Unter PII versteht man Daten, die den Nutzer eindeutig identifizieren können und
die eine eindeutige Verbindung zwischen des Nutzers echtem Leben und dessen online
Persönlichkeit ermöglichen [101]. Um nun Privacy Enhancing Technologies (PET)
besser an die Bedürfnisse der Nutzer anzupassen, müssen diese personalisiert und
gezielt auf spezifische Probleme fokussiert werden. Abschließend ist zu sagen, dass,
obwohl wir unsere PII schützen wollen, wir weder das technische Wissen noch die
richtigen Werkzeuge dafür haben und wir aber auch nicht unsere Privatsphäre für
mehr Benutzerfreundlichkeit opfern wollen. Außerdem ist die Zeit, die es kostet alle
verfügbaren Privacy Policies zu lesen, Schätzungen zufolge umgerechnet 780 Milliar-
den USD wert, wenn man von einem normalen Gehalt ausgeht [13]. Daher sollten
die Werkzeuge, die entwickelt werden, transparent sein für die Menschen, die diese
nutzen wollen, um so einen einfachen Einstieg zu ermöglichen. Die Nutzer mit zu
vielen Optionen zu erdrücken, ist ebenso der falsche Weg, wie ihnen diese gar nicht
anzubieten.
“On Improving Privacy and Security Through User-Informed Design.”
iv/xvii Zusammenfassung
Trotz des hohen Aufwandes, war die Entwicklung guter PETs, meiner Beobach-
tung nach, bisher nicht erfolgreich. Daher lege ich in der vorliegenden Thesis das
neue Konzept, des User Informed Design dar, das neue Wege beschreibt, wie Sys-
teme entworfen werden sollten. Im ersten Abschnitt meiner Arbeit präsentiere ich
meine Beweggründe, basierend auf Angriffen, die wir auf realen Systemen ausge-
führt haben, zusammen mit der verwandten Literatur. Danach präsentiere ich eine
Sicherheitsanalyse bestehender Betriebssysteme und ein Model, welches zu deren
Evaluation verwendet werden kann, um Angriffe zu verhindern. Zusammenfassend
kann man sagen, dass jedes System früher oder später versagen wird. Der Nutzer
sollte aber über diese Probleme informiert werden und muss wissen, wie er mit
diesen umgehen sollte, um seine Privatsphäre zu schützen. Daher präsentiere ich
in dieser Thesis das User Informed Design und verwende dieses im Verlauf meiner
Arbeit, um das Privacy Dashboard zu entwickeln. Ich schließe meine Arbeit mit
einer Diskussion über weitere Einsatzszenarien für das User Informed Design. Diese
neuen Ansätze basieren auf Konzepten aus der Verhaltensforschung sowie aus dem
Bereich der Ökonomie.
“On Improving Privacy and Security Through User-Informed Design.”
v/xvii
Abstract
What is Privacy? It is “a state in which one is not observed or disturbed by other
people”. Or at least that is what Oxford Dictionary of Modern English provides us
with [4]. However with the state of “disturbance” or “observation” today, it may
be a little hard to define. Let’s take the term “disturbance”; the way we interpret
a behavior as annoying or unsolicited varies from individual to individual. When
we move this to the virtual and mobile world, the borders we draw become even
less obvious, as will be presented. The factors influencing user perception of what
can be called “disturbing” in terms of privacy depend on the context as much as
information itself. Now, for the “observation”. It is no longer just about “other
people”. “Other people” observe us through computers and mobile devices. These
computers are everywhere, even in devices that users do not normally consider as
such, for instance cars or refrigerators. That is why it is important to talk about
privacy in the context of computers and not only the personal world. It is my strong
belief, and hypothesis of the following thesis, that users do care about their privacy;
they just do not know how to protect it.
In my work I postulate the following... First, it is time to stop thinking of users
as enemies. They are our allies and we need to collaborate with them to develop
privacy tools that they need and find intuitive. We are all users. Second, people do
care about protecting at least their own Personally Identifiable Information (PII)
- data that can be used to find the link between an online personality and a real
person, and distinguish them from other people [101]. Thus, to improve adaptability
of Privacy Enhancing Technologies (PETs), we need to make them more personal,
targeted at specific problems and focused. Finally, although users want to protect
their PII, they neither have the technical knowledge and the right tools, nor will they
give up usability for privacy. It has been estimated that the time required to read all
privacy policies, if paid as a regular wage, would equal $780 billion [13]. Therefore,
the tools that are being developed need to be transparent, and allow for exploration
and education for the people who are willing to use them. Overwhelming people
with choice is as bad as not giving it to them.
I have observed that despite all the efforts to develop good PETs, we are still
failing. To this end, I present the following dissertation which is a complete work
towards building a new way of creating systems. I call it User-Informed Design
(UID). In the first part of my Thesis, I show what motivated me, based on attacks
we performed, along with literature research I have done. Next, I talk about the
security analysis of Operating Systems and build a model for such evaluation, so
that attacks can be prevented. It can be concluded that any system will fail sooner
or later and, most importantly, we need to make sure that the user is aware of the
shortcomings of their device, knows how to handle it, and protect their privacy.
“On Improving Privacy and Security Through User-Informed Design.”
vi/xvii Abstract
Thus, I present the development of UID, which I then use to create the Privacy
Dashboard, and various other interesting privacy-preserving solutions. Finally, I talk
about the possibilities of enhancing UID with more research and novel approaches
taken from behavioral psychology and economics.
“On Improving Privacy and Security Through User-Informed Design.”
vii/xvii
Academic Achievements
This thesis is based on papers I have published between November 2013 and August
2016, as follows:
1. “Dark Side of the Shader: Advanced GPU-aided Payload Delivery”
Janis Danisevskis, Marta Piekarska, Jean-Pierre Seifert. Published at In-
ternational Conference on Information Security and Cryptology, Seoul, Korea
in 2013.
Contributions: I co-designed and fully implemented the attack as well as co-
authored the publication.
2. “A Metric for the Evaluation and Comparison of Keylogger Perfor-
mance”, Tobias Fiebig, Janis Danisevskis and Marta Piekarska.Published
at Workshop on Cyber Security Experimentation and Testing, USENIX Secu-
rity, San Diego, USA in 2014.
Contributions: I performed the tests, contributed to the creation of the attack
scenarios and co-authored the publication.
3. “What does the Fox Say? On the Security Architecture of Firefox
OS”,M. Piekarska, Bhargava Shastry, Ravishankar Borgaonkar. Published
at International Conference on Availability, Reliability and Security, Freibourg,
Switzerland
Contributions: I was the main contributor to the paper, lead writer and eval-
uator of the system.
4. “Because we Care: Privacy Dashboard in FirefoxOS”,Marta Piekarska,
Yun Zhou, Alexander Raake, Dominik Strohmeier. Published at Web 2.0 Se-
curity and Privacy, IEEE Security and Privacy Symposium 2015.
Contributions: I was the main contributor to the paper and the lead writer.
Additionally I have designed and implemented all of the features presented in
the publication and evaluated in the User Studies.
5. “Can I Let You In? On Event-Centric Context-Aware Authoriza-
tion”, Philip Attfield, Paul Chenard. Marta Piekarska, Julia Narvaez,
Mike Hendrick, Simon Curry. Published at International Conference on Sys-
tems, NexComm 2016, Lisbon, Portugal, 2016. Awarded with Invitation for
Publishing in IARIA Journals,
Contributions: I co-authored the paper, contributed to evaluation of the sys-
tem, and provided critical analysis of existing models. Moreover, I created a
plan for new applications of the system.
“On Improving Privacy and Security Through User-Informed Design.”
viii/xvii Academic Achievements
6. “Towards Improving Privacy Awareness Regarding Apps’ Permis-
sions on Linux Based Mobile OS”, Nurul Momen, Marta Piekarska.
Published at International IFIP Summer School on Privacy and Identity Man-
agement, Karlstad, Sweden, 2016.
Contributions: I created the concept, significantly contributed to development
of the feature, created the prototype and equally co-authored the paper.
7. “Hide Me If You Can. On User-centric Location Accuracy.”,
Marta Piekarska. Published at International Conference on Advances in
Human-oriented and Personalized Mechanisms, Technologies, and Services,
SoftNet 2016, Rome, Italy, 2016.
Contributions: I was the sole author of this work and publication.
8. “Back to the Future: Evaluation Model for Wearables Based On the
Past Experience”,Marta Piekarska, Shinjo Park and Altaf Shaik. Pub-
lished at International Conference on Communications, Computation, Net-
works and Technologies, SoftNet 2016, Rome, Italy, 2016. Awarded with Best
Paper Award and Invitation for Publishing in IARIA Journals
Contributions: I was the main contributor to the paper, lead writer and eval-
uator of the system.
Moreover I have presented three posters:
1. “Hide Me If You Can, Location hiding on Firefox OS”,
Marta Piekarska, Published at ACM WomEncourage, Manchester, England,
2014.
Contributions: I was the sole author of this work and publication.
2. “Hide me if you can. Location Blurring on Firefox OS”,
Marta Piekarska. Published at IEEE Security and Privacy Symposium, San
Jose, USA, 2014.
Contributions: I was the sole author of this work and publication.
3. “s-Tor-y on User Acceptance of PETs”,
Marta Piekarska, Miguel Rios Quintero. Published at International Con-
ference on Security and Management, WorldComp 2016. Las Vegas, USA,
2016.
Contributions: I was the main contributor to the poster, lead writer and the
evaluator of the system.
“On Improving Privacy and Security Through User-Informed Design.”
ix/xvii
Usage of Published Papers in this Dissertation In the following work, I made
extensive use of the research I conducted and papers I published between September
2013 and July 2016. Many of the pieces were written with brilliant and smart people,
and today it is hard to distinguish who was responsible for which part.
I noted where I used direct quotes from the original works. In addition, I would
like to underline that:
•Chapter 2 combines research published in Papers 1 and 2. I was honored to
work with Janis Danisevskis, Tobias Fiebig and Jean-Pierre Seifert.
•Chapter 3 is based on work published in Papers 3 and 8. This could have not
been done without Bhargava Shastry, Ravishankar Borgaonkar, Shinjo Park
and Altaf Shaik.
•Chapter 4 is an extension of work I have published in Paper 4 with Yun
Zhou, Alexander Raake, Dominik Strohmeier. However, the tools for Firefox
OS were developed as part of a project that involved many people: Frank
Wagner, Martin Kurze, Alina Hua, Alex Fowler, Lou Schreier, Dave Huseby,
and many more.
•Chapter 5 describes work that has also been published in Posters 1 and 2, as
well as Papers 5, 6 and 7 with Nurul Momen, Philip Attfield, Paul Chenard,
Julia Narvaez, Mike Hendrick, Simon Curry.
•Chapter 6 revisits some of the previously discussed work, which was published
in Papers 4, 6 and 7.
•Chapter 7 is based on Papers 5 and 8.
I have also been invited to chair sessions and conferences as well as supervised
two Master Thesis:
1. Session Chair at The Eighth International Conference on Advances in Satel-
lite and Space Communications 2016. Session on Satellite and space commu-
nications. 2016. Lisbon, Portugal
2. Track Chair at International Conference on Innovations in Information Tech-
nology. Track on Software and Embedded Systems, 2016 Al Ain, United Arab
Emirates.
3. Supervisor of a Master Thesis by Nurul Momen: “Towards Improving Pri-
vacy Awareness Regarding Apps’ Permissions on Linux Based Mobile OS”.
2016.
4. co-Supervisor of a Master Thesis by Craig Kershaw: “The Effect of Feedback
on Location Sharing and Perceptions of Privacy”. 2016.
My PhD research was partially founded by a BMBF Software Campus scholar-
ship granted by the German Ministry for Education and Research. ((founded? Do
you mean funded?))
“On Improving Privacy and Security Through User-Informed Design.”
xi/xvii
Resume
Education •Technische Universität Berlin – Doctor of Philosophy, 2013 - Present.
•Technische Universität Berlin – Master of Science, 2013 (Double De-
gree). Thesis: Development of a Linux kernel Exploit with GPU aided
payload delivery. GPA: 94%, Degree with honors.
•Warsaw University of Technology – Master of Science, 2013 (Double
Degree). GPA: 94%, Degree with honors.
•Warsaw University of Technology – Bachelor of Science, 2011. Elec-
trical and Computer Engineering, Computer Systems and Networks The-
sis: Voice Encryption on Android Platform Electrical and Computer En-
gineering, Computer Systems and Networks Thesis: “Voice Encryption
on Android Platform” GPA: 92%.
Experience •Linux Foundation
May 2017 – Present
Director of Ecosystem, Hyperledger
•Blockstream Inc.
January 2016 – April 2017
Security Architect
•Apple Inc.
May – September 2015
Software Operations Intern
•Deutsche Telekom, Innovation Labs
September 2013 – Present
Project Leader, Chief Engineer, and Architect: “Future of Mobile Pri-
vacy”
Technical Leader: “Firefox OS Security”
•Technische Universität Berlin
October 2012 – Present
Doctoral Researcher: “User-centric privacy tools on mobile OSes”
Assistant Researcher: “Firefox OS security”
Student Researcher: “Side Channel Attacks on settop-box”
•Yahoo Inc.
July 2014 – September 2014
Security Intern: Account abuse detection system.
“On Improving Privacy and Security Through User-Informed Design.”
xii/xvii Resume
Scholarships •German Federal Ministry of Education and Research. April 2015
– September 2016.
Research Funding - Privacy Enhanced Phone
•Ubicrypt Summer School for Outsanding PhD Candidates.July
2013.
Scholarship, Ruhr-University Bochum
•Outstanding Sports Achievements Scholarship. 2007 – 2010
For Achivements in: Show jumping horse-riding
•Technical University of Prague March 2010
Scholarship, ATHENS program
•Technische Universität München April 2009 – October 2009
Scholarship, LLP Erasmus Student Exchange program
Presentations •Rebooting Web of Trust Workshop
Invited Participation, New York, USA, 2016.
•United Nations ID2020 Symposium
Invited Participation, New York, USA, 2016.
•Invited talks on user-informed privacy
Hackers 2015, Santa Cruz.
•Secure and Confidential Communication
Panel Discussion, Symposium “Let’s Talk Privacy”, 2015, Berlin.
•People Will Talk: Development of the Privacy Dashboard in
FirefoxOS
Presentation, W2SP S&P IEEE 2015, San Jose.
•Privacy Initiative between Mozilla and Deutsche Telekom
Presentation, Mobile World Congress 2015, Barcelona
•W3C Workshop on Privacy and User Centric Controls
Invited Participation, Berlin 2014.
•Invited talks on user-informed privacy
Hackers 2014, Santa Cruz.
•User-centric development of privacy features to FirefoxOS
Workshop on End-User Privacy: Data Collection, Use, and Disclosure,
Berlin, Germany, 2014
•Future of Mobile Privacy
Presentation at Mobile World Congress 2014, Barcelona
•Future of Mobile Privacy
Presentation at CeBIT 2014, Hannover
•GPU-mediated direct-memory-access attacks in a smart phone
Talk, Cambridge Computer Labs, August 23, 2013.
“On Improving Privacy and Security Through User-Informed Design.”
xiii/xvii
Organization •W3C workshop on Blockchains on the Web
Boston, USA, 2016.
•Symposium “Let’s Talk Privacy”
Berlin 2015.
•W3C Workshop on Privacy and User-Centric Controls
Berlin 2014.
•SocHack:Edu
Warsaw 2012.
•Random Hacks of Kindness
Warsaw 2011.
“On Improving Privacy and Security Through User-Informed Design.”
Contents xv/xvii
Contents
Zusammenfassung iii
Abstract v
Academic Achievements vii
Resume xi
1 Introduction 1
1.1 Motivation ................................ 1
1.1.1 Summary of Part One: Attacks Performed on Mobile Devices 1
1.1.2 Summary of Part Two: Creating a Model for Security Analysis
of a New System ......................... 2
1.1.3 Summary of Part Three: Development of Privacy Tools with
the User in the Driving Seat ................... 4
1.1.4 Summary of Part Four: Lessons Learned from User-Informed
Privacy (UIP) and Outlook to the Future ........... 4
1.2 Hypothesis and Plan of Proof ...................... 5
1.3 Contributions ............................... 6
2 Device Vulnerability Below the User 7
2.1 Literature Analysis ............................ 7
2.1.1 At the Very Bottom ....................... 7
2.1.2 Operating System Security ................... 8
2.1.3 Location Security ......................... 8
2.2 GPU-Based Attacks ........................... 10
2.2.1 Background ............................ 10
2.2.2 First Attack ............................ 12
2.2.3 Second Attack .......................... 13
2.2.4 The Proof-of-Concepts Attacks ................. 14
2.2.5 What Does This Bring? ..................... 18
3 Comparing and Evaluating Systems 19
3.1 Security Evaluation of Firefox OS .................... 20
3.1.1 Adding Web to Native Apps .................. 20
3.1.2 Architecture of Firefox OS .................... 21
3.1.3 Comparison and Evaluation of Firefox OS ........... 23
3.1.4 Web-Based Operating System Security Issues ......... 26
“On Improving Privacy and Security Through User-Informed Design.”
xvi/xvii Contents
3.2 Building a Model for Security Evaluation ............... 28
3.2.1 Human-assisted Attacks. ..................... 28
3.2.2 Remotely-controlled Attacks .................. 29
3.2.3 Shift from Laptop to Mobile Devices .............. 30
3.2.4 Model: How to Evaluate a New Device? ............ 32
4 User-Informed Design (UID) in Privacy and Security 35
4.1 Literature Research ............................ 36
4.2 Presenting User-Informed Design (UID) of Privacy Tools ....... 37
4.2.1 User-Centered Design (UCD) and User-Informed Design (UID) 38
4.2.2 First Step ............................. 40
4.2.3 Second Step ............................ 42
4.2.4 Third Step ............................ 46
4.2.5 Methodology ........................... 47
5 Development of User-Informed Features 49
5.1 Adjustable Location Accuracy ...................... 50
5.1.1 General Solution ......................... 50
5.2 Permission Control ............................ 53
5.2.1 User-Informed Privacy (UIP) Matrix .............. 54
5.2.2 Framework ............................ 54
5.2.3 Evaluation ............................ 57
5.3 Event-Centric Context-Aware Authorization .............. 59
5.3.1 Framework ............................ 60
5.3.2 Policy and Policy Management ................. 64
5.3.3 Example .............................. 69
5.4 User-Informed Design (UID) of Privacy Tools ............. 71
6 Lessons Learned 73
6.1 Privacy Dashboard ............................ 73
6.1.1 Guided Tour ........................... 75
6.1.2 Implications for Design of Privacy Dashboard ......... 77
6.2 Permission Meter ............................. 78
6.3 Adjustable Location Accuracy (ALA) ................. 79
6.3.1 User Study Evaluation ...................... 80
6.3.2 Improvements ........................... 82
7 Beyond User-Informed Design (UID) 85
7.1 Mobile Device Management (MDM) .................. 85
7.1.1 Implementation .......................... 85
7.1.2 Lessons Learned ......................... 87
8 Conclusions 89
8.1 Summary of the Work Presented .................... 89
“On Improving Privacy and Security Through User-Informed Design.”
1/114
Chapter 1
Introduction
Mobile devices, wearables, laptops... all are inherently insecure. If we look at the
device-security stack, as shown in 1.1, each of the layers can contain holes and can
be broken. This is universally true - regardless of the form factor of the device.
We strive to build systems that are extremely secure and solutions that will
protect users in case the systems prove to be insecure. Encryption of communication
has been around for a while, and we have amazing cryptographic tools, such as Sec-
comp, SE-Linux, trusted execution environment, process isolation, and sandboxing.
We even have secure Operating Systems (OSes), such as CubeOS and SubgraphOS.
It seems that we have devised every possible solution to make our devices as secure
and attack-resistant as possible. However, we still run into problems, mainly be-
cause we forget about the last layer on that security stack, the one that is not shown
in that figure: “the user”. They choose the passwords that are too short or too sim-
ple. They do not read our guidance about good security policy. They are surprised
how much “The Internet” knows about them. They give away that data while not
understanding the consequences, willingly installing malware on their devices, and
breaking our “perfect” solutions. However they use our devices and products.
1.1 Motivation
Despite our best efforts to educate users, we are failing. Users still do not under-
stand the consequences of their behavior in their online reality, and they become
disappointed with the products, not themselves, when their data is leaked. This
dissertation discusses whether, and how, we can change this situation. It consists of
four main parts.
1.1.1 Summary of Part One: Attacks Performed on Mobile Devices
Part one describes two attacks we have performed on mobile devices. Our work led
to the publication of two papers (one written by Danisevskis et. al. [40], the other
by Fiebig et. al. [53]) and developing a novel method of evaluating keyloggers.
We performed two attacks using the Direct Memory Access (DMA) capabilities
of a mobile Graphics Processing Unit (GPU). While the first attack was more severe
and showed how DMA can be abused for a privilege-escalation attack, the second
attack allowed us to precisely log the user’s key presses. What is surprising is
that both of the attacks could be deployed easily through the Market Place and,
“On Improving Privacy and Security Through User-Informed Design.”
2/114 Chapter 1 Introduction
Figure 1.1: Device Security Stack
in fact our first application was approved by Google Play store. In both cases,
we went through the path of misusing the GPU, which is mainly used to offload
computationally-intensive graphics-rendering tasks to the CPU. Due to its primary
task, the GPU is designed to process large amounts of data with low entropy. This
allows massive parallelism and long pipelines, thus trading latency for bandwidth.
A very good summary of how GPUs work is provided in the works of Giesen et.
al. [59] and Luebke et. al. [95].
Vasiliadis et. al. [157] demonstrated how GPUs can generate polymorphic
versions of malicious code, which is then executed on the CPU. Their work was a
huge breakthrough because it showed that GPUs can be used for attacks and code
development. They anticipated that there could be potential attack vectors from
the unrestricted access to the framebuffer. However, until we completed our work,
no one had actually succeeded in designing and performing such an attack.
In parallel, Carlson [35] published work examining the potential of using co-
processors, such as the GPU, for both defensive and offensive purposes on mobile
devices in particular. He showed techniques for using the mobile GPU for signature
verifications, encryption and decryption, memory tracking and dynamic dissasem-
bly. Following our attack, Ladakis et. al. [89] have presented one step further; they
have created a GPU-based keylogger. Abusing the same DMA mechanism as we
did, they were able to monitor the system’s keyboard buffer directly from the GPU.
However, they still required a CPU process to control the execution of the malware,
and they still needed root privileges to start the attack in the first place.
1.1.2 Summary of Part Two: Creating a Model for Security Analysis of
a New System
Part two discusses how to analyze a system. There has been a shift in the design
of mobile operating systems from custom stacks to web-based platforms. Vendors
“On Improving Privacy and Security Through User-Informed Design.”
1.1 Motivation 3/114
of systems such as Chrome OS and Firefox OS (FxOS) expand the browser from
being another app to a whole platform. Such systems consist of a browser back-
end, residing on top of a Linux kernel, with applications built on top of the browser.
Chrome OS, Firefox OS or Tizen are not exceptions, rather a signal in a shift towards
web-based platforms with similar architectural choices. This convergence between
the mobile device and the web, however, brings unexplored security problems. Both
fields, mobile security and web security, have been well explored in the last few
years, but the combination has not been well understood. In our work, we wanted to
systematize the knowledge about evolving web-based mobile platforms and examine
their impacts on end-user security. By contrasting the security design of existing
native mobile platforms like Android and iOS with the web-based OSes, we were
able to point out the gaps in the security architecture, and outline a possible remedy
for browser OSes. We focused on the Firefox OS, because the open nature of the
project offered good insight into the design and allowed for the introduction of the
proposed solutions.
However, we noticed how quickly the market is evolving. In the past 20 years,
we observed that development was mostly driven by devices, technological infras-
tructure and location-based services [23], [91][127]. The devices became smaller,
more powerful, with greater personal mobility and were more personalized [93]. The
1980s were driven by PCs, the 1990s were about the Internet and connected laptops,
the 2000s became about mobile phones, and later, smartphones. Finally in the last
5 years we have seen an explosion of the Internet of Things (IoT). The most recent
addition to that domain are wearables. Around 2010 we first saw development of
fitness trackers. However, the new generation, developed mainly by computer and
software companies, is more advanced and can perform much more sophisticated
tasks [3]. The market of wearable technology is predicted to rise to over $37B by
the end of 2020 [128]. The consequences of such expansion on privacy and security
may not be well understood.
With the evolution of the form factor, connectivity and sensor technologies,
we can carry around devices that are as powerful as the supercomputers used to
place astronauts on the moon. However, the attack vectors have developed as well,
and each new device type introduces a new space for vulnerabilities. Although early
laptops were mostly susceptible to physical attacks, connecting them to the Internet
introduced malware. When smartphones became popular, with all the mobility,
network connectivity and sensors, more data was collected, turning them into more
personal devices, and a more rewarding target. Thus it was timely to try building
a good model for the analysis of systems, rather than starting from scratch every
time. We did that by observing how the attack vectors and vulnerabilities changed
when we moved from laptops to mobile devices. In the model that we present, we
initially define the assets that a device can hold. Then we try to find the threats, and
identify the vulnerabilities and ways to exploit them. Finally we predict the risks
connected to those we can focus on, designing countermeasures and implementing
defenses.
This part of the dissertation is based on two published papers by Piekarska et
“On Improving Privacy and Security Through User-Informed Design.”
4/114 Chapter 1 Introduction
al. [121] where we talked about the security evaluation of Firefox OS, and a second
paper [120], where we build a model for evaluating wearables and IoT devices based
on the experience from smartphones and laptops.
1.1.3 Summary of Part Three: Development of Privacy Tools with the
User in the Driving Seat
Part three of the dissertation comes from the two previous ones. We found that
the lack of user-friendly, privacy preserving tools the biggest problem; it is the user
who breaks their own security. One of the explanations of that phenomenon is that
users do not care about their privacy. On the other hand, when not in the virtual
world, users tend to protect their personal information – they hide their diaries,
they avoid disclosing private information in front of strangers, and they protect
their correspondence. Our hypothesis was that when it comes to the mobile reality,
the problem lies in a lack of privacy-protection mechanisms and in the education of
the users. The tools are not designed together with the people who will use them.
If user studies are done, they are conducted post-factum to evaluate a solution.
Thus we have created a new way of thinking about building privacy-preserving
tools: a User-Informed Design (UID). We were inspired by user-centered design
(UCD) that emerged from Human-Computer Interaction (HCI). It is a software-
design methodology for developers and designers that helps them make applications
that meet the needs of their users. The approach of UCD is created in three main
iterative steps as the requirement for exploration, design research and evaluation.
Our method is different from User-Centered Privacy (UCP), where the rule is to
design with people in mind. We design with people in the room. We used User-
Informed Privacy (UIP) to develop several features, all combined under the Privacy
Dashboard. We discuss the process and the design of each of those features in detail
in part three of this work.
The development of the UIP method was one of the main contributions to this
dissertation, and part three of it is based on several publications that describe the
process in general [122], as well as various frameworks that we have created using it:
Adjustable Location Accuracy [119], Permission Control described by Momen and
myself [106], Event-Centric and Context-aware access control framework described
by Attfield et. al. [19].
1.1.4 Summary of Part Four: Lessons Learned from User-Informed
Privacy (UIP) and Outlook to the Future
In part four, we talk about everything that we learned from our research:
•The user studies that were performed on the tools we have created.
•The conclusions we draw from building the Privacy Dashboard, the Permission
Control and Adjustable Location Accuracy tools.
“On Improving Privacy and Security Through User-Informed Design.”
1.2 Hypothesis and Plan of Proof 5/114
•How the model we built in part two was applied to the IoT, and how the
framework designed to be Context and Event-centric was applied to Mobile
Device Management (MDM).
•Finally, we look into what can be done beyond UIP, as the method proves
to be useful. Moreover, inspiring research has been done in psychology and
economy that we think can be applied to the field of Privacy.
Mobile devices in general, and smartphones in particular, are crucial to our
daily life. We rely on them, we carry them around, we store all sorts of sensi-
tive data on them. Our observations have pointed out that present trends require
security measures to prevent the invasion on personal data. However, individual
privacy remains vulnerable to several attacks due to the lack of proper knowledge.
Most of the time, users remain uninformed about disclosing sensitive information.
Misconception regarding consequences is usually responsible for privacy-unaware be-
havior. The absence of easy-to-use tools and the complex representation of Terms
and Conditions play major roles behind these bad practices. Sometimes subscribers
are also compelled to compromise their personal information in order to use certain
services. It is hard to convince them to use a proactive approach while only rigid
binary options are provided. As a result, users tend to ignore the privacy statement.
Our two main observations were that individual preference cannot be taken into a
stiffframework and that flexible transparency and personalized tolerance scales are
required in order to design user-friendly tools.
Since smartphones took over the market, the security models changed. As we
observed this, we tried to apply the evaluation model built in part two, to predict
the future problems that will await smart wearables. We then applied the newly
gained knowledge to the IoT and observed that it introduces much more than just a
combination of all of the previous attacks. We attempted to present what we see as
the biggest challenges that IoT will face and proposed defense mechanisms that we
should build prior to the attacks. Having the Framework we built for Context-Aware
and Event-Centric Access Control, we created an MDM solution.
We conclude with some ideas on improvements of UID. We want to understand
what drives the decisions that users make with respect to their privacy and security.
We believe that it would be very interesting to apply work of Kahneman [79] and
continue with research in Behavioral Privacy and Security.
1.2 Hypothesis and Plan of Proof
To summarize: our hypothesis is that, assuming devices are inherently insecure,
the way to bridge the gap between the users’ understanding of how mobile phones
work and improving the adaptability of Privacy Enhancing Technologies(PETs) is
to include users into the process of designing the PETs (and not only in the post-
implementation studies). We do this throughout the work by:
1. Studying existing solutions.
“On Improving Privacy and Security Through User-Informed Design.”
6/114 Chapter 1 Introduction
2. Observing if we can build a model to identify problems.
3. Evaluating systems.
4. Proposing a general method of solving the problems.
5. Developing some features with that method.
6. Presenting the lessons learned and looking towards the future enhancements
of our method.
1.3 Contributions
Most importantly throughout three years of my Doctoral studies I have developed
a new method of working on Privacy tools. I called it the User-Informed De-
sign (UID) of Security and Privacy Tools. The lead-up to this were publications
that contributed to the field with two real-world significant attacks. Both of
them targeted Samsung mobile devices and abused the GPU drivers. Moreover,
my research on Security of Web-based OS and Building models for security
evaluation can serve as a basis for future work in many fields. With the use of UID
I have developed multiple tools that advanc the academic world and industry: De-
veloping Adjustable Location Accuracy,Permission Control Mechanism,
and a Framework for Context-aware and Event-Centric Device Manage-
ment. Additionally, I have built several prototypes that could be used as a basis for
future research and development in Secondary-user Mode and Remote Privacy
Protection. Finally I did extensive research on the psychology and methods of ed-
ucating the users on privacy and security, which resulted in building the Guided
Tour for the Privacy Dashboard. As can be seen throughout the pages of this
work, all of my efforts have been thoroughly backed up and systematized with lit-
erature analysis, which I also see as a significant contribution. There are not nearly
enough Systematization of Knowledge papers in the field.
“On Improving Privacy and Security Through User-Informed Design.”
7/114
Chapter 2
Device Vulnerability Below the User
Although we believe that it is the user who is the weakest link in the security chain
of the security stack, we do observe that the devices we create today are not secure.
We have secure solutions to problems we know. However it is an inherently lost
battle; every day numerous people are trying to break the walls we build. And
with enough pressure, there will always be a crack. In this chapter, we present the
literature analysis and attacks that were my motivation to work on UIP.
2.1 Literature Analysis
2.1.1 At the Very Bottom
It has long been the case that the data is the most valuable part of a device. The most
popular method to steal private information is to create a program intercepting key-
presses, so-called “keyloggers”. Smartphones, however, lack the basic requirement
for such an attack: a physical keyboard. To get data offthese devices it is better
to abuse access to its memory directly. Commonly, certain elements are able to
bypass the Central Processing Unit (CPU) and use DMA to offload high–bandwidth
memory transfers. This powerful feature is useful for debugging and forensics but
can also be abused as an attack vector.
DMA-based attacks have been a popular method of gaining unrestricted ac-
cess to devices for a while now, mostly through peripheral buses such as FireWire,
PCI [36] and Thunderbolt [140]. In 2004, Dornseif [46]showedthefirstFirewire–
based DMA attack. The author used an iPod with modified firmware to obtain
unrestricted access to the main memory of a PC. Boileau [28] extended the attack
to Windows XP in 2006. The concept was further turned into a tool [97]. Breuk
and Spruyt [31] presented multiphase attacks with direct access to the machine.
The first paper to introduce the term DMA malware was written by Stewin and
Bystrov [148]. They present a keylogger that abuses the DMA engine of dedicated
hardware to launch undetectable attacks against the host.
The first touchscreen keyloggers, which read out information in ways similar
to a PC’s keyboard buffer, appeared around the late 2000s, and were published
by Sagiroglu and Canbek [134], Maggi et. al. [98] and Jacob Aron [18]. Around
2012 and 2013, we moved to more advanced attacks that used various sensors on a
mobile device to determine the user input, which included motion sensors [34][168],
accelerometers [112][21] and cameras [146].
“On Improving Privacy and Security Through User-Informed Design.”
8/114 Chapter 2 Device Vulnerability Below the User
2.1.2 Operating System Security
Moving from hardware vulnerabilities, we need to deal with Operating Systems and
all the vulnerabilities they introduce. One of the first documents that presents
the security challenges of mobile devices is [55]. Although outdated, and written
before the smartphone revolution, Friedman and Hoffman gave a great insight into
the differences between security problems faced by mobile OS vendors and those of
stationary computers. Becher et al. analyze mobile network security, attack vectors
using the back-end system and the web browser, as well as the hardware layer.
They go even as far as to discuss the user as an attack enabler. In [135], the authors
analyze the approaches to mobile security adopted by Android and iOS. They give
a good overview of the attack surface and challenges of mobile OSes; however, their
description is high-level. Insight into mobile operating-system security is provided in
Enck’s dissertation [49]. In another of his works, Enck [47] discusses the results of the
smartphone research, including efforts in designing new OS-protection mechanisms,
and performing security analysis of real apps. The author looks into what works,
what has clear limitations, and the promising directions for future research. Papers
like [24][45][170] provide some focus on the state-of-the-art mobile OSes, for which
they present some security enhancements, but they do not cover them fully.
The security of an Operating System, the evaluation of its vulnerabilities and
the systematization of knowledge in the field is almost an impossible task. New and
interesting research is published every month, and the field changes constantly. To
say we have a good understanding of security challenges of a given OS is to say that
the device is out-of-stock and is no longer in use. As long as there are updates to
the Operating System and people are using it, there will be new problems connected
to it.
2.1.3 Location Security
One of the features that has been developed thanks to the UID is Adjustable Loca-
tion Accuracy. While the topic has been tackled for a while by various researchers,
solutions were never really well developed into a product. Probably one of the rea-
sons for this is a wide attack plane and the fear that users would not be aware of it.
Wenrnke et al. [164] proposed the classification of attacks into single position, con-
text linking, multiple position, combination of context-linking and multiple-position
attacks, and compromising a trusted third party. Two factors that determine the
ease of abuse are the access to the knowledge about the context (geography, popula-
tion density, etc.) and history (whether they have only one time data or can gather
information over time).
One of the most popular ways to provide location anonymity is k-anonymity,
in which a region where the aim is to hide user’s position so that it cannot be easily
distinguished from k-1 other users. However, if the attacker has access to even just
single-position reports from enough participants, the attacker can easily identify the
cloaking region and the users within it. Moreover, if the region is not well chosen,
“On Improving Privacy and Security Through User-Informed Design.”
2.1 Literature Analysis 9/114
Alice’s position can be revealed at the moment she joins the cluster. This is the
case when Alice is in a sparsely populated area and thus a well-defined, small region
gets extended to include her. If the attacker gets that updated position, Alice’s
precise location is revealed. Most k-anonymity implementations require some kind
of trusted third party, which can be compromised. A detailed description of flaws
can be found in work by Shokri et al [144].
Moreover, anonymizing tools are prone to context-linking attacks. If a user
always reports their position in a certain place, and they are a single person that
occupies that building, it is easy to identify them. To prevent such attacks, the
k-anonymity model was proposed. However, if the query is performed continuously
for a certain amount of time while the user is moving, and thus the cloaking region
contains a different set of people, Alice’s real position can be easily revealed. It is
also easy to reveal the precise position if the attacker can observe the user for a
period of time. Once a physical person is linked to their pseudonym, the history can
be retraced.
Another way to distort the precise location is through position dummies —
sending multiple fake coordinates along with precise ones. This, however, can be
broken with map matching and context linking. By “map matching”, we mean a
situation where the user’s region can be narrowed down by correlation with a map.
For instance, it is less likely that the user is on a lake or in a forest if a bridge
or a hotel is nearby. This method in general is very hard to prevent — almost all
solutions, including ours, are vulnerable to a certain degree.
Spatial obfuscation and coordinate transformation can be broken with methods
of combining context-linking and multiple-position attacks, as well as the maximum-
movement boundary attack. In all of those cases, the attacker needs to gather
multiple data points and some context information. Later the attacker can increase
the probability of finding Alice by shrinking the region. The attacker can also try to
find the intersection of the reported regions, or define the distance that a user could
cover within a given time, thus revealing the user’s path. A basis for performing
context-linking attacks is given in [37], where the authors study behavioral patterns
of the users, and show that that 40% of updates are from the users’ primary location
and 80% are from their top 10 locations. Our solution is not prone to that attack,
because we always report the same position for a given cell.
Most solutions attempt to hide the position reported from the GPS chips.
However, there are multiple other methods of obtaining data about where the device
is. These side channels are yet to be studied in detail and include the WiFi hotspots
(to which the user is currently connected) and their IP address. The IP address
of a device correlates with the region it belongs to on a ZIP-code level. Without
some form of IP masking, any fuzzing algorithm is easily defeated. An interesting
proposal against this problem has been presented in [50]. Not only do users report
their position to the applications, but increasingly they choose to reveal who is with
them there. This can often be observed on social networks where people “tag” the
names of their friends so that others can join them. Even if the location is obfuscated,
such information can be used to improve the attacks and reveal a lot of private data
“On Improving Privacy and Security Through User-Informed Design.”
10/114 Chapter 2 Device Vulnerability Below the User
about the user, their friends and other people in the same social group. The problem
has been extensively studied in [2], where the authors tried to quantify the effect on
location privacy. They have later proposed a user–collaborative privacy-preserving
approach for Location Based Services(LBSs) [143][142].
2.2 GPU-Based Attacks
There are many well known vulnerabilities in the devices available on the market.
The previous section detailed how various elements of the stack can be abused. If
we compare a device today to a feature phone from the early 2000s, one of the
most salient differences is the size of the display. Today, the graphical interface and
touch interactions are the basics of any mobile device. These changed the way we
think of them, but also introduced a whole new attack vector; the underlying GPU
technology had to be given DMA in order to maintain the performance.
The first attack was designed during my master’s studies under the supervision
of Janis Danisevskis and Jean-Pierre Seifert. We wanted to abuse the mechanism of
the DMA of a GPU on a mobile device to copy some chosen data. While analyzing
the drivers of a popular smartphone, Janis found a bug that allowed for the verifi-
cation of the theoretical attack vector in practice. Eventually, we could deploy the
first fully GPU-based malware and affect millions of devices. Although we informed
the manufacturer and gave them time to fix it, we found that they did not manage
to update everything. Thus, later we could still perform another attack, effectively
stealing the input that a user entered on their smartphone or tablet.
While our first attack was severe due to its novelty and ease of installation
on many devices out there on the market — we did manage to get it into the
Android Play store — the second one was the first zero-privilege keylogger applicable
to Android mobile phones and Android tablets. On mobile devices, the memory
that contains the graphical output is restricted only to privileged processes. So
if an attacker can access it, the damage is very significant, much more than on a
Personal Computer. If we start with (as we did) an unprivileged attacker, the impact
becomes very severe because the information that was supposed to be confidential
(e.g. contents of a message displayed on the screen containing a password or a One
Time PIN sent by the bank) becomes accessible to a malicious third party.
2.2.1 Background
The following section appears as presented in the original paper [40]. This includes
all the graphics As basic knowledge of the architecture of the System on Chip (SoC),
(DMA) and basics of GPU programming, to describe how we were able to perform
the attacks.
“On Improving Privacy and Security Through User-Informed Design.”
2.2 GPU-Based Attacks 11/114
CPU
MMU
GPU
Peripheral bus controllers
UART
I2C
SPI
GPIO
...
Display
Controller
Memory
Controller
Memory DMA
Controller
Peripheral bus controllers
USB
Sata
SDIO
AC97
...
System Bus
SoC
MMU
Figure 2.1: A typical System on Chip (SoC) that can be found in a smartphone
System on Chip (SoC) Architecture
A typical smartphone SoC combines multiple building blocks on a single silicon
substrate. This consolidation of integrated circuits is very efficient, both in terms
of manufacturing cost and in terms of integration size. Figure 2.1 shows a very
simplified sketch of an SoC, with focus on the communication between the building
blocks. The system bus connects all the building blocks and the main memory,
which is usually not part of the SoC itself. On a real SoC, the system bus is usually
divided and comprises many physical interfaces and protocols. Everything that is
connected to the system bus, including each byte of the main memory, is addressable
through a unique physical address. An entity that can put addresses on the bus and
then read from, or write to, the addressed resource is called a “bus master”. On one
hand, the distinguishing feature that makes a smartphone such a versatile gadget
is its capability to run third-party applications. These applications, however, can
be buggy or intentionally malevolent and must be considered untrustworthy. The
CPU which executes them is, on the other hand, the most prominent bus master in
the SoC. To confine potentially misbehaving entities, the operating system provides
each application with a virtual address space by means of a memory management
unit (MMU). The operating system of a smartphone carefully abstracts from the
peripherals in the system. However, there is one building block that can be pro-
grammed by user applications and act as bus master at the same time: the GPU. As
shown in Figure 2.1, the GPU also deploys an MMU to restrict access to the system
bus, thus confining user jobs. The GPU’s MMU must be very tightly controlled by
the operating system so as not to jeopardize the integrity of the system.
Graphics Processing Unit (GPU)
The programmer uses a high-level API such as Open Graphics Library (OpenGL) or
Direct3D to program the GPU. The primitives that comprise an OpenGL program
are riddled with terms stemming from the field of computer graphics. Figure 2.2
shows a typical graphics-processing pipeline (GPP). The input to the GPP is a set
of attributes,uniforms and textures. Attributes are user–defined and hold per-vertex
meta–information. Vertices are three–dimensional vectors that denote endpoints of
“On Improving Privacy and Security Through User-Informed Design.”
12/114 Chapter 2 Device Vulnerability Below the User
Figure 2.2: A typical graphics-processing pipeline (GPP)
lines or corners of polygons. The set of vertices represents the geometry of the scene
to be rendered. Uniforms hold per scene meta–information provided by the user,
such as a rotational matrix. Textures are used to increase the level of detail by
mapping images onto the geometry’s surface. Recent versions of the graphics pro-
gramming APIs allow the use of shaders: user–provided programs that are loaded
into the programmable stages of the GPP. Shaders are written in an extension-
specific language (e.g. GLSL in the case of OpenGL [108] or HLSL in the case of
Direct3D) and compiled into an architecture–specific instruction set by the GPU
driver. The first stage of the GPP operates on the geometry of the scene. Here, the
user–defined vertex–shader is used. On recent GPUs, this might be accompanied
by either geometry or tessellation shaders, or both. As a result, a set of primitives
(triangles on a 2D plane with additional meta–data) is produced. In the following
rasterization phase, meta–information about each sample of the rendered scene is
generated through interpolation along the primitives. These samples, called frag-
ments, are processed in the last programmable stage: the fragment–shader. The
result is written to the output buffer.
2.2.2 First Attack
A full description of the attack can be found in my master’s thesis. We have since
improved several elements and published a paper, which has contributed to my
Doctoral work. Therefore, I include it in this dissertation in a more concise version.
Interested readers should refer to the paper [40] or to the Thesis itself.
The attack was a multistage process, as shown in Figure 2.3:
1. We needed to gain control over the memory-protection mechanism of the GPU.
2. We could misuse it to get read and write access to the highly privileged memory
regions.
3. We would copy some data into it, which we chose to be an executable shell
code.
4. We executed the code
5. We got root access to an unprivileged process.
It seemed very plausible that because OpenGL can be used to render an image
to the screen with no changes, and an image is just pure bits, the same mechanism
“On Improving Privacy and Security Through User-Informed Design.”
2.2 GPU-Based Attacks 13/114
Figure 2.3: The attack vector
could be used to do a copy of binary data. That was exactly what we did: an
identical scene rendering to mimic the C–library function memcpy.
The geometry of an identical scene is a flat square that fills the whole space that
is visible to the user. Thus, we created a single square using a set of four vertices,
which we just pass through the vertex shader to the next stage. We do not need to
change it or alter the geometry; we are rendering an identical scene. However, we do
want to add the executable code. This is done by using a texture that is mapped onto
the square. We fit it into the correct position by passing the texture coordinates to
the GPP as attributes that are assigned to the varying texture_coord in the vertex
shader. Next, the texture is sampled in the fragment shader, and because nothing
changes in the dimensions of the image, an exact copy of each texture pixel (texel)
is rendered. As we described Section 2.2.1, the framebuffer is just another region
in main memory. With this, we achieve steps one and two; the GPU can render an
executable code into the memory space of our choice.
2.2.3 Second Attack
The second attack goes one step further and takes a slightly different approach.
When a user types on a display-based keyboard, it is often good practice to give
the user some feedback on which key was pressed. This is especially necessary for
passwords. The techniques used are to either leave the last letter visible or display
a little preview, a so–called “tooltip”, of the key pressed on the keyboard, as shown
in Figure 2.41
Anything that is at any given point kept visible or not encrypted, yet contains
valuable information, sounds like a tempting attack target. Although looking over
someone’s shoulder is not feasible, reading out from the display by remotely accessing
1Picture comes from our paper[53].
“On Improving Privacy and Security Through User-Informed Design.”
14/114 Chapter 2 Device Vulnerability Below the User
Figure 2.4: Android soft-keyboard with a currently active tooltip for the letter “j”.
it may sound like a good way to try stealing the credentials. As was previously
described, anything that is shown on the screen is subject to double buffering, and
GPUs have DMA, which can be abused, as presented by Ladakis et. al. [90], Fischer
et. al. [54], Wendt et. al [162]us[40].
Let us assume a malicious attacker has installed a keylogger on a victim’s
device. The installation may either be performed by tricking the user into installing
a seemingly harmless application containing the keylogger or by utilizing a known
flaw that allows the installation of arbitrary applications. The attacker then receives
the gathered keystroke information via the network.
2.2.4 The Proof-of-Concepts Attacks
Both of our attacks targeted a very popular smartphone model running Android,
which employs an ARM Mali MP 400 GPU. Based on a statement from the vendor,
this GPU can be found in every fifth Android phone and most Android tablet de-
vices [160]. We found that several firmware versions were vulnerable to our attack.
Furthermore, the successor of the targeted model was vulnerable as well.
The goal of the first proof–of–concept exploit was to elevate the privileges of
a normal user–space application by patching the kernel’s text section. After write
access to the kernel memory is established, this is fairly straightforward, and has
been described before [94,118]. In the follow-up attack, we decided to abuse another
bug that was found in the drivers, which allowed us to read out the contents of the
framebuffer that stored whatever was displayed on the screen, thus proving that an
attacker can gain access to privileged output information from the “user world”.
“On Improving Privacy and Security Through User-Informed Design.”
2.2 GPU-Based Attacks 15/114
The Bugs
This section appears as originally presented in the papers [40] and [53], to precisely
describe what lead us to being able to perform our attacks.
The Mali driver stack is split into a part that resides in the Linux kernel and
a set of user–space libraries. The kernel driver exposes its interface to the user
through the device node /dev/mali. Access to this device node is not restricted.
This is not a problem because the kernel driver provides abstractions that allow,
in principle, safe usage by multiple users. Most notably, for our cause, it provides
each user session with a private address space that is controlled by means of the
GPU’s MMUs2. The user–space libraries make up the larger part of the driver.
They provide common APIs such as the embedded graphics library (EGL) and the
OpenGL. The lower end of this part of the driver stack connects to the user–kernel
interfaces of the framebuffer and the Mali driver. Their task is to populate the
session address space, lay out graphics-processing jobs in the allocated memory, and
submit them to the kernel driver. It should be noted that the values written to
the registers of the Mali GPU come directly from the user space. With the above
mentioned, we can state the following assumption, that must hold for the driver
stack to sustain the kernel-enforced memory-space isolation. User should be able to
run any code the GPU as long as the session address space of the GPU does not
violate the memory isolation imposed by the kernel.
The Mali kernel driver provides the user with three mechanisms to populate
the session address space of the GPU. The user can (1) call mmap on the open session,
which will allocate physical memory tiles, map them into the caller’s address space
and, at the same time, into the session address space of the GPU. The Mali driver is
accompanied by the universal memory-provider (UMP) kernel driver, which allows
the user to allocate physical memory buffers. Applying mmap on the UMP session,
the user can have those buffers mapped into their process’s address space. As the
Mali driver is UMP-aware, the user can also (2) attach UMP buffers to the session
address space. The third mechanism allows the user to (3) supply the kernel driver
with physical addresses. This mechanism was introduced to map the framebuffer
memory into the GPU’s session address space. As long as the driver performs a
range check on the supplied addresses, this is a viable technique. But in the case
of the driver we assessed, this range check was misconfigured so as to include all of
the physical memory. This allowed any user to map any part of the main memory
into the GPU’s session address space. Thus, the assumption we stated above was
violated.3
Further investigation showed that the SecureID that is used to reference the
active UMP has a maximum entropy of only 20 bits. Despite the design specifica-
tion stating they should be 32 bits long, only 20 bits are usable, because the offset
2The Mali MP 400 GPU has one geometry processor (GP) and up to four pixel presenters (PP).
Each of these processing cores has its own MMU.
3The Original Equipment Manufacturer (OEM) was informed about our findings and fixed the
bug.
“On Improving Privacy and Security Through User-Informed Design.”
16/114 Chapter 2 Device Vulnerability Below the User
Figure 2.5: This figure depicts the chain of events for each single key that is logged.
(a) After a key has been pressed on the soft-keyboard, a UMP buffer is attached
and the corresponding bitmap for the pressed key is put into the buffer. (b) The
buffer is attached by the GPU, which then (c) uses it to render the full scene onto
the display. Then in (d),whilethebuffer created in (a) is still present, the keylogger
attaches it. (e) The Keylogger creates a MD5 sum of the buffer contents, sends
these out and releases the buffer.
parameter of mmap is used to store the SecureID, which is forced to have page granu-
larity. In addition, the reference implementation assigns SecureIDs starting with 1,
incrementing until an unused SecureID is found. The UMP-allocated space is used
to render the contents of the framebuffer. This means that an unprivileged process
running on a target device does not require extended privileges to obtain read-access
to graphical system output. Thus, if the output is not fully handled within some
kind of a hardware-protection solution, all bitmaps used in the pre-processing stages
of system output can be accessed. We have found that the tooltips presented to the
user when they press a key (see Figure 2.4) follow exactly this mechanism. The Mali
GPU framework receives the bitmap through an attached dedicated UMP buffer, as
shown in Figure 2.54. If we can identify the buffer that is used to store the tooltips,
then read and analyze it, we can create a precise log of the user’s keystrokes.
Details of the First Attack
We wanted to use the newly gained power, and abuse the lack of range checking on
mmap, to do a privilege-escalation attack. We chose to patch a system call with code
that changed the caller’s user id to 0 or root, based on the work by McAllister [100].
It was simply turning C-code of:
commit_creds(prepare_kernel_cred(NULL));
into assembly, to later pass it as a texture to the GPU, which looked innocent, as
shown in Figure 2.65. The system call that got patched was sys_reboot, because
it is only invoked when powering-down the system, so the side-effects of our proof-
of-concepts attacks were smaller. After finding the physical address of the right call,
4Picture comes from our paper[53].
5Picture comes from our paper[40].
“On Improving Privacy and Security Through User-Informed Design.”
2.2 GPU-Based Attacks 17/114
Figure 2.6: The payload as 4-by-4 texture padded with 6 words of 0 when inter-
preted as RGBA pixel format
we could do the ioctl call on the device node /dev/mali. Two additional changes
that were important included rounding the physical address to the nearest page
boundary and configuring the Mali GPU write–back–unit, because it has an address
register that denotes the target buffer where the GPU will output the result of
the GPP. The address was the syscall to be patched in the Mali–virtual address
terms: sysc_mali_virt_page plus the information that we lost by rounding earlier.
Finally, we were able to perform a copy with some additional hacking involved. We
encountered some pitfalls; these have been described in the paper and my Thesis,
and so are not included here. Even though the copying was not as exact as we
anticipated, the exploit worked flawlessly.
Details of the Second Attack
In the second attack, we approached the problem differently. We assumed that the
contents of the framebuffer most interest the attacker, and decided to see if we can
read it out without any special privileges. We wanted to show that the low-privilege
access to system output is a severe issue on mobile systems.
The attack was performed in three steps. First, we needed to find a buffer
where the tooltip is stored. Then, we wanted to get the data out of that memory
region. After looking at the memory and investigating it closely, we found that the
framebuffer sizes are static and dependent on the screen resolution and the pixels per
inch of a display. Next, we compared the contents and distribution of the buffer sizes
when keys were pressed or not. It turned out that certain sizes only appear when
the tooltips are shown. This made our task easier - to be able to use a designated
system call on the UMP buffer to determine its size before it got attached. So, if the
size of the buffer did not meet our criteria, we would just not allow for attaching it.
One security measure created by the manufacturer was to reestablish the SecureID
of the buffers from scratch after every reboot. To work around it, we just started
with SecureID 1, and checked the size of the buffer. Each time it did not match
the expected size, we incremented the ID to 2, and so on, until we found the one
we were looking for. This added some computational load on the CPU, reducing
the stealthiness of our attack; on the other hand we never found the SecureID to be
higher than 30.
Extracting data from a found framebuffer meant translating the bits of the
bitmap into a valid character. It proved to be quite a computationally exhaustive
task, because even the smallest tooltip was represented by 80kb of data, and the
“On Improving Privacy and Security Through User-Informed Design.”
18/114 Chapter 2 Device Vulnerability Below the User
bigger the display, the greater the stored information. Mobile devices are not able to
process such data fast enough to keep up with the typing speed of an average user.
Even with the use of the pattern-recognition algorithm that we decided to use, it was
still too much. We followed the suggestions of Gao et. al. [57] and simply compared
data with the device from one previously gathered on a “learning” device6.We
did this offline by calculating MD5 hashes of the data in the framebuffer to further
reduce the bandwidth usage and storing it locally together with the timestamp and a
soft overflowing cycle counter to identify each press uniquely. When the connection
was available available, we could send all of the recorded strokes over the network.
The process of key extraction is shown in Figure 2.5. During our proof-of-concept
attack, we did analysis on a remote database, but in reality we would use techniques
proposed previously by Bernard [22] or Schlegel [138].
2.2.5 What Does This Bring?
With two minor bugs — a poor range check and a lack of proper implementation of
the UMP-buffer attachment — we showed how a whole system can fall apart. The
attacks we presented allowed for privilege escalation and key logging. We observed
that before any output is rendered on the screen, ARM Mali GPU devices store it in
UMP buffers. Moreover, we showed that if an unprivileged application gained access
to this memory, it could steal sensitive information from the user’s screen, regardless
of whether it was input by them or only displayed. The information could be text
messages, photos, emails or even the mobile Transaction Authentication Number
issued by banks to authorize transactions online. Similarly, Keychains or Credential
Stores, where the user protects all their passwords under a single Master Key, are
now readable when decrypted into the framebuffer.
Most importantly, none of this could happen if users understood what was
happening on their devices and how to protect themselves. Malicious applications
are mostly installed by the users themselves when they are tricked into clicking “yes”
through social engineering and phishing. It is impossible to stop many of these
attacks. As part of their necessary education, we must create a situation where
users can at least understand that it is better to store as little as possible, and to
store less sensitive information on their devices. We must help users to understand
that the devices we build are inherently insecure, and it is also their responsibility
to think of the consequences if their data is leaked.
6This of course assumes knowing what device we are attacking and having access to it. Further
discussion on methods that did not include having access to the device was presented in the
paper.
“On Improving Privacy and Security Through User-Informed Design.”
19/114
Chapter 3
Comparing and Evaluating Systems
It can be said that John von Neumann was the last person to know and understand
a computer as a whole. However, von Neumann died in 1957. The reality is that
modern computers are systems so complex that no one is able to grasp the whole.
Fifteen years ago we had laptops. Having a personal device like that made it eas-
ier to collect data and potentially access private information through Trojans and
spyware. In 2007, Steve Jobs introduced a smartphone: a tiny Personal Computer
with permanent Internet Connectivity and calling capabilities that people carried
around non-stop. This shift changed the attack vector, and threat models became
slightly different. Another jump is happening today through smart wearables: the
embedded medical devices, fitness trackers, smartwatches, and potentially glasses
or connected jewelry. It is another increase in capabilities and a decrease in size.
It is also another opportunity to collect even more sensitive data about the owner.
Inevitably, the attacks will again change and will become more user-data focused.
Not surprisingly we see the urge to connect the devices together into IoT, mostly
for usability purposes (keeping multiple copies of the same data) as controlling each
machine separately becomes a burden. But this interoperability brings an unfore-
seen consequence; the number of attack vectors does not simply sum up with each
device we add, it multiplies.
We have successfully launched two attacks on Samsung devices, proving that
even small holes can escalate into big problems. Going from bottom up, we now
look at the vulnerabilities above the kernel level and analyze a web-based Operating
System, Firefox OS (FxOS). Since the introduction of the first smartphones, the
market has grown dramatically. We mostly witnessed a division between Google’s
Android and Apple’s iOS platforms, but many other players have tried to come to
the game as well: Samsung’s Tizen, Mozilla’s Firefox OS, and Canonical’s Ubuntu
Touch OS to name a few. Actors involved in the smartphone production and sales
process — Original Design Manufacturers (ODMs), Original Equipment Manufac-
turers (OEMs), and telecom companies — have had varying levels of concern about
the growing duopoly of Google and Apple. While some, like Samsung, have intro-
duced their own mobile OSes, others, like Mozilla, have benefitted from the progress
on a unified HTML5 standard [123] to transform an existing browser into a platform.
The World Wide Web Consortium’s (W3C) work towards HTML5 and device API
standardization has had two side-effects that have made web-based platforms more
attractive to these actors: standardization offers actor-neutral APIs that ensure in-
teroperability, and the process itself is non-discriminating and neutral. Furthermore,
“On Improving Privacy and Security Through User-Informed Design.”
20/114 Chapter 3 Comparing and Evaluating Systems
in the particular case of FxOS, actors could influence not just at the API level but
also platform development itself. We decided to investigate how FxOS was designed
from the bottom-up with security in mind, and if platform developers have indeed
learned from the mistakes of native OSes like Android. We attempted to provide
a comprehensive understanding of the FxOS security architecture and formulate a
threat model for Web-based Mobile Operating Systems (WMOS). We also analyzed
the FxOS architecture and pointed out its shortcomings. Finally, we stated open
problems that give directions for future research.
After performing this work we realized there is no good framework that would
suggest how to evaluate a new device or an OS in a systematized way. The literature
research did not provide us with any good methodology for that. We performed our
own analysis of security and privacy threats on laptops and mobile devices, and
we listed the vectors and threat models. From that, we built a threat model that
discusses how different factors influence the impact of various attacks, thus giving
an incentive to perform them. We later applied our model to the next big thing: the
wearables. Finally, based on our research we could present some countermeasures
and defenses to be applied to the identified problems.
3.1 Security Evaluation of Firefox OS
Mobile and web have converged into WMOS. Web brings to the mobile OS the
increased capabilities but also the possible security issues. Although we saw dis-
cussions on each of the fields separately, our work is one of the first to analyze the
security of the two combined. In this section we systematize the knowledge about
still evolving, web-based platforms and examine their impacts on end-user security.
3.1.1 Adding Web to Native Apps
On a WMOS, an app is more or less what a webpage is in a browser. However, unlike
a webpage, an app has some access to the device APIs such as the storage API for
offline access and data storage. It is a bit of a mix: the web application contains
the same elements as a web page would (the source code is written in HTML5,
Javascript and CSS) but it contains a manifest file like a native app would. Even
though the applications can, to some extent, take control over the device, they do
not necessarily need to be installed on the phone. Often only the manifest is stored
on the device, and the rest is hosted on a remote server to which the user connects
through the app’s icon (which serves as a shortcut). It means that the web app’s
code cannot be verified and is controlled by an untrustworthy third party. As a
result, the web applications have to be treated with great care, and the methods of
threat identification and damage containment become crucial on WMOS. Moreover,
using the browser as the basic OS means that the same attacks we know from the
web are now part of the threat model. Additionally HTML5, JavaScript and CSS
vulnerabilities expose system resources and user’s private information, e.g. contacts
or e-mails.
“On Improving Privacy and Security Through User-Informed Design.”
3.1 Security Evaluation of Firefox OS 21/114
One could argue that it is enough to just make people aware that their phone is
“just like their Firefox or Chrome browser” and they will be more careful. However,
many studies have shown how warnings and notifications fail. Felt et. al. [51] found
that during installation, less than one fifth of participants checked what permissions
were requested by Android apps, and no more than 3% of them actually knew what
the consequences of granting them meant. Even more scary was the survey by
Akhawe et al. [14]; around one quarter of users click through phishing and malware
alerts, and three quarters continued through SSL warnings. These studies indicate
that usability studies must go hand-in-hand with the security and User Interface
(UI) design of a platform, especially because the security implications of a web-
based platform is not well understood. What is certain is that on WMOS, the web
pages are seamlessly built into the system, which can be misleading to the user.
They do not know that by clicking on an app, they trust more than a web page on a
laptop: they are effectively opening the same connection to the same remote server.
A suspicious request for credentials is much easier to ignore than one coming from
a malicious app.
Section 3.1.2 and Section 3.1.3 are direct quotes from the original paper “What
does the Fox Say? On the Security Architecture of Firefox OS” published at ARES
2014 [121]1. We compared FirefoxOS with the Android 4.4 and iOS8 - the most
recent versions of the systems at the time of writing. Today both of them have
advanced in terms of privacy and security. However, because FirefoxOS is a project
that is no longer supported by Mozilla, we kept the original comparison.
3.1.2 Architecture of Firefox OS
Any app for FxOS was a web app, meaning it uses HTML5, JavaScript, CSS or any
other open-web technology. Moreover, FxOS, did not allow for binary applications
(also known as “native apps”). Consequently, all system calls, including storage ac-
cess, are done through the Web APIs. Additionally, the second mechanism divided
applications into three trust groups: certified, privileged, and web. Only certified
apps, most trustworthy, came pre-installed on the phone could access the Telephony
API. Privileged apps were the third-party products that went through the verifi-
cation process in the Marketplace. The last group, applications not requiring any
special permissions, did not undergo any vetting and could be installed from arbi-
trary sources2. In any case, all programs had to be installed prior to launching, and
all had to contain a Manifest file that lists the required permissions, regardless of
their access level. In FxOS, similarly to iOS and Android, the main principle was
the one of least authority; by default an app had absolutely no access to any API,
and the requests in the Manifest should be as few as possible. Every time an app
calls an API, Gecko consulted the Manifest file to check if the app was listed and if
its type (certified, privileged or web) was sufficient to grant the rights. Depending
on the API type, the access was either granted, or set to ’prompt’. In the first case,
1All graphics appear as original in the paper
2A detailed description of each API is available at https://wiki.mozilla.org/WebAPI
“On Improving Privacy and Security Through User-Informed Design.”
22/114 Chapter 3 Comparing and Evaluating Systems
Figure 3.1: The review process in FxOS
if the app successfully passed through the Marketplace, it got implicit access to the
API. However, if the API access threatened user privacy, or user could be expected
to understand the impact of their choice, they were presented with a prompt. The
user could change their decision at any time in the settings. Unlike in Android, the
app had to function even if it was not granted the access it asked for.
App Vetting and Distribution
Placing an app on the phone was possible through one of the three paths:
•Certified applications came pre-installed with the device.
•Web apps could be downloaded from any place; at the cost of limited access to
the web APIs, the developer did not have to do anything apart from placing
an app on his server and convincing people to run it. This type of software
could be treated similarly to a bookmark — clicking on an icon connected user
with the web page on which the application was stored.
•Hosted apps came together with a Manifest that needed to be stored on the
device.
To distribute a privileged or web app through a Marketplace, developers needed to
package the code into a zip file containing all of the HTML5, JavaScript, and other
resources as well as the manifest file. The detailed process of app vetting can be
found in [43], while Figure 3.1 shows the general concept. Mozilla only allowed
installation from their own Marketplace, and the review assumed different security
levels for the store and the validator.
Sandboxing
Each each application could only access a limited number of Web APIs, data, and
resources like database, cookies or offline storage, as it was opened in a separate
sandbox. The core of Gecko was common for all Firefox products. The main process,
“On Improving Privacy and Security Through User-Informed Design.”
3.1 Security Evaluation of Firefox OS 23/114
b2g, was analogous to a browser window, but run with the highest privileges. Any
application that was opened in FxOS was treated just like a tab in a browser - it
would start in a runtime environment that was a child process of b2g, got a limited
number of capabilities through the Web APIs, and whenever the application wanted
to make a privileged call, b2g had to decide whether to grant or deny it. The apps
were not able to directly communicate between each other, nor could they open
each other directly. If it was necessary, app A could create an event and send it
to the b2g, from which a listener of app B intercepted it. Programs accessed the
mobile-phone functionality through the Web API and could not talk directly to the
hardware. Everything was mitigated by Gecko.
Two applications installed on Firefox had a completely different set of cookies,
different local data, and different permissions. Moreover, if they both opened an
iframe tag in HTML5, pointing to the same address, the site was rendered separately
for each of them, granting each a separate cookie, and was not be aware that it
was opened twice on the same device. This had impact on usability: data entered
on one page was not passed to the other, which meant users had to enter login
credentials separately for each iframe. With the concept of a browser as a web
platform, this approach was also leveraged when web app A wanted to open app
B. Web applications were hosted on a server, so by opening an app, the user was
forwarded to that website. Thus, when app A opened app B, simply another instance
of a web page was rendered, and as in previous cases no data was shared between
two instances of app B. This of course meant that a lot of resources were duplicated,
because there was no one common directory to store the data. If the app was not
of a web type, opening B in an iframe failed because it did not point to any server.
To prevent covert channel attacks, the fail message was the same whether an app
was not there or if it was a packaged app. The sandboxing mechanism extended to
the access-control model as well. The user granted permissions only to a particular
instance of an app: if an app opened a website and requested access to location
information, the permission did not transfer to the same website opened by another
app. For highly web-interactive applications, there were two sandboxes: one for the
program and the other for the web contents it opens.
3.1.3 Comparison and Evaluation of Firefox OS
In table 3.2, we compared platform-security features related to different threats for
FxOS, Android, and iOS.
End-user Privacy
Android apps and ad libraries have been known to leak device-identifiers like IMEI/IMSI [48].
Being able to uniquely identify a user via such IDs, and subsequently using that in-
formation to track a user across applications, poses a significant privacy threat to
the user. Unlike Android, Firefox OS did not expose an API/permission to third-
party applications to query the device for its unique ID, and only certified apps had
“On Improving Privacy and Security Through User-Informed Design.”
24/114 Chapter 3 Comparing and Evaluating Systems
access to these IDs. Given the security audit that the FxOS went through, one
would expect privacy threats to be caught at the stage of audit.
Although a mandatory part of the 3GPP specification, a user-level API to
query the level of encryption of the cellular connection from the modem is missing
on Android [15]. According to all specifications, users should be informed when
the connection they are making is unencrypted. None of the Firefox phones ever
provided such ciphering information to the end user. Unfortunately, even with our
work we did not manage to bring the issue to Mozilla’s attention.
Operational Security
Third-party apps were not given root access on FxOS. Both Android and iOS have
to be ’jailbroken’ to obtain administrative privileges for the device. FxOS could be
opened as a root with Android Debug Bridge (adb) daemon. However, in production
builds, the adb daemon run with lower privileges. It should be noted that a mo-
tivated adversary, including the user, could use privilege execution payloads for an
adb shell to gain root access and take complete control over the device. The FxOS
security model did not preclude such an attack. Device locking mechanisms at the
time were poorly developed. All three systems enabled PIN locking, and Android
and iOS also allowed for pattern drawing, but as [155] shows, these methods are
easily breakable. It is good to see how we moved forward, though definitely more
could be done.
OS Security
Android provides a crypto API to applications (via the javax.crypto package) and
offers encryption of block devices (via Linux’s dm-crypt layer). Crypto APIs for
more advanced privacy-aware applications like Off-The-Record (OTR) chat are pro-
vided by third-party libraries. Firefox OS never developed a crypto module, and
relied on third-party implementations (the official email app on Firefox OS used
node.js libraries). Given the uncertain nature of security audits for third-party li-
braries, support for a system crypto-package in Firefox OS seemed indispensable.
Furthermore, Firefox OS lacked support for disk encryption. It is dangerous to let
developers provide their own cryptographic tools, because the vulnerabilities of such
tools mostly lie in the implementation details, as was well described by Schneier in
his essay Security Pitfalls in Cryptography [139]
Privilege Escalation Containment
Several measures have been introduced in iOS and Android to dissuade attackers
from employing advanced programming techniques such as Return Oriented Pro-
gramming (ROP). These include Android and iOS’ support for full Address Space
Layout Randomization (ASLR) and a No-eXecute (NX) bit for preventing code exe-
cution on the program stack and heap. Firefox OS was conspicuous in not supporting
either.
“On Improving Privacy and Security Through User-Informed Design.”
3.1 Security Evaluation of Firefox OS 25/114
Permission System
As Android inherits from the Linux Kernel, it makes use of all traditional security
mechanisms such as process isolation and file permissions. In initial versions, the
user had to decide whether to grant the requested permissions. The user had to
agree to the whole list, or the app would not be installed at all. In iOS, all apps
were treated equally, having unlimited access to all resources and not giving any
warning on what they were using. However, there were cases when the user had to
give consent (e.g. for the address book or photos).
Both Android and iOS approaches were subject to attacks, and it has been
shown that the models used are not secure [65]. Google and Apple are slowly
moving away from their initial design, asking the user on first use to consent to
certain permissions. FxOS took an approach in between Android (where the device
owner is burdened with all decisions) and iOS (where users have no say at all). The
method of educating users was visible in the permission model. The permissions
that were non-intuitive to the user (but had privacy impact), or were simple to
understand, were presented to the users. To make things even better, a rationale
was provided to make sure that the decision was made consciously. We hoped to
add an expert mode, where the user could gain control over all of the APIs if they
felt they knew they were advanced enough.
App Vetting and Distribution
Android in Google Play, the official Market place, uses a service called the Bouncer.
It runs every uploaded application in the Google cloud, simulating the execution of
the app on an Android Phone and checking the app for malware, spyware, Trojans
and suspicious behavior. Since version 4.2, Android comes with a malware scanner,
but it has a very low success rate [161]. The app-vetting process in iOS is quite
secretive. Supposedly, the process contains several stages where the software is
tested for vulnerabilities, malicious behavior, instability and the use of unauthorized
protocols.
Neither iOS nor Android app-vetting processes proved to be secure [159]. To
avoid the problems, FxOS implemented a dual process. Additionally to avoid some
well known issues, developers could not use sensitive JavaScript constructs (like
eval), HTML5 iframe usage was limited, the Content Security Policy disallowed
inline code, and the Same Origin Policy was enforced. HTML5 protections are
further described in Section 3.1.2. The detailed analysis of the Market Place security
can be found in [43].
The app-vetting and distribution process was clear, caliming to filter out the
malicious code. Mozilla combined human reviews, to control the rightful use of the
API according to the description in the Manifest file, and usage of automated tools.
We appreciate that all of the documents and guidelines for the app-reviewing process
was publicly available so that all developers can access them. This always brings the
open-source potential to increase the overall quality of the reviewed apps compared
“On Improving Privacy and Security Through User-Informed Design.”
26/114 Chapter 3 Comparing and Evaluating Systems
to the competitive, shaded approach. Additional advantage was low entry barrier for
developers: there was no registration fee. From a user perspective, no requirements
on sign-up and account creation to download software or use the phones was a great
idea.
Element that positively distinguished Firefox from other platforms was its
higher openness. Unlike any other system (Android included) anyone could sub-
mit a solution or add functionality to Firefox OS, as long as the extension passes
the review process. In addition, the open-source nature of the code gives a good
opportunity to spot and identify the bugs.
Sandboxing - Scope for Security Extensions
The sandboxing mechanism on FxOS was somehow similar to what Android and
iOS proposed. Because it was a web-based OS, additional care had to be taken to
prevent typical HTML5 attacks, as well as escapes from the sandbox. Sandboxing
content that was not stored on the device remained an unresolved problem remained.
Sandboxing left a lot space for improvement. The system or b2g process in
FxOS violated the principle of least privilege, and the content processes had access
to all system calls to the kernel. The over-privileged b2g process, if compromised,
offered full device control to the attacker. Content processes (in which apps are
instantiated) did not need a full set of Linux system calls available to them. Even-
tually, after the publication has been made Seccomp was added.
3.1.4 Web-Based Operating System Security Issues
With the Web, more attacks are possible: not only the traditional buffer or stack
overflows and other zero days, but also Phishing [71], cache poisoning, stealing in-
formation from storage and global variables [165], mutation-based XSS [68], and
traditional cross-site scripting [29]. With sandboxing being the basic mechanism of
protection, traditional techniques of intrusion detection through system-wide mem-
ory scanning are not really possible.
Mozilla with FirefoxOS had good mechanisms in mind to protect them from the
bugs introduced in the customization process. They enforced a Partner Certification
Process in which any vendor change had to be approved by Mozilla in order to get
the Mozilla Technology logo. However, it was a two-step process with manual code
inspection, and thus it did not scale well, unfortunately. They also did not avoid
the second big issue, before abandoning the project altogether: having many devices
with very outdated software. This was mostly due to the delegation of the system
updates to the OEMs, as we have predicted in our research.
An interesting attack vector for WMOSes comes through the built-in adver-
tisements. Let us assume a malicious, free app with code added for advertisement
support that an unaware user installs on their device. With the granted permissions,
the attacker connects to the networks and stays idle until another collaborating app
is installed. The two apps coordinate and a request for installation of extra com-
“On Improving Privacy and Security Through User-Informed Design.”
3.1 Security Evaluation of Firefox OS 27/114
Threat Mitigation Android iOS FxOS
Data Inception SSL support yes yes yes
VPN support yes yes no
APIs use SSL yes yes no
OTR support no no no
Malicious App least privileges
permission model
yes yes yes
Market Place
malware preven-
tion
yes yes yes
third-party app
stores
no yes planned
App permission
restrictions
yes yes yes
App removal yes yes no
Safe browsing no yes no
Physical Access device lock no yes PIN only
bootloader OS
replacement
prevention
yes yes yes
root access no no settings
Encryption API yes yes no
Credential Stor-
age
yes yes no
Remote compro-
mise
exploit mitigation yes yes no
sandboxing yes yes yes
timely update
mechanism
yes yes 6 month
System integrity app signing yes yes yes
update signing yes yes yes
Figure 3.2: Comparison of security features on Android, iOS and FxOS
“On Improving Privacy and Security Through User-Informed Design.”
28/114 Chapter 3 Comparing and Evaluating Systems
ponents is displayed. The user agrees, enabling code injection and, for instance,
SMS subscription to premium-rate services. Even with solutions that try to prevent
interactive advertisements or limit usage of JavaScript in the manifest, there are
projects that help generate ads in an iframe.
The WMOSes suffer from the web-based attacks, and then come the classic
cellular ones. Mulliner et. al. showed how to use malicious text messages [107],
and Borgaonkar [30] used the Unstructured Supplementary Service Data (USSD)
to attack feature phones and smartphones remotely. The severity of such attacks
varies depending on the incentive of the attacker; it can be a simple denial of service,
man-in-the-middle and communication interception [62], or handset damage.
3.2 Building a Model for Security Evaluation
As the devices have been developed, the biggest changes were in the form factor,
connectivity and building in more sensors. These improvements changed the attack
vectors — compared to PCs, laptops were much more mobile, while smartphones
became much more connected than laptops. The question is: How can we predict
what dangers lay ahead of us when a new device type is introduced, rather than
just observing the attacks post-factum? This section is devoted to a model we built
based on the experiences from the past. We look at the attacks that have been
performed on laptops and mobile devices, how they shifted with the change in the
market, and finally we propose a model.
3.2.1 Human-assisted Attacks.
One of the most powerful attacks is theft. Assets include the handset itself and the
subscriptions connected to the mobile device, such as mobile network and streaming
services. However, the incentive to get physical possession of the device is much
higher; data on it may be much more valuable. With a theft, not only is the
confidentiality of Private Information broken, but also its availability. Moreover, we
tend to store passwords on our mobile devices, which means that the physical access
gives the attacker access to all of the victim’s accounts.
Leakage of data occurred in several cases when a device was either shared or
improperly handled. A famous example is the Edison Chen photo scandal, where
a technician stole sensitive photos while fixing Edison’s laptop [166]. It is not un-
heard of that poorly handled second-hand storage devices are also a good source for
information leakage [58].
Phishing always evolved into the new class of devices and services. Malware and
websites are developed for popular laptop and smartphone platforms. Identity-theft
malware is particularly focused on stealing bank-account, credit-card or personal-
identification numbers so the attacker can cause social and financial damage.
“On Improving Privacy and Security Through User-Informed Design.”
3.2 Building a Model for Security Evaluation 29/114
3.2.2 Remotely-controlled Attacks
Both smartphones and laptops extensively use the Internet, but how they are con-
nected is radically different. Laptops use wired Ethernet or Wi-Fi. Thus, ARP
spoofing can redirect all traffic into the attacker’s machine on the same network,
which could be used for eavesdropping clear data or to perform Man-in-the-Middle
(MitM) attacks on encrypted data. Flooding against the network can affect avail-
ability of the network. If some nodes of the network, like gateways or DNS servers
are under Denial-of-Service (DoS) attack, it will also affect availability.
Mobile networks, on the other hand, have a different structure compared to
the Internet. While no single authority is monitoring the entire Internet, cellular
networks are structured in a centralized way and controlled by the network operators.
This makes traditional Ethernet attacks impossible. However, because the control
plane is accessible, there are DoS attacks on wireless channels. The way that the
network is being accessed on the two devices — a large number of short connections
on a smartphone versus scarcer, longer, heavier loads for laptops — also influences
the way that a mobile network was designed and thus its weakest points [5,111].
Cellular network equipment such as a femtocell is a popular target among
attackers [61][156]. Because it provides cellular coverage and connectivity to the
cellular core network via the Internet, this target is much more useful than the
regular Ethernet. The newer generation network employs newer security measures,
and attackers can jam it, thus forcing a victim to use an older generation network,
which has more attack vectors.
Next come the attacks that were introduced with sensors and the delegation of
tasks offthe device. Sensors in smartphones and wearable devices carry information
about users that they are not aware of. [33,103,105]. “The Fappening” (leakage of
private photo of celebrities) started from hacking cloud storage linked to them, in-
cluding iCloud. [81] Wearable and IoT devices have tight constraints on the size and
power-consumption requirements, which makes it important for as much processing
as possible to be done externally, and leaves simple tasks like command relay and
sensing on device [149]. Although users did not have to care about location privacy
on laptops, because the precision was not high enough and the device was not mo-
bile enough, smartphones changed this game. They have GPS and network-based
location capabilities, and the mobile operating system exposes the location to appli-
cations via unified APIs. Location-aware apps can share the location via Internet,
or store location information with a time-stamp inside EXIF data of pictures taken
by smartphone. Location could be identified by the cellular network, even without
using GPS. Cells in the mobile network are identified by cell ID, which corresponds
to the approximate location of the user inside the cell. Mapping cell ID to the
location could be done by using services like OpenCellID. Misimplementations in
mobile-network standards allowed external attackers to track users of GSM [88] and
the LTE network.
A new platform encourages new types of malware. Laptops share the same
operating system as desktops, which also means that traditional PC malware runs
“On Improving Privacy and Security Through User-Informed Design.”
30/114 Chapter 3 Comparing and Evaluating Systems
on laptops, too. Smartphones, on the other hand, always include a mobile network
connection and they feature sensors that know about the operating environment
and how the user is interacting with the device. Unlike with previous generations of
devices, mobile phone data is accessed through a uniform API provided by the mobile
OS. Malware could be installed manually by a careless user or automatically using
vulnerabilities in the devices themselves. Any connectivity methods, including text
messages, websites or Bluetooth, could be used to transport the malware. Malware
can steal user’s data (such as photos, messages and credentials) outside of the device,
or infiltrate additional malware to gain more control of the device or prevent the
user from using the device.
To summarize: the more features and sensors, the more exciting tools and the
more attack vectors. How do they change with the move from laptops to mobile?
3.2.3 Shift from Laptop to Mobile Devices
The term “smartphone” was not widely used until Apple released iPhone in 2007.
The iPhone used network resources more actively than previous phones, contained
various sensors, and opened application development to the general public. The
smartphone boom encouraged the creation of other smartphone platforms like An-
droid and Windows Phone, accelerated development of faster mobile-processor and
wireless-communication standards like 4G LTE and IEEE 802.11n/ac, and intro-
duced new issues specific to the smartphone.
Network as a Portable Resource
The mobile network evolved from a voice-centric to a data-centric network. The
initial wireless LAN was slow and available in limited places. However, after the
introduction of smartphones and public hotspots, small cells increased the avail-
ability of mobile networks, and increased user’s expectations of mobile networks.
Although network-hungry applications were introduced to smartphone platforms,
older data-network infrastructure was not optimized, causing signaling storms to
disrupt network services.
Smartphone manufacturers increased battery capacity and reduced the power
consumption of mobile processors. Because battery power density is limited, most
power-saving technologies are implemented on the silicon side. Laptop processors
included various ways to save power: a HLT instruction to temporary turn CPU into
low-power state, dynamic clock-frequency scaling, and power control based on work-
load. Smartphone processors are more power-constrained; some processors support
dynamic power control of multi-core CPUs. While multi-core CPUs on laptops are
using homogeneous cores, ARM’s big.LITTLE enabled heterogeneous multiprocess-
ing on smartphones. Because the high-performance mobile processor also consumes
higher power, it is only enabled on higher workload, and low-performance processors
are used on lower workload. Battery depletion attacks become more important as
devices are more power-constrained.
“On Improving Privacy and Security Through User-Informed Design.”
3.2 Building a Model for Security Evaluation 31/114
Location Tracking
Smartphones are designed to be “always connected”, and capable of locating the user
precisely. Positioning information could be used to organize photographs by place,
get recommendations of local shops based on location, etc. Unintended disclosure
of location information may cause physical harm to the victim or their property.
When you upload a photo tagged at a vacation resort, it tells a thief that you are
not at home. Collecting the repeated location information can deduce user’s habits
and may help in planning the crime. Compared to a traditional IP-based location
service, which only provided coarse location information, GNSS on smartphones
provided very precise location information.
Sensor-rich Environment
Laptops and early mobile phones are not usually equipped with sensors, and operating-
system support for sensors is always limited. The original iPhone included ac-
celerometers, and proximity and ambient sensors. Google mandated an accelerom-
eter, compass and GPS from the earliest version of Android. [63] Modern smart-
phones employ various sensors: accelerometer, gyroscope, light/proximity, digital
compass/magnetometer, temperature, barometer, heartbeat, etc. Among these sen-
sors, accelerometer and gyroscope are widely available on smartphones compared
to the other sensors, and sensors react quickly to the human interaction with the
device. These properties made accelerometers and gyroscopes popular sensors to
misuse.
Raw sensor values could be used in both benign and malicious ways. Ac-
celerometers and gyroscopes are not only used for detecting the motion of device, but
also for detecting user interactions with the device. Cai et al. presented TouchLog-
ger [33], which uses accelerometer values to read user input on the touchscreen-
based numeric keypad. Tapprints by Miluzzo et al. [105] improved the accuracy of
detection by using both the gyroscope and the accelerometer, and was capable of
detecting input on a virtual QWERTY keyboard. Michalevsky et al. [103]showed
how to abuse the gyroscope as a microphone. Direct data injection into the sensor
is difficult on smartphones though, because they are packaged inside the device.
Voice Assistant
Voice assistants like Apple’s Siri, Google’s Google Now and Microsoft’s Cortana
allow users to interact with a smartphone by voice command. Because the voice
assistant is designed to access the whole smartphone, its activation can be done se-
curely. Yet new versions of Google Now and Apple Siri included an always-on option.
This is not always followed by speaker authentication. Apple’s Siri lacked speaker
recognition until iOS 9 [17]. Anything saying “Hey Siri” can activate the voice as-
sistant. Xbox One also enabled a voice assistant even when the device is turned off;
an advertisement saying “Xbox On” turned on numerous Xboxes unknowingly.
“On Improving Privacy and Security Through User-Informed Design.”
32/114 Chapter 3 Comparing and Evaluating Systems
Because the voice assistant can recite user data like e-mails, notes and messages,
the attacker can trigger the command and record the response to further analyze
this. Diao et al. [44] presented an attack on Google Voice Search without special
permission. The attack will play voice a command via TTS, and extract the data
from the voice assistant. Because voice is transmitted through the air, it detects
the current context by obtaining environmental information like light, time, etc. to
avoid being caught. Kasmi et al. [80] improved this attack by injecting a voice signal
via EMI through the ear-microphone.
3.2.4 Model: How to Evaluate a New Device?
Personal computers have been around since the 1980s, laptops since the 1990s and
smartphones since the mid 2000s. This has been shifting the attack vectors as the
technology is built iteratively. Every new device reuses or improves upon existing
solutions, which means it will inherit most of the faults of its predecessors. However,
it is probably pointless to hope a universally secure stack can be built. The best
strategy may be to aim at one where each layer isolates potential damage from
affecting the higher parts.
The model shown in Figure 3.33is a cycle of six steps, where we (0) define the
assets a device can hold, (1) then identify the threats, (2) seek vulnerabilities, and
(3) discuss how these can be exploited in various attacks. To efficiently design the
(6) defenses and countermeasures we first prioritize the findings by (5) analyzing
the risks, or how probable it is that a certain attack will be performed. Returning
to Figure 1.14, such a cycle should be performed on each layer of the security stack.
First is the Network Infrastructure that allows any device to stay connected. All
physical elements that comprise a device, together with the drivers that allow for
communication with the upper layers, are subject to inspection on the Hardware
Layer. Finally the Operating System itself, the governor of the behavior of the
device and access controller to all the data, comes to play. On top of that, we have
applications. Either pre-installed or coming from third parties, they pose significant
threats to the security of the whole ecosystem. What is above is even more sensitive:
the user data generated by the owner of the device, i.e. all of their Personally
Identifiable Information(PII).
On a high level, the evaluation is similar to that suggested by the common
criteria [151], as shown in Figure 3.4. We combine the definition of the assets on
each layer of the stack, analysis of incentives in previously seen solutions, along with
new attack vectors - like sensory information when moving from feature phones to
smartphones. If something has already been analyzed and evaluated, we use the
knowledge we have about the threat models and analyze whether the same vulner-
abilities and attacks apply to the new device. We also revisit the risk assessment,
because elements that might have been crucial for previous generations are not nec-
essarily as important anymore. Moreover, additional threat analysis is performed
3Figure comes from our original work [120]
4Figure comes from our original work [120]
“On Improving Privacy and Security Through User-Informed Design.”
3.2 Building a Model for Security Evaluation 33/114
Figure 3.3: Model for security evaluation of a new device. We base the analy-
sis on similarities with previously introduced devices and the assessment spiral of
identifying assets, threats, vulnerabilities, attacks, risks, defenses and mitigations
on various levels of the stack
Figure 3.4: The cycle of creating a security enhanced solution.
“On Improving Privacy and Security Through User-Informed Design.”
34/114 Chapter 3 Comparing and Evaluating Systems
because the same assets can be subject to new threats. If there are new resources,
the assessment is performed from scratch and based mostly on knowledge and ’gut
feeling’. Finally, with all this input we build a report that proposes defense mech-
anisms for a new device before it is even subject to any attacks. Compared to the
Common Criteria model, such an approach puts more emphasis on building defenses
and predicting attacks in order to mitigate them, rather than creating a risk analysis.
“On Improving Privacy and Security Through User-Informed Design.”
35/114
Chapter 4
User-Informed Design (UID) in Privacy and
Security
We have shown that no system can be said to be secure under all circumstances. The
only thing we can assume is that computers and mobile devices are insecure. We have
built a model to evaluate new, upcoming devices and predict how to best secure them
before they become subject to attack. But, as has been said by many researchers
and repeated multiple times in this work: even with the most secure and bulletproof
systems, they have to be created in a way that would be intuitive and usable for
the people. Many security failures are actually design errors: bad engineers build
systems that ordinary people can not use. It is important to remember that no
system exists in a vacuum with no one touching it. Most probably, at some point,
someone will have to operate the system. And that someone will, most probably,
do the least predictable, silly and insecure thing the designer would never have
imagined.
That is why we have decided to focus on developing a new approach to pri-
vacy. Instead of building tools that aim at the top-security, bulletproof solutions,
we focused on asking the basic question: What is the difference between the way
that the mobile devices work and the way people think they work? We looked at
the literature, and we then designed a process, that we called the User-Informed
Design (UID) 1.Thewayitdifferentiates from user-centered design is that instead
of evaluating the tools after we build them, giving the people only a chance to say
“Yes” or “No”, we actually ask them what they want and need. The prototype we
created gave them many more options and solutions than they needed. In fact, it
was quite inconvenient to use. That way they could judge various solutions and
choose which ones they liked the best. With that, we implemented the best-rated
ideas as proof-of-concept demonstrators and verified how they worked. In this chap-
ter we describe the first phase: the Literature Research and the three stage UID.
In the next chapter, we move to a detailed description of the three features that we
have later developed, inspired by the creation of this process:Adjustable Location
Accuracy, Permission Control and a solution for business: a Context-Aware Autho-
rization mechanism. Finally in Chapter 6, we will talk about the lessons we learned
from the user studies we have conducted on various stages of the development of the
Privacy Dashboard and the specific features.
1In the original paper we referred to it as user-centered design (UCD). However after reconsidering,
we realized that the User-Informed Design much better describes the process
“On Improving Privacy and Security Through User-Informed Design.”
36/114 Chapter 4 User-Informed Design (UID) in Privacy and Security
4.1 Literature Research
Hang et. al. [67] discuss mobile device-sharing behaviors and the concerns people
have. Similar works by Jedrzejczyk et. al. [75], Wentz et al. [163], Shklovski et.
al. [141], and Lin et. al. [92] gave us a basis for understanding user perception
of security and privacy. Additionally, a methodology for practical experiments to
replicate real perceptions of privacy risk and capture the effects of actual information
disclosure decisions was suggested by Keith et. al. in [82]. Jung et. al conducted
a survey on different mobile OS users and found an “expectation gap” towards
transparency and the actual usage of their agreed permissions [77]. Their findings
expose the failure of the current methods to make informed privacy decisions.
Fu and Lindqvist [56] looked into the way people understand Android’s location
permissions and how users’ behavior changes after we educate them on a permission’s
true meaning. In [153], Tsai et al. found that people feel the risks of using location-
sharing technologies outweigh the advantages. Some studies by Wilson et. al. [167]
contradicted a common belief that the mere possibility of setting privacy profiles
would make people more privacy-conscious. A method for quantifying the effects of
sharing data was developed by Benisch [25]. Other works on the factors impacting
users’ decisions to give away information can be found in [116], [83], [117] or [152].
All of the above gave us an understanding of the mindset of the users, and led
us to design a system with the choice of per-app settings with various adjustment
mechanisms.
Practical solutions that provide overall control of a device still remain rather
rare in the field. In [27], the authors describe a “Privacy Panel App” in which
users can define policies based on a computed rating of privacy impact. Works like
[125], [124], [87], or [66] present solutions to particular issues, such as encryption
or location blurring, but do not support a holistic approach, and would be hard to
combine into one tool.
From iOS8, Apple has taken on a big shift towards preserving user privacy,
at least on the surface: famously in the battle between the company and the U.S.
government where Apple did not give up the encryption keys2, and various speeches
given by the CEO, Tim Cook. It is a big shift from being extremely collaborative in
the past3, to hiring external experts to help fix security breaches [74]. Additionally
more controls have been added to the privacy tab settings. However, the control
given to the user is very binary and not well explained. It still requires a certain
level of proficiency to set up properly. Android has also been following these pri-
vacy trends. In version 4.3. we have seen a brief experiment with App Ops [133],
which sadly has quickly disappeared from the system; it provided information about
what the phone did and what resources it accessed. Google decided to change the
long-criticized permission model allowing for installation of apps even with partial
permissions and withdrawing them after installation.
2https://www.apple.com/privacy/government-information-requests/
3http://www.forbes.com/sites/erikkain/2013/12/30/the-nsa-reportedly-has-total-access-to-your-iphone/
“On Improving Privacy and Security Through User-Informed Design.”
4.2 Presenting User-Informed Design (UID) of Privacy Tools 37/114
Android has been a target system for some permission-control frameworks.
CRePE by Conti et. al. [39] is a context–related policy enforcement for Android,
and can be used to explain fine-grained policies. Examples of leveraging the role-
based access control include DR BACA [131] and CtRBAC [169]. In MockDroid [26]
the authors modify the OS in such a way that the user can “mock” the applications’
access to the resources. Similar approaches have been proposed in AppFence [69].
Zhou et al. propose a flexible, dynamic control of fine–grained permissions in dif-
ferent scenarios with TISSA [171]. Ni et. al. created a role-based access-control
mechanism that creates policies from natural language [110].
The current research trend is mostly focused on securing and hiding informa-
tion to meet the privacy requirements. There is a debate whether to introduce more
control in the interface [115][13]. Decision-making for important private data based
on a cumbersome method could result into a complete rejection from the subscribers
[84]. Additionally, there are hurdles to overcome that are caused by a lack of under-
standing and a lack of transparency regarding consequences [132]. Misunderstanding
and lack of knowledge are primary reasons why users consent[132][84]. Though sub-
scribers are compelled to agree to access in order to use an app, the consequences are
kept hidden from them [158]. Patil and Lai conducted their survey and described the
imbalance between privacy awareness and burdensome configuration control [115].
Acquisti and Grossklags probed the common practice to make the user respon-
sible for privacy-sensitive decisions [11]. It has been shown that people do not have
sufficient knowledge to make informed decisions. In fact, they will often choose
wrong settings due to misperceived consequences. Finally, Kelly et. al demon-
strated that users’ behavior can be persuaded to align with good privacy practices,
and becomes more important than enticing services [84].
4.2 Presenting User-Informed Design (UID) of Privacy
Tools
In this section, we present work previously published in a paper “Because we care:
Privacy Dashboard on Firefox OS” discussed at the Web 2.0 Security and Privacy
workshop, in the IEEE Security and Privacy Symposium in 20154.Wedescribe
the process of UID and how it helped to build the Privacy Dashboard. In our
initial work we have used the term User-centric privacy, to distinguish from User
Centered Design(UCD) and Privacy-by-Design (PbD). We felt, however, that this
phrase was not capturing the main difference of our approach. UCP and PbD focus
on developing systems with the user in mind, which means that after implementing
solutions, they introduce user studies as a form of usability evaluation. Meanwhile,
we took advantage of the experiences in the field of Human Computer Interaction
(HCI) and a process called User-centered design (UCD). It is a software methodology
for developers and designers to help them make applications that meet the needs
4All graphics in this section come from the original paper [122]c
!2011 IEEE
“On Improving Privacy and Security Through User-Informed Design.”
38/114 Chapter 4 User-Informed Design (UID) in Privacy and Security
Figure 4.1: User-Informed Design (UID) process
of users by employing three main iterative steps: exploration, design research, and
evaluation.
Looking at the previous research described in Section 2.1, we were sure that
people have an expectation of privacy preservation on smartphones, but do not
know how to execute it. It was not clear what that expectation was or how they
understood mobile devices. Thus, we have created the UID process by adopting
UCD into Privacy and Security. As shown in Figure 4.1, UID comprises several
iterations of user studies and technical work, starting with a user study. During this
phase, we evaluate the gap between user expectation and the system, we observe
how people use existing solutions, and how their understanding differs from reality.
We create a set of general design guidelines for the planned prototype, which is then
implemented without a strong focus on the technical details but rather on delivering
multiple solutions to the same problem. In another study we ask users to evaluate
various approaches and see which one fits users’ needs best. Finally, we proceed to
implement a more precise, and technically detailed prototype, which can be assessed
in Usability studies and turned into a product.
4.2.1 User-Centered Design (UCD) and User-Informed Design (UID)
The immediate question emerges: How does the proposed design model compare
to the previously developed UCD? The International Standard 13470 [73] gives the
guiding principles for human-centered design, providing requirements and recom-
mendations along with the activities that should be performed throughout the life-
“On Improving Privacy and Security Through User-Informed Design.”
4.2 Presenting User-Informed Design (UID) of Privacy Tools 39/114
Figure 4.2: User-Centered Design (UCD) according to the ISO 13470 standard.
cycle of a product. However, the document itself does not clearly specify the exact
methods for each phase. The main idea behind UCD and UID is very similar:
understand the users, the task and the environment we will be working with and
continue to refine the model based on the user-centered evaluation. In both cases
users are involved in some way throughout the whole process. Figure 4.2 shows the
phases recommended in ISO 13470 that should comprise a full UCD cycle. First,
the designer is responsible for identifying the user group, the way they will be using
the product, and the environment in which the product will be used. Next, the
requirements are defined both on the business and human level. From there, one
can move to creating the design solutions, which can be done in stages, from rough
ideas to a final implementation. The final stage of the process is the evaluation
through usability testing with actual users. As the exact execution of each stage is
not precisely specified, there are many ways the UCD can be realized; for instance,
it can be combined with a waterfall or agile approach. What emerges from the
ISO-standard documents and from the description of the UID in previous sections is
that while UCD is more process-oriented, UID is more implementation-oriented. In
many ways, UID could be seen as a perfect complement of UCD. The first step is to
identify the target group. In UCD the second phase, where the needs are identified
and goals defined, is not strictly required to work with a group of potential users.
In UID this is the first phase, and we always work with our future customers. The
third phase of UCD can be implemented as steps 2 and 3 of UID. Finally phase 3
of UCD is what we strongly recommend for UID anyway. Thus, we believe that by
better specifying the details of UCD with the introduction of UID, we can create
a better and more usable model that would look similar to the Figure 4.3.The
detailed description of the three steps of UID follows in the next sections.
“On Improving Privacy and Security Through User-Informed Design.”
40/114 Chapter 4 User-Informed Design (UID) in Privacy and Security
Figure 4.3: Proposal of combining UCD and UID to enhance both
4.2.2 First Step
No theoretical design of a methodology can be claimed useful without a practical
implementation. Thus, we have employed our solution and conducted a three-step
study that resulted in the creation of a Privacy Dashboard for FirefoxOS. We divided
the introductory study into two parts with a summary session at the end, during
which we surveyed 24 users in a lab, who were recruited over social-media channels.
The participants were selected to resemble everyday experts, i.e. people between
20 and 40 years old, who had an average knowledge about mobile devices and data
privacy. We eliminated people who had high technical knowledge but made sure
that everyone had used a smartphone for at least a year prior to the study and used
a variety of services like email or instant messaging on a regular basis. In this first
phase we wanted to identify the fears and motivation barriers, and understand the
mental model of the participants, as shown in Figure 4.4. Next, we asked the users
to talk to product and engineering experts and tell them about their understanding
of privacy and security in particular. The interviewers worked for local Mozilla
offices and were instructed to conduct an interview that would be a discussion on
security and privacy. They were given probing questions to start with (how often do
you share your phone with other people? do you have brands that you trust more
than others? what does trust mean to you? have you heard about personalized
“On Improving Privacy and Security Through User-Informed Design.”
4.2 Presenting User-Informed Design (UID) of Privacy Tools 41/114
“Everyday experts” focus group
Identification of user needs, concerns and fears about personal data
and related privacy issues on mobile phones and related services
Participatory feature design session
-Create a shared understanding between everyday and product experts
-Prioritize features that really matter according to user needs
Design guidelines
Define design guidelines for the development process
based on the results of the introductory study
Figure 4.4: The concept of the introductory study and the key results of each step
for the User-centric Design (UCD) process
advertisement?) , but mostly the interview was supposed to give space for the
participants to express their opinions in a more open manner. These two phases
informed the most important step, thus fulfilling the first step of UID. We found
that although users feel uncomfortable sharing their private data, they were forced to
accept the non-transparent privacy policies. Opting out from a service or otherwise
being unable to take actions was described as overwhelming and impossible. The
three needs participants expressed were to feel in control, have ease of use, and be
able to take actions. Today, we understand that the choice is between sharing data
on the Internet and being able to use services, or preserving privacy and accepting
the position of a web-outsider. It is a very rough decision to make, and the three
main complaints users raised were:
1. The apps cannot be trusted because it is not clear why they need access to
certain data, and what will be the consequences of the disclosure. Some of the
ways they talked about it was: “I am worried when I need to share data, that
it’s absolutely irrelevant to a service that I want to use”, “I feel uncomfort-
able when services use my data for creating personalized advertisements”, and
“Why does a service need to check my email contacts at first-time use?”.
2. The app providers could not be trusted due to bad media coverage or bad
image. The statements we heard were, for instance: “I am worried when brands
whose services I use get negative press in the media.”, and “[This company] is
known for bad habits against their employees, I assume they also do not take
user-data best practices seriously”
3. Lack of comprehensive understanding of how the app or a solution works and
“On Improving Privacy and Security Through User-Informed Design.”
42/114 Chapter 4 User-Informed Design (UID) in Privacy and Security
how to use it. They did not feel that they can even approach a system because
they do not have the appropriate education.
While hearing these complaints, we were surprised to see that the motivation to
search for alternative services was very low, and users expressed resignation in the
form of “Eventually, there is no chance not to use this service, so I disclose all the
data knowing that this might be wrong”. This is particularly worrying, because
it shows a certain trend of giving up and accepting the current state, where the
Internet will not work without Private Information disclosure. People just convince
themselves that they have nothing to hide and their data is not important enough
to be protected, and they would rather have someone else deal with the technical
issues than learn anything on their own. In the end, quite understandably, they will
sacrifice some usability for privacy, but not to a point where they cannot access the
desired service at all.
4.2.3 Second Step
It is hard, if it is even possible at all, to build a solution that will cater to the needs
of basic users and technologists alike. While the users have higher expectation of
usability, the technologists will definitely look for high flexibility and bulletproof
algorithms. While not entirely contrary, these goals are hard to achieve in one tool.
For us, the crucial part was development based on the UID, so we focused on the
basic users and their needs first. We defined this group as people with little-to-no
technical knowledge, sometimes even first-time smartphone users, who believe that
the technology is either scary or quite the opposite: very safe and protective. What
was common was that users currently feel they do not have enough knowledge and
education on good privacy threats, yet they want to be in control of what happens
with their data. Our report showed they want to:
•Be sure that their personal information is safe and secure, especially when a
device is lost or stolen
•Know that their data will not be inadvertently exposed to unauthorized people
•Have control, but also need some help in setting up and managing their devices
and data
While maintaining the high technical quality of our solution, we had users and
their needs in mind. Thus we did not want to let striving for a better solution
compromise the usability. Moreover, as we were designing a framework for a very
specific market (low-end smartphones), several factors needed to be included in the
designs:
•We will be working with devices that have low computational power.
•The tool has to be as protective as possible, within the legal limits in a variety
of countries.
“On Improving Privacy and Security Through User-Informed Design.”
4.2 Presenting User-Informed Design (UID) of Privacy Tools 43/114
•To get the devices certified and available on the popular market, the tool will
not protect devices from legal interception or other governmental surveillance
methods.
•The main purpose is not only to empower the users, but also to make them
aware of the threats, so they can adjust their everyday practices.
•The tool has to ensure a high-quality technical solution: half-baked privacy is
worse than no privacy.
Overview of the Features
As a result of the initial study, the features that have been identified as most impor-
tant to the everyday experts were connected to education on good privacy practices
and device control. One of the first things that showed up in our research was a
self-reported lack of understanding and knowledge of good privacy practices. To
tackle this issue, we created a set of screens that explain all the features introduced
in the Privacy Dashboard. Upon completing the Guided Tour,theuserseesthe
main panel and can explore the settings. The Guided Tour is available at any time
later for every feature separately or as a whole feature.
To address the issue of a lost device, we have created a few mockups. The
minimum set of features that our users requested included remotely locking the
device with a passphrase, tracking it, and wiping the data when there is no chance
of finding the lost device. It was interesting to see that people believed that their
privacy is more valuable than the information itself. That, of course, meant also
implementing a good backup solution that would provide flexibility over where the
data is stored.
Today, some of the issues that the participants identified are already being
resolved, however they felt that the offered tools are not good enough. For instance,
the Directive on Privacy and Electronic Communications [1] states that the location-
based services (LBS) must be permission-based. So if an app wants to use the user
location, it needs to request access to it, giving the user the opportunity to opt-out
or turn the service off. However, most current applications simply stop working
if they do not get the permission. Thus, we introduced the Adjustable Location
Accuracy. Similarly, the solutions for sharing a device between multiple users are
not sufficient in the opinion of the respondents.
Finally, people feel there is a lack of transparency — they have no clue how
much information is shared and when. Previous research showed that many of the
privacy decisions are context-dependent. For instance, Consolvo et. al. [38] found
that users disclose information about their location based on who was requesting
it, why they asked for it, and how detailed the information should be. Thus, we
created the Permission Control feature, where users get an overview of what each
app is doing and when, to provide a basis for an informed decision on allowing or
withdrawing access to certain resources.
“On Improving Privacy and Security Through User-Informed Design.”
44/114 Chapter 4 User-Informed Design (UID) in Privacy and Security
Implementation
We later decided to continue development of the Adjustable Location Accuracy
and Permission Control. Moreover, using the UID we also addressed the business
world’s problem with a Context-aware Access Control mechanism. All three will be
described in the next chapter. This section focuses on the elements of the Privacy
Dashboard that we developed on Firefox OS. This OS gave us the flexibility to
adjust and change the system according to our needs. It was also beneficial to test
our solution on a system that was not known to most participants of the study.
Guided Tour
The Guided Tour was mostly a communication challenge. We refer to the original
paper for the background information on how we concluded that the messages we
create have to be short, simple, carry emotional load, and give a sense of participa-
tion or gamification. We mainly based our research on work by Mellers et al. [102],
which covers the decision-making and psychology of judgment. We wanted to get
an idea of what good privacy practices are. All of that brought us to creating three
characters that guide the user through the tour. Each screen consists of a graphic
and text. In fact, there are two versions of the image: one that suggests a bad deci-
sion and one that is encouraging. The text is short and describes a feature, focusing
mostly on the reasons why it is important and the consequences of a given decision.
In the initial prototype, we let participants judge two versions of the Guided
Tour. One version was interactive, and allowed for adjusting the settings as they
went through it. The other version served only as explanation. During the evalua-
tion, users were definitely in favor of the static version of the Guided Tour, and we
ended with the design shown in Figure 4.5. To allow for delegation of the technical
tasks, we introduced the possibility of sharing the privacy settings with others and
downloading settings from trusted sources.
Remote Privacy Protection (RPP)
While conducting the first phase of the UID, one of the participants told us: “Trust
is something that is earned, not enforced. Why does everyone take my trust in
technology for granted?” Today, the ability to track a device after it has been lost
or stolen requires the user to sign up to some kind of a centralized database. This
raises very reasonable doubts about privacy. To follow the guidance the people gave
us, we created a solution that is account-less, where users can remotely control their
device through the Short Messaging System (SMS).
During the first run, a user is asked to set up a passphrase that serves as a
Master Key for the Remote Privacy Protection (RPP) required for logging into the
settings and identifying the owner. Next, they are asked to set up phrases that
will be used to activate the commands remotely through the SMS: locate, lock,
ring. The message format is always the same and consists of the code name “rpp”,
the command, and the passphrase that is up to 100 characters long and has to be
“On Improving Privacy and Security Through User-Informed Design.”
4.2 Presenting User-Informed Design (UID) of Privacy Tools 45/114
Figure 4.5: First two wireframes of the Guided Tour. Each panel consists of the
topic, a title and a short explanation. Instead of simply describing the feature, the
Guided Tour gives the consequences of the decision.
composed only from standard characters. The message sent is hashed so that, if it
was intercepted, the attacker would need to guess the passphrase and the commands
(which can be customized phrases). An event listener in the RPP checks all incoming
messages to verify if it is an activation SMS message and if so, starts the verification
process. If the message is correct, the device acts accordingly:
•Lock: the phone turns on the lock screen with a passphrase.
•Locate: the phone sends out the current geoposition and locks itself. It is still
responsive to the new locate commands.
•Ring: the phone locks itself and activates the ringer with the highest possible
volume.
Such a solution comes with several drawbacks that would need further investigation
if they were to be introduced into a full product. We discuss them, along with
possible solutions, in the full paper [122].
Adjustable Location Accuracy
Adjustable Location Accuracy was developed to allow for more fine-grained control
over sharing a position with third-party applications. As the network operator is
constrained by the limits of “legal interception”, and we are constrained by the
certification process of the network operator, we cannot prevent the phone from
revealing the location to the operator; the detailed technical explanation is given
in Section 5.1. From the initial study, we learned that there was a need for a
setting on a per-application basis, which allows users to disable the location, give
“On Improving Privacy and Security Through User-Informed Design.”
46/114 Chapter 4 User-Informed Design (UID) in Privacy and Security
the precise coordinates, choose a fake position that is returned to the app, or adjust
the granularity by a given number of kilometers.
Advanced Secondary-User Mode
Elements under protection of the Secondary-User Mode can be divided into three
groups: apps, data and resources. The applications installed on the phone carry
a lot of information about the owner of the device. It is sufficient to say that,
advertisement libraries used for targeted advertising send their potential clients not
only the location, gender, or age of the device’s user, but also the list of installed
applications [99]. For that reason, in the Advanced Secondary-User Mode (SUM),
users can decide which apps will not only be accessible, but also visible on the
phone. This means that after entering the Secondary-User Mode, the apps will
disappear from the screen and from the internal search engine. There is a list of
predefined apps that will always be removed, such as the Settings and the Privacy
Dashboard (because the person using our phone should not be allowed to change
any options). Upon entering the Secondary-User Mode, the data stored in each of
the defined elements is substituted with an empty list, as if no-one had ever used
the phone. Once the phone is handed back to the owner, the original information
is restored. The databases include Contacts, Call History, SMS History, E-mails,
Photos, Browser History. Lastly, there is a possibility of limiting the access to
the resources, such as Wi-Fi, Cellular Data, etc. The Secondary-User Mode covers
multiple use cases. Firstly, it may be used as a “guest mode” to hand over a device
for a short period of time. Secondly, it may be used as a permanent solution for
parental control: the parent would create a profile for the child and lock it down,
and can unlock the functions as the child grows. Lastly, when properly handled,
Secondary-User Mode could be a mobile equivalent of an “incognito mode” in a web
browser.
While Android and Apple both currently provide some version of parental
control and guest mode on the mobile device, it is very far from what we propose.
In the case of Apple’s “Guided Access” a secondary user has limited ability to
interact with the screen; they can still see what is on the phone and potentially
abuse the parts they can access. On Android, there is a guest mode but it still
assumes good actors and only separates some of the user data (not all of it). We
know that with powerful devices, it is possible to build a real incognito mode, where
different instances of the same operating system would be running on one device.
However, it is not in the interest of the handset manufacturers nor the governments
to have users buy fewer devices and keep their data on better-protected devices.
4.2.4 Third Step
Once the prototype from Phase 1 was ready, we could conduct a lab-based evaluation,
and verify which of the proposed ideas were most appealing to the users. Once again,
we invited a group of 26 people, although this time the distribution was even. We
“On Improving Privacy and Security Through User-Informed Design.”
4.2 Presenting User-Informed Design (UID) of Privacy Tools 47/114
had 38% males and 62% females of ages between 17 and 55, mostly college graduates
(38.5%) or still studying (30.8%). Almost one fifth had a higher-education degree.
Only one of the participants had never used a smartphone before. The study was
designed to answer three research questions:
•Does our Privacy Dashboard bridge the gap and meets users’ expectations?
•What are users’ learnability, performance and satisfaction using the privacy
features?
•Does the Guided Tour work? Does it actually have an impact on people’s
awareness? Do they accept it?
We started by recording the time each participant needed to go through the Guided
Tour. Next, we analyzed their attitude and behavior while configuring the privacy
settings. We verified the ease of learning and use of the privacy settings, along with
the people’s satisfaction of performing the task, global satisfaction connected to the
features, and we collected their comments. Each participant learned and performed
the tasks on an Alcatel One Touch phone with the FirefoxOS installed.
4.2.5 Methodology
Before proceeding, I shall describe the methodology in greater detail. For all of
our studies, we have used a university-supported platform for recruitment. Unless
otherwise stated we made sure that the Kruskal-Wallis omnibus test showed no
significant differences across age, gender and education. We made sure that all our
participants were not familiar with smartphones. The number of participants in
each study is specified in each case.
For Phase 1, we used a focus-group study to discuss the user expectations,
their opinions and fears. The study was conducted in Brazil because this was where
Firefox OS was widely introduced, and also where we could find a group of users
who were interested in technology but not yet familiar with smartphones.
In Phase 3, we used Card Sorting for judging various UIs and designs. To find
the ideal number of features and options, we used A/B testing. The rest was done
using self–reporting. Phase 3 was conducted entirely in the lab. When it came to
verification of certain features, we did a Moderated Remote Usability Study, where
users spent a week using the phone on their own. We could monitor and verify if
they actually used the phone, and we discarded the results from participants who
were not active.
Each participant learned and performed the tasks on an Alcatel One Touch
phone with the FirefoxOS installed. We obtained their permission to record all of
their activity, and they were aware that anything that happened on the phone would
be used for research purposes.
The biggest limitation of our studies was the sizes of the study groups. We had
trouble finding enough participants, and so building reliable large-scale and long-
lasting studies was hard. As part of the future work we would like to investigate
“On Improving Privacy and Security Through User-Informed Design.”
48/114 Chapter 4 User-Informed Design (UID) in Privacy and Security
better methods of recruitment and possibly re-run some of the studies. Moreover,
using better, more attractive phones could be an advantage. We found that the
devices were, in some cases, a barrier to the non-technical users.
“On Improving Privacy and Security Through User-Informed Design.”
49/114
Chapter 5
Development of User-Informed Features
As we developed the Privacy Dashboard through the process of UID, it became clear
that it is very interesting to apply the same technique to some of the most challenging
problems the participants named. Initially we decided to work on Location Privacy.
It has been a very attractive and hot topic over the past few years, however there
is an interesting phenomenon to be observed. The number of solutions proposed
in academia does not translate to ones implemented by the industry or adopted
by the users. It significantly declines as we move from labs to households. Once,
could argue that it is due to predominant belief that users ignore privacy settings so
there is no need for such offering. To verify, we applied the user-informed approach,
developed our solution with users, and discussed their expectations. This chapter
presents the technical implementation, while the next chapter includes conclusions
from our work.
The next feature we took care of was the Permission Control panel in the
Privacy Dashboard. We already know from previous research that the way Terms
and Conditions are presented today encourages users to ignore them and disclose
sensitive information unintentionally. However, crafting privacy recommendations
that fit everyone is not possible — the notion of privacy strongly depends on the
individuals. Through the process of UID, we found that the key is to educate and
empower users by giving them a tool that provides full transparency about what
is happening on their device. To improve the intuition and give some anchoring
points, we experimented with a Privacy Meter - an indicator of the privacy impact
of a given app.
Finally, the third tool related to the business world. Talking to the enterprise
customers, we recognized that the mechanisms designed to control access to the
resources on the devices were designed a while ago and did not really fit current
requirements on power or connectivity. The threat models changed, the use cases
changed, and so the policies should have changed too. The users expressed a need to
create more extensive, flexible, context-aware protection mechanisms. As a response,
we created a framework for policy-based authorization and a tool for a reliable
synthesis of custom policy-based authorization. We turned around the old device-
focused approach and faced the users, asking in what context and during what
events they use their devices. With that, we were able to abstract the framework
from particular devices and create tools across multiple platforms: from mobile
devices to the IoT.
“On Improving Privacy and Security Through User-Informed Design.”
50/114 Chapter 5 Development of User-Informed Features
Figure 5.1: Flow of the Extended Location Settings
5.1 Adjustable Location Accuracy
According to the data collected during the Data Privacy Day back in 2011 by Mi-
crosoft [96], over half of the people surveyed have concerns about sharing their
location data, and 49% say they would be much more comfortable doing it, if they
had a chance to actually control it properly. Yet neither Android nor Apple has
offered a solution that goes beyond an on/offtrigger. Academia has been looking
at the problem for a while, as we discussed in Chapter 2. Let us now focus on the
implementation we propose.
5.1.1 General Solution
We chose to develop the Adjustable Location Accuracy for FirefoxOS, because the
system was well understood and provided a standard W3C geolocation API that
gave us the opportunity to expand beyond Firefox phones. For that reason it was
important to change only the way that the app receives the geolocation data, not
the way it was calculated on the driver level or the way the signal is handled by
the GPS chip. The four options that the users chose in the first iteration of UID,
as described in Section 4.2.3 are presented in Figure 5.11. The user can choose
the typical settings of precise position or give no location at all, but also adjust it
through a blurring mechanism or hide it by providing fake coordinates. As a result
of the UID, we picked an interaction where initially a global setting is defined for the
whole system. Next, for certain applications, one can create an exception list with a
different adjustment. Users definitely preferred a slider for the choice of granularity
and self-defined input for fake position.
1Figure comes from the original paper [122]c
!2011 IEEE
“On Improving Privacy and Security Through User-Informed Design.”
5.1 Adjustable Location Accuracy 51/114
Figure 5.2: Fixed location setting. The user can either choose from the predefined
list of regions and major cities, or input their own coordinates. The location reported
to the app will always be set to the defined values.
Hiding the Location
Situations named by the participants that required a fake location ranged from
simply hiding their position from third parties to getting access to services they
would not be able to get to, in normal circumstances. There are two ways to
input new values: one is a predefined list of continents and major cities, where
the coordinates are set to the center of the city, the other is entering the GPS data
by hand. Both methods are shown in Figure 5.22.
Adjusting the Accuracy
Our analysis of the Market Place store applications showed that out of 300 apps
that request access to the location services, only 29% of them really needed precise
position, e.g. the navigation apps. The rest could easily work with approximate
values. For location adjustment, after obtaining the precise coordinates from the
available sensors, we used an obfuscation method to provide a level of uncertainty. To
change the granularity of the location data provided to the applications, we modified
a grid algorithm [104]. The world map is divided into a grid of a size defined by the
user, and the system determines in which "cell" the device is currently. Instead of
returning the precise location, we report the middle point of that cell to the app, as
shown in Figure 5.33. The equations to calculate the latitude and longitude positions
need to take into account the changes in the grid size towards the poles. We refer
to the detailed explanation of the algorithm in the original paper [119].
2Figure comes from the original paper [122]c
!2011 IEEE
3Figure comes from the original paper [122]c
!2011 IEEE
“On Improving Privacy and Security Through User-Informed Design.”
52/114 Chapter 5 Development of User-Informed Features
Figure 5.3: Approximating the position with a grid algorithm
Implementing per-app Settings
The feature allows the user to create a system-wide, global setting and an exception
list on a per-origin or per-app basis. The first challenge in achieving the goal was
to make the geolocation service configurable. We wanted to make it possible for
certified apps to change the behavior of the geolocation service without changing
the existing geolocation API. We are using the SettingsDB, which is a system of
storing configuration data and notifying the underlying subsystems about the con-
figuration changes in FxOS. Any database that is globally accessible would serve
the purpose. To make the geolocation service aware of the SettingsDB, we im-
plemented a geolocation-settings singleton that would receive the callbacks from
the SettingsDB, and store the live configuration state for the geolocation subsys-
tem. By deriving from nsIObserver, the class requests to observe callbacks for the
“mozsettings-changed” event, and processes the geolocation settings changes when
they happen. Because the geolocation service singleton exists only in the parent
process, the geolocation-settings singleton is also available only there. To enforce
this, the constructors are all private and the static getter function creates and re-
turns the geolocation settings object only when called in the parent-process context.
Once the settings system was in place, we modified the main geolocation class to
look up the origin/global settings from the watch ID, and then process the location
data before reporting it back to the requesting code.
Application–transparent Obfuscation
On the first level, a user is asked to define the global, system-wide setting that will be
the default choice for any app. In addition, one can create a list of applications that
will be subject to a different adjustment. The apps can be grouped by vendor name,
sorted alphabetically or by the trust level. By the latter, we refer to the definition
of “web”, “privileged”, and “certified”, in order to give the user some idea of how
much they can trust the applications. To speed up the process, we also include a
search engine. By grouping the apps by vendor name, the user can see which of them
may potentially collaborate. Additional ways to improve that mechanism would be
“On Improving Privacy and Security Through User-Informed Design.”
5.2 Permission Control 53/114
adjusting the location granularity based on the application type. However, Firefox
OS does not currently provide information that would allow for such grouping.
Adding the Context to Decision Making
In addition to the per-app setting, there is another level of definition. Following the
concepts presented in other fields, our solution gives the user a choice of determining
temporal and spatial intervals where a chosen level of adjustment will be used. This
means that the user can define time span and space where the location granularity
will be changed. This is useful when entering an area or a time-slot of increased
privacy, such as being in a hospital or during the night. This adjustment is available
for each app separately and on the global level. To increase the usability, spatial
definition is achieved by choosing a region on the map. The size of this hiding spot
is a minimum of 5 by 5 kilometers to reduce the chances of identification. The
temporal setting is achieved similarly to calendar appointments; the user defines
the start and end point and possibly the repetition. It can, of course, be argued
that this mechanism is impaired by combining the map with Points Of Interests
(hospitals, churches and others). However, we are aiming at a solution that allows
for high usability and flexibility, and the user is informed of the risks when the
spatial constraint is chosen.
Creating Profiles
We also decided to include user profiles. Following the conclusion that there are
situations where one might want to switch to a stealthy mode or (quite the opposite)
need all applications to access everything for a short while, we allow users to save
their settings into profiles. This allows them to adjust the granularity ahead of time
and activate the mode when needed. The profiles can also be activated based on the
temporal and spatial information.
5.2 Permission Control
Moving forward we wanted to look at how to improve transparency on the device. Al-
though explicit privacy agreement is required to complete the app installation, users
remain unaware of the consequences of their decisions. This “unintentional autho-
rization” is a key cause of disclosing privacy-sensitive information. As described in
Chapter 4, people are slowly accepting a world where they need to provide highly
sensitive data to get access to the services. They get around this by creating various
personas and believing these cannot be linked [72]. The method we developed as a
result of the UID allows the representation of the privacy Terms and Conditions in a
simple and easily adoptable structure. In addition, we extended the Permission Con-
trol to include a Privacy Meter that provides a convenient, proactive and efficient
interface to the user and gives them a transparent overview of what is happening on
their device.
“On Improving Privacy and Security Through User-Informed Design.”
54/114 Chapter 5 Development of User-Informed Features
5.2.1 User-Informed Privacy (UIP) Matrix
Privacy Terms and Conditions today seem to completely mismatch the requirements
of reality. It is no surprise that people do not understand the consequences of ac-
cepting the agreements, if these are written in a complicated legal language [20][84].
Moreover, while the definition of what is a violation of privacy varies from person
to person, the Terms and Conditions are one-size-fits-all.
In the European Union (EU), business entities formulate their own Terms and
Conditions from the EU regulations for data protection [1], and also they can seek
certification from third parties such as TRUSTe. However, these only contribute
to users blindly accepting them [11][132][84]. UID and discussions we had with
the participants of the process showed that we needed an alternative method of
representing the privacy agreements; one that gives an idea of what the right decision
is, and what the consequences will be. With the use of data visualization [113], we
created a tool that helps in the subconscious understanding of the implications.
However, for the more advanced users, we chose to keep the typical, descriptive
methods.
Figure 5.4 shows we present the matrix that was used to map the Terms and
Conditions into a static framework. Columns of the matrix contain types of data
that is being shared, while rows are populated with how strongly a person feels
about disclosing such information. Every user can shuffle the columns accordingly,
influencing how their Privacy Matrix looks. Data in the topmost row (row 1) should
contain the least sensitive information, about which the user cares the least. On the
contrary, entries across the diagonal (at the bottom right) should include information
that is most valuable. A single entry in the Matrix is also a multi-level one, as shown
in Figure 5.5. It stores the data type and the default tolerance level along with the
expected consequences of revealing the information, and possibly references to the
sources. This multi-level scheme hides the details from users who are not particularly
interested, while providing a greater level of description to the ones who need it.
Additionally, in our model, the user can blacklist only particular elements of a
privacy clause. If the user sets C3.45, it decodes into: “tolerance at level 3 in group
C but does not accept clauses 4 and 5". Now, the actual decision-making process
can be automated and convenient for the user.
5.2.2 Framework
We have created the User Interface(UI) as presented in the Figure 5.6 to summarize
which permissions are accessed by the installed applications, and offer transparency
and ease understanding of the complex flow of the user information. All applications
that are installed on the device are displayed to the user who can sort them based
on the trust level (web, privileged or internal), their vendor, the privacy risk (the
privacy score of the application) or alphabetically. There is an alternative view
where the user can see what resources are accessed by which app.
There is a need for Privacy Enhanced Technologies (PETs) that grade appli-
“On Improving Privacy and Security Through User-Informed Design.”
5.2 Permission Control 55/114
Figure 5.4: An instance of the matrix showing theoretical structure of the solution.
cations from “safe" to “one that breaks your privacy", but such a metric is very
individually defined, so a universal consensus would be hard to achieve. We propose
a “Privacy Risk Score” (PRS). This is a purely subjective measurement that users
adjust themselves based on how they feel about each permission. If they consider
certain information more sensitive, they can give that permission a higher number.
The more sensitive permissions an app uses, the higher its PRS. This metric allows
users to understand what impact some apps have on their subjective privacy, and
possibly allows them to look for alternatives that would not use as many highly rank-
ing permissions. We decided to propose such a model, based on research showing
how individual and context-dependent the definition of privacy is [76][9].
To provide a visual indicator and not only textual information, we include a
colored risk bar next to each application in the Transparency Control tab of the
Privacy Dashboard, as shown in Figure 5.7. If a threshold of 85 is reached in the
overall PRS for a given app, the bar is in red. If the PRS is between 45 and 85, it
is yellow; and below 45, it is green. Numbers were chosen (rather then based on our
assumptions) and can be adjusted by the user with “smart privacy notifications”
(SPN). SPN also allows the triggerring of notifications whenever an app tries to
access a permission that has a permission priority set to a particular threshold. The
permission priority ranges from 0 to 100 with a 5-step increment on a sliding scale.
The values were chosen based on the research on decision-making by Gigerenzer
et. al. [60], anchoring by Kahneman and Tversky [78], and over-transparency by
Acquisti et. al. [8]. Today we calculate the PRS based on a mean value. However,
it is worth noting that better algorithms should be investigated, but we leave that
for future work. Learning from the Adjustable Location Accuracy requirements, we
also introduced the possibility of saving the user settings into one of three modes:
normal, concerned and stealth. They come with pre-defined values for the SPNs,
but can be overwritten. For a detailed description of the implementation, we refer
“On Improving Privacy and Security Through User-Informed Design.”
56/114 Chapter 5 Development of User-Informed Features
Figure 5.5: An element of the matrix showing moderately detailed information
Figure 5.6: Elements of the Privacy Dashboard and its Transparency Control
framework. Sliding bar allows us to define the permission priority according to
individual preference, while risk overview for an app depicts overall privacy concern
“On Improving Privacy and Security Through User-Informed Design.”
5.2 Permission Control 57/114
Figure 5.7: The risk overview for an app depicts overall privacy concern
to the original paper [106] and Nurul Momen’s Master.
5.2.3 Evaluation
As many empirical studies show, the traditional privacy Terms and Conditions are
ignored in most of the cases. Additionally, the policies for different business entities
may vary based on offered services and business strategies. It is very difficult (or
nearly impossible) to distinguish the differences and consequences for millions of
apps and services. European data-protection policy is usually taken as a reference
point for analysis4.ThePrivacy Meter provides a much easier way to understand
what is happening on the device.
Some large service providers outsource the privacy-management solutions to
third parties. An example here is TRUSTe5which has a customer list that includes
all the big players such as Apple, eBay, IBM, Cisco, HP, LinkedIn and Oracle. Yet,
although they follow a particular service, the policies and the representation of the
agreement usually do not overlap. Therefore, it is difficult for users to keep track
and be aware of the privacy policies for each of the services. As a result, failure to
raise privacy awareness is observed in many scenarios [132][84].
Schlegel et.al pointed out the ongoing demand for a transparent way to quan-
tify, interpret and control individual privacy [137]. They also tried to get rid of the
usual cumbersome interface for privacy-related information. Their survey on 43 users
determined a satisfying level of accuracy, followed by an indication that using a con-
venient interface can support users’ positive behavior. The authors concluded that
transparency opens another dimension for the users to evaluate the consequences,
and encourages them to be more aware of the implications. It also helps them to
4http://www.echr.coe.int/Documents/Handbook_data_protection_ENG.pdf
5https://www.truste.com/window.php?url=https://download.truste.com/TVarsTf=
9GEA1GX6-488
“On Improving Privacy and Security Through User-Informed Design.”
58/114 Chapter 5 Development of User-Informed Features
Figure 5.8: Privacy risk bar for each of the permissions visualizes easily perceivable
consequence.
provide consent with definitive judgment. This work aligns well with our choices for
a simple and transparent interface that alerts users only when necessary.
Dayarathna carried out research on developing a taxonomy for the privacy met-
rics [42]. The paper discusses a demand for an effective framework that quantifies,
measures and understands privacy. It also introduces a conceptual model for the
information-privacy metrics and elaborates on the basic foundation, the constructs
and the dimensions of the metrics. This work further underlines the necessity for a
static framework, which aligns with our research, though we chose a much simpler
UI, and included the notification generator.
Bagueas et.al introduced a complete solution to handle privacy protection
through pervasive computing [85]. In this case, the system takes care of the sensitive
data on behalf of the users according to their predefined preferences. It has helped
us to realize the requirement for taking user preferences into consideration. Park and
Kautz addressed privacy-preserving activity recognition [114]. They used a dynamic
Bayesian network to introduce efficient handling of situations where human activity
can be recognized without exposing the actual appearance of the inhabitants of a
smart home. It was interesting that this novel and privacy-preserving effort remains
almost unnoticed by the users due to a poor UI. It has showed us the importance of
“On Improving Privacy and Security Through User-Informed Design.”
5.3 Event-Centric Context-Aware Authorization 59/114
a good design, but also how our research can be applied to the market.
As mentioned earlier, we have been facing a two-faced challenge: our efforts to
make the privacy implications transparent for the users, and to fill the gap between
current solutions and user expectations. To pursue these goals, it was challenging
to determine the required depth of transparency that should not result in an inef-
ficient control. The right balance between flexibility and usability has, once again,
proven to be hard to achieve. In this work, we created a framework to change the
privacy requirements into a structured matrix that is used for the Privacy Meter.
While available solutions present privacy in difficult-to-understand and non-standard
terms, our solution aims to provide a simple comparison tool. Most of the existing
frameworks fail to take into account users’ individual preferences.
5.3 Event-Centric Context-Aware Authorization
Having developed tools for everyday users, we focused on what happens when they
go to their place of work. How do their employers deal with their “everyday” devices
when in a work environment? As users are accustomed to a world of choices and
possibilities, they find the restrictive device-management policies extremely frustrat-
ing [109]. We have observed that in the domain of access control, the driving factor
was the capability of the devices and not the needs of users, which stands against the
concept of user-informed and user-centric designs. In the case of individuals, we care
only about the privacy of a single person. However, in an enterprise, poor systems
may expose sensitive intellectual property assets such as financial data, customer
information, engineering documents, company directories, proprietary applications
and other items necessary for successful business activity [129]. It is fairly common
to create solid authentication mechanisms to validate whether the current user is a
legitimate one, but rarely do we see systems that are able to verify how the assets
on a device, such as a camera or microphone, are used.
Through UID we have evaluated that the policy decisions should be based on
the state of the device and the user’s context at the time they request access to a
given resource. This meant that the decision on whether to grant access has to be
made based on data that is asset-specific and dynamic, and a policy authority should
take full responsibility for the secure usage of a given resource. Each decision has to
take into account the priority of policies and all the variables that influence it. These
requirements cannot be fulfilled by traditional, static methods. We have followed
the UID and created a real-time, dynamic, contextually-aware policy model with a
definable policy enforcement, which provides more than just a binary decision.
Such a solution follows a UID from the enterprise perspective as well. The
management of policies is typically handled by a single Master Administrator that
is irreplaceable and all-knowing. In the framework we present, there is a hierarchy of
policies where multiple people can create policies with various priority levels. There
is no longer an issue with account-system updates or changes in the underlying OS
or contradicting rules. Through the UID we identified the problem of a lack of au-
“On Improving Privacy and Security Through User-Informed Design.”
60/114 Chapter 5 Development of User-Informed Features
Figure 5.9: Framework Overview. The Policy Decision Kit (PDK) captures the
user’s design intent at Level 1, and generates an instance of the PDP at Level 0.
thorization mechanisms suited for mobile and distributed information systems. As a
solution we created a framework and an architecture that has general application to
the management of many device types, including the IoT. In Sections 5.3.1 and 5.3.2
we present the description of the framework, its architecture and method of policy,
and policy management that is an extended version of the paper I wrote with At-
tfield et. al., “Can I Let You In? On Event-Centric Context-Aware Authorization”
published at ICONS 2016 [19]6.
5.3.1 Framework
We base our framework on an XACML model [52], and similarly it comprises two
layers. Figure 5.9 shows the data flow from the user’s design intent, through Level
1 and Level 0 to produce an instance of the framework.
Level 0: Includes the operational components that communicate together to im-
plement the policy-authorization process in a live system. This level also includes
compilers that synthesize policy specifications in the Policy Object Language (POL)
into executable Policy Decision Point (PDP) instances for use in a live system.
Level 1: Contains the Policy Design Kit (PDK), a GUI-based policy-design en-
vironment that simplifies the authoring of complex policy sets. It also manages
the end-to-end process of creating an instance of a Level 0 framework, ensuring its
integrity, authenticity and correctness.
As shown in Figure 5.10, the operational architecture of the framework has
four components: Policy Enforcement Point (PEP), Policy Decision Point (PDP),
Policy Information Point (PIP), and Fabric. A PEP is an agent located on a device,
such as a mobile phone, which monitors events on the device that represent requests
to access resources under policy control. For each such request, the PEP creates a
query message that contains details of the request and the state of the device at the
time of the request, and transmits it to a PDP. The PEP then waits for a response.
When the PEP receives a response, it enforces the decision contained therein.
6All figures come from the original paper
“On Improving Privacy and Security Through User-Informed Design.”
5.3 Event-Centric Context-Aware Authorization 61/114
Figure 5.10: Framework Level 0 Architecture. The Level 0 architecture consists of
four components: Policy Enforcement Point (PEP), Policy Decision Point (PDP),
Policy Information Point PIP), and the Fabric (depicted by the double-headed ar-
row).
A PDP is a server that may be located remotely or, sometimes, co-located with
the PEP in a device. The PDP listens for queries from PEPs and, using the data
within the query and policies specific to the PDP instance, transmits a response to
the requesting PEP.
A PIP is a data source, external to the PDP, containing information needed
to evaluate policies. PIPs are typically directory services. The PDP has special
features that exploit access to PIPs efficiently.
Depending on the nature and disposition of the PEPs and PDPs and the ap-
plication for which they provide an authorization service, PEP queries and PDP
responses may be transmitted through the Fabric that may be any of a wide va-
riety of external media. As will be detailed further, these may be as complex as
User Datagram Protocol (UDP) over Universal Mobile Telecommunications System
(UMTS) [136] or as simple as shared memory. The Fabric is depicted as a double-
headed arrow in Figure 5.10.
Policy Enforcement Point (PEP)
From an abstract point of view, queries and responses consist of an array of elements
and a Fabric-dependent header. Each array element represents a PEP state attribute
or data pertinent to the request. For responses, each array element contains an
attribute of the response, such as a verdict or information qualifying the verdict,
for example a stipulation. Array items are identified by their indices, as agreed in a
dictionary shared by the PEP and the PDP. The array is referred to as the dynamic
array, and each of its elements is referred to as a dynamic element.
“On Improving Privacy and Security Through User-Informed Design.”
62/114 Chapter 5 Development of User-Informed Features
Figure 5.11: Policy enforcement. The Policy Enforcement Points (PEPs) are re-
mote agents located on the managed device that are triggered by access requests.
PEPs query the PDP for authorization and enforce the response.
Depending on the type of resource, the PEP can be located at the driver level
or implemented as an application. This is where we introduce the first level of
innovation compared to XACML. The PEP consists of a Trigger, a Policy Query
Module (PQM) and an Effector, as shown in Figure 5.11. The Trigger, interrupting
the normal flow of an event, gathers the relevant dynamic content and sends it to
the PQM.
The PQM creates queries and enforces the intent of responses on the operation
of the device. Depending on the Trigger and any included dynamic attributes, PQM
may look for additional environmental variables to form a query. It then sends the
query to a designated PDP (local or remote). Finally, it obtains a policy decision
and provides the response to the appropriate Effector for that Trigger. The process
followed by the PQM varies according to the nature of the PEP. This includes
consideration of the state of the device (online or offline), whether the device policy
cache may be used, and which PDPs may be consulted for a policy response.
The Effector enforces verdicts returned by the PQM. It provides a meaningful
control path for each possible verdict/stipulation combination and ensures a sensible
user experience for all outcomes. Caching is shared by all PEPs on a given device
because the wireless network can present an inconsistent and sporadic connection to
all back-end services. Policies can direct caching of responses for a specified period
of time. This reduces fabric bandwidth demand on the network and allows for near-
instant response (to queries that match the same conditions) without recourse to an
external connection. This is useful for devices that have a high-frequency request
cycle, such as a camera that may query as frequently as 60 times a second. If there
is no cached decision and the PDP is not reachable, a simple set of decisions is
stored on the device to provide a fail-safe response for all PEPs. If the network does
not allow for a real-time decision to be received, the response is equivalent to that
provided by traditional MDM solutions.
Policy decisions may include stipulations that require logging. There is a com-
mon service that collects log data and delivers it efficiently. For example, if multiple
control points are logging and the network is unavailable, the common service will
hold the data and deliver it as a bulk payload when the connection is restored. This
“On Improving Privacy and Security Through User-Informed Design.”
5.3 Event-Centric Context-Aware Authorization 63/114
Figure 5.12: Policy Decision Point (PDP). The PDP is a server that issues re-
sponses to queries from PEPs. Its responses are based on PDP-specific policies.
PEPs and PDPs communicate through a two-way channel.
relieves individual control points from tracking and performing these functions. The
heartbeat function maintains periodic communication between the PEP and the
PDP, confirming PEP presence and state. The PDP may respond to heartbeats
with control messages.
The installation of PEPs to devices depends on the nature of the implemen-
tation and the protected resource. PEPs protecting hardware resources are imple-
mented at the hardware resource-driver level. For example, a camera resource needs
to be implemented in the camera driver. PEPs protecting data files, for example,
may be implemented as a part of a file-system driver or as a part of an application.
The PEP ensures a sensible outcome for the end-user or application when an access
attempt is denied. Devices do not appear broken; applications do not crash.
All of the presented framework has been implemented as part of an MDM
solution deployed on the market, which will be described later in Section 7.1. The
engineering effort requires access to device-level drivers because all of the policies
are enforced there, which means the proposed solution is not one of plug-and-run.
It requires collaboration between the implementers and the device manufacturers.
Policy Decision Point (PDP)
The Policy Decision Point (PDP) is a server that issues responses to queries received
from PEPs. The responses are based on policies that are specific to the PDP and
make use of the dynamic values in the query, as shown in Figure 5.12. To ensure
availability, a PDP always returns a response to a PEP even if no relevant policy is
“On Improving Privacy and Security Through User-Informed Design.”
64/114 Chapter 5 Development of User-Informed Features
found. A PDP is defined as permissive or restrictive depending on whether it sends
an "allow" or "deny". A PDP can handle large collections of policies (thousands
or more) without significant performance degradation. The PDP is stateless. This
improves performance and simplifies interactions with PEPs and the management
of PDP farms. For audit and forensic analysis purposes, a PDP generates a log of all
query transactions. A PDP may respond to PEP heartbeat signals with commands
that affect the state of the PEP or host device. One of the operations that may
be requested is maintenance for a PEP instance. The decision engine of the PDP
interfaces at the level of abstract query objects. This allows the use of various
harnesses to adapt the PDP to a variety of communication fabrics. Furthermore,
this provides for the creation of systems with multiple, concurrent PDP instances
for higher bandwidth needs.
The PDP is one of the most security-sensitive aspects of our framework. How-
ever, its architecture minimizes the opportunities for malicious intervention. The
code of the PDP itself, as well as the representation of policies, is attestable (signed
and authenticated) through a chain of trust. The process of creating PDPs and their
policies and delivering them can be made verifiably secure by booting in a Trusted
Execution Environment. The PDP is synthesized by a compiler, which directly
generates its executable, including the embedding harness, from policy specifica-
tions in the proprietary POL. The PDP offers no programming API or facility for
source-code tampering or modification, and the executable may be signed. To limit
the attack surface, the PDP has only one I/O channel on which encrypted packets
are received (queries) and transmitted (responses). It has an output channel on
which logging information is emitted. Moreover, in order to prevent unauthorized
access and modification of the policy database and rules, the database is password-
protected. Only the compiled PDP and the database server have the key, which is
generated at compile time. The PDP runs in Trust Zone and the key generation
relies on the root of trust. Essential database fields may also be encrypted. Lastly,
when it comes to network fabric, queries and responses are transmitted using a pro-
tocol that is resilient to Man-in-the-Middle attacks and spoofing. The PDP drops
incoming packets as soon as protocol discrepancies are detected.
Policy Information Point (PIP)
Corporate data, particularly personnel data, is often stored in active directories and
databases that deliver fast data retrieval and consistency. Policy formulation often
requires precisely this kind of data. The PDP has been designed to connect to
directories. The POL provides directory specification and search functionality to
make the data available for policy evaluation by the PDP.
5.3.2 Policy and Policy Management
From the previous description, it is clear that the behavior of devices governed by the
system depends on the policies embedded in the PDP. A policy is a rule that dictates
“On Improving Privacy and Security Through User-Informed Design.”
5.3 Event-Centric Context-Aware Authorization 65/114
what actions should be taken for a particular event under a given set of conditions.
Individual policies are gathered together into policy sets, which together address
all of the events and circumstances of interest to the policy authors. This is again
something that XACML does not fully support, because the handoffbetween entities
cannot be done in a trustworthy way.
The objectives of policy management include the following:
•The method of expressing policies is rich enough to express policy-author intent
under a wide range of circumstances, some of which cannot be foreseen. It
is succinct so that the resulting policy sets remain manageable in size and
complexity.
•Policies are written only by those who have the required authority for the
resource being governed.
•Policy sets created by many authors are combined with a clear order of prece-
dence, with any conflicts or logical problems detected and addressed.
•Information referenced by the policies to arrive at decisions, in PIPs or other-
wise, can be written by properly authorized authors only.
•Compilation and deployment of PDPs uses a controlled process, and is done
by authorized individuals.
A result of a stringent implementation is a chain of trust for the policy data,
beginning with the creation of the policy and extending through to the deployment
of PDP. This chain of trust ensures that the integrity of policy intent is maintained.
The establishment of the chain of trust requires two types of policies: Level 0 policies
which specify how PDPs respond to device requests, and Level 1 policies which
govern access control for data used to create Level 0 policies. These also secure
custody of the data. The set of policies is defined by the respective employees.
Each person who contributes to building the policy hierarchy tree has their own
set of privileges and scope in which they can define what can or cannot be done.
Thus, the contextuality is defined by humans, not by training; people higher in the
hierarchy have the power to determine context in which I may or may not use my
device.
Policy Object Language (POL)
Level 0 policies are specified in a formal policy language, the POL. The POL is terse
and declarative to facilitate synthesis and static verification of policy sets. It follows
a pattern of: Subject, Agent, Object, Action and Environment (SAOAE). Each
component has a corresponding clause in the model which consists of an arbitrary
expression that can reference dynamic data from the query, static data from a PIP,
or data from the policy set itself. The clauses organize policy considerations along
the following lines:
“On Improving Privacy and Security Through User-Informed Design.”
66/114 Chapter 5 Development of User-Informed Features
•Subject: the identity of the entity making the request, e.g. the user.
•Agent: the means by which the request will be carried out, e.g. the program
that will make the access.
•Object: the elements and items affected by the request and being acted upon
by the Agent.
•Action: the specific function that the Agent applies to the Object.
•Environment: information in the request that would be observable at the PEP
but independent of any given event, such as time or location.
The POL provides a mapping feature that allows query and PIP data to be
tagged along arbitrary lines. Tags can be tagged themselves, forming chains which
begin with query or PIP data, allowing large numbers of specific data elements to
be aggregated into manageable categories for expressing policy intent. Support is
provided for regular expression constructs, simple geolocation constructs, and time
intervals. For example, a tag chain could be constructed to map an identifier from
a mobile device into an employee number, and then into a role or a location or an
authorization level. Clauses in SAOAE statements can reference tag chains in their
expressions. The number of tags and length of tag chains is not dictated by the
language. It can be used to construct both policies with a wide scope and general
application, or policies for specific situations with very fine granularity.
Designated authorities are permitted to write the policies for those areas within
their domain. As shown in Figure 5.13, policy stakeholders are arranged in a hier-
archy and are allowed to delegate authority to those beneath them. This permits
higher-ranking authorities to reserve for themselves the rights they need and to del-
egate policy decisions to others as appropriate. Policies may be marked as default,
to permit policy-set closure if no lower-ranking policy authors provide an applicable
policy. We will discuss more policies and policy design in Section 5.3.3.
This policy-ownership tree model can be applied in various ways to simplify
the management of complex policy sets. For example, one approach might use an
organization chart to map the hierarchical tree such that levels of policy authority
correspond to levels of organizational authority. Another example might use a high-
ranking policy set to specify some coarse-grained generic default policies and one
or more lower-ranking policy sets to create categories of specific exceptions to the
generic rules.
The POL and its constructs define Level 0 policies that address the manner
in which the system responds to requests from devices. To implement the chain
of trust as outlined above, another layer of administrative (or Level 1) policies is
required to govern the authentication and authorization of stakeholders as they
perform their duties within the system. These duties can consist of policy and data
entry, configuring PIPs, compiling and deploying PDPs, introspection and debugging
of policy sets, and administration and management of the administrative policies
themselves. Administrative policies are defined and enforced in the PDK.
“On Improving Privacy and Security Through User-Informed Design.”
5.3 Event-Centric Context-Aware Authorization 67/114
Figure 5.13: Policy Ownership Tree. Policy stakeholders are arranged in a hier-
archy, each responsible for their own domain and allowed to delegate authority to
those beneath them.
Policy Design Kit (PDK)
As shown in Figure 5.14, the PDK is a general-purpose authentication and autho-
rization platform that provides controlled access to the data and tools and a policy
life-cycle framework. After authenticating to the PDK, users receive a fine-grained
set of abilities dictating allow/deny access to various resources. Data objects in
the PDK have a strict chain of ownership back to a user. These abilities combine
with the object ownership to enforce Level 1 policy. Relationships between users are
established in the PDK to enforce rules for sharing data, and to establish relative
positions in the policy-ownership tree. In many ways, the PDK provides the full UI
and abstraction layer so that a non-technical or high-level employer and manager
can also implement their policies that will be combined later into the policy tree.
The PDK provides policy models that allow users to express their Level 0
policy intent at a level of abstraction suitable for a given application. For example,
a policy model that governs a resource according to a time interval and a location
might present an interface that captures just the interval and the location. No
knowledge of the POL is required to use the models. Data to populate a policy
model’s tags is also captured by the model at an appropriate level of abstraction.
The PDK provides a number of policy models; however, a given user only has access
to those for which they are authorized by Level 1 policy. While PIPs usually contain
common organization data, often additional data is required for policy authoring.
For this reason, the PDK provides models that allow users to capture extra data
and explicitly share the information with other selected policy authors by using the
PDK’s authorization mechanism. The shared data is available to a second user’s
“On Improving Privacy and Security Through User-Informed Design.”
68/114 Chapter 5 Development of User-Informed Features
Figure 5.14: Policy Design Kit (PDK). The PDK is a policy life-cycle management
framework with a set of tools for the policy administrator to conduct policy author-
ing, introspection of policy sets, PDP generation, sanity checking and deployment.
policy models but cannot be viewed or changed. As an example, an employee in
HR may be responsible for maintaining employee data, while an IT employee may
dictate network policies using a role-based scheme bound to that employee data.
The PDK provides a set of user-administration functions that allow users au-
thorized by Level 1 policy to manage other users in the PDK. These functions include
the typical user life-cycle operations, as well as the assignment of abilities to users
which grant them access to various functions, policy models and shared data in
the PDK. Users and their policy sets can also be assigned to their positions in the
policy-ownership tree. The PDK provides tools for the generation and compilation
of PDPs from policy-set trees. Effectively the user interacts with the PDK only
when creating policies. Everything else happens in the back-end and is distributed
to the devices and enforced in a specified context.
The first step in the process is to synthesize POL code from the policy model
and captured data. The POL code is then compiled to produce C++ code and any
“On Improving Privacy and Security Through User-Informed Design.”
5.3 Event-Centric Context-Aware Authorization 69/114
associated database. The compiler carries out simple semantic checks as well as
a sanity check, which verifies that the policy set possesses certain properties. For
example, it verifies that there is no ambiguity where two policies might apply to the
same query, and it verifies that there is a set of query data capable of activating
each policy in the set. The last step is to compile the C++ code to produce the
executable PDP. The entire flow can be executed step-wise or automatically as a
single step. We cannot provide further detail on how the compiler has been written
because it is part of the Intellectual Property protection and core product today.
The PDK provides functionality to define server elements, to transfer the exe-
cutable PDP to them, and to set them running. Server elements can be designated
as database servers, PDP servers or both. This allows flexibility in the number of
PDPs deployed, and in load-balancing the number of executing PDPs to the number
of corresponding database servers.
Debugging and Introspection
The executable PDP produced by the compiler has minimal I/O. I/O is limited to
receiving requests, emitting decisions, and providing a log file. The log file indicates
which policy was used to arrive at a decision; however, it doesn’t provide any in-
formation on how that policy was selected. As policy-set complexity increases, so
does the need for tools to analyze and debug them, and examine them forensically.
To address this, the PDK provides features for log-file analysis and management as
well as policy-set introspection.
Authorized users of the PDK can collect PDP log files from various servers,
combine and analyze them. The log entries for any given device may be distributed
among a number of PDPs in a farm. Combining them allows the entire history
for any one device to be examined. The combined log is elaborated to allow any
sequence of events to be found quickly.
Introspection refers to the activity of studying policy-set behavior by simulating
the PDP look-up process, while displaying all intermediate results. For a given
simulated request, each of the policies considered are displayed in rank order, along
with the values of all policy clauses. Intermediate terms and tag expressions in the
clauses are also evaluated and displayed, providing detailed information showing
why any given policy was selected, rejected or not evaluated. This capability allows
authors to evaluate different scenarios against policy sets and groups of policy sets
to determine if desired outcomes are being produced.
5.3.3 Example
To illustrate how this framework model differs from existing solutions, let us use
the following example of ACME Corp. It generally allows unmonitored camera use.
However, in the Seattle factory, camera use is forbidden except for managers during
work hours. In this example, Mike is a manager and Dan is a developer. For the
entire Acme Corp, only three policies are required to manage camera use:
“On Improving Privacy and Security Through User-Informed Design.”
70/114 Chapter 5 Development of User-Informed Features
•The corporate default allows usage.
•The exception condition in Seattle that denies use in the factory (and logs this
occurrence).
•The exception to the Seattle exception, allowing managers use during working
hours (and logging those attempts).
Policy 1 is placed at the top of ACME’s policy-ownership tree and is given
default status. Policy 2 is beneath it in the tree, in a policy set for the Seattle
division, and marked default as well. Policy 3 is also placed in Seattle’s policy set,
but not marked default. ACME’s corporate database is used as a PIP to link Mike
and Dan to their roles.
The SAOAE clauses for Policy 3 might include:
Subject = managers;
Agent = n/a;
Object = n/a;
Action = use camera;
Environment = during working hours, in the Seattle factory;
Verdict = allow;
Stipulation = duration = 300 seconds; Stipulation = log.
// The 5-minute cache duration greatly reduces the query rate of 60 PEP
// queries per second (based on camera imaging at 60 frames/second).
Policy 2: the SAOAE for clause stipulates logging for a "deny" verdict, resulting in
the PEP forwarding this logging information to the PDP. After the policy author
uses introspection to verify that the policies accomplish their purpose, the PDP is
generated and the policies are active.
This efficient structure accomplishes the following: policy ambiguity is man-
aged by policy hierarchy, ACME Corporation is allowed a broad set of corporate-wide
policies, and the Seattle division has local control over those issues that concern it
specifically. The Seattle division can provide an exception to the corporate policies,
and then override that exception for very fine-grained control. Policy enforcement
is dynamic, and verdicts vary depending on dynamic data from the device, as well
as to changes in PIP data and the policies themselves. Dan the developer is able
to use his camera freely. However, when he enters the Seattle factory, his use of
the camera is denied. Mike the manager can use his camera in the Seattle factory
during work hours. However, after work hours, his camera is disabled in the factory.
No changes have been made to the PEP, PDP or policy set; only dynamic data in
the query has changed.
Now, let’s say that ACME Corporation promotes Dan, so he is now a manager.
In the normal course of events, Dan’s record in the PIP is changed by Human
Resources. Subsequently, if Dan attempts to use his camera during working hours
in the Seattle factory, access will be allowed. No change is made to the device, the
“On Improving Privacy and Security Through User-Informed Design.”
5.4 User-Informed Design (UID) of Privacy Tools 71/114
PEP or the PDP; only PIP data has changed, and so has the behavior of Dan’s
camera.
This discussion has used the camera as the controlled resource. Other resources
made visible by PEPs could be controlled in a similar manner. On a mobile device,
the following resources are among the candidates for control:
•application install or launch
•microphone, screen, keyboard and device access
•Wi-Fi, Bluetooth, data, telephony and SMS
•network access and file access
•privilege escalation
An appropriate PEP can be installed on the device to control any of these resources
in a similar fashion.
The bottom line is simple; any device where a PEP is deployed can be managed
by the framework. The use of the PIP linking Mike and Dan to their roles allows
the policies in this example to apply to them whether they are using their phone or
tablet or other device. If ACME has any factory-specific in-house hardware, it could
be controlled as well. The direct interaction with the framework is performed by
ACME’s policy administrators using the PDK, which allows each administrator to
focus on their areas of expertise. The device users have no input to the framework.
Their experience with denial of camera use is dictated by the framework, which in
ACME’s case displays a security violation message.
5.4 User-Informed Design (UID) of Privacy Tools
At the beginning of Chapter 4, we described a model of UID. It is a process in which
we designed privacy-preserving solutions starting with the most simple thing: talking
to the people for whom we will be building these tools. While it may not seem like
a great novelty, it has shifted the way we thought about the process and architect
the features. Having a better understanding of how users think the mobile devices
work, but not making assumptions about what they need, has brought a massive
improvement over the existing methods that lead to designing privacy-preserving
solutions that worked great, but the adaptability rate was low.
In the case of the three particular features that we took beyond the proto-
type phase of Privacy Dashboard, we have learned even more. Through the UID,
we added to Adjustable Location Accuracy the per-application settings, context-
dependent adjustments and profiles, which we would not have considered previously.
We will further discuss how the explanation included in the Guided Tour achieved
the goal of educating the users on threats connected to revealing their data, and gave
them the opportunity to protect it according to their needs and concerns. After de-
veloping the tool, we have found interesting research by Puttaswamy and Zhao [126],
“On Improving Privacy and Security Through User-Informed Design.”
72/114 Chapter 5 Development of User-Informed Features
where the external servers are treated as simple encrypted data stores, and we think
that the Adjustable Location Accuracy could benefit from building on top of this
solution.
Moreover, it was through the same methodology that we found there was a need
to implement the Privacy Meter — a tool to present a transparent impact of an app
on user privacy. As an extension to the existing Privacy Panel, we offer an overview
of applications installed on the phone and the data access they are requesting. Users
can assign priorities to each permission to define their tolerance threshold. Based
on those priorities, each app gets a "privacy score" that is displayed in the panel.
In addition, whenever a "sensitive" permission is accessed, an alert can be triggered.
The framework was everything people asked for: it is simple and intuitive on the
basic level, but gives a detailed explanation and empowers them, if they want it.
Finally, we found that current methods of controlling access to the device are
hard to manage, and they work under the assumption that policies can be defined
once in a lifetime. This might have been the case in the device-centric world of PCs
but is no longer valid with BYOD mobile devices, wearables and the IoT. Thus,
we created a context-driven policy-definition and enforcement framework that takes
into account the state of the system, time, location and any other programmable
factors to validate access to a device. In Section 7.1, we will show how we applied
it to an MDM use case. It showed us how extensive and complex systems can be
subject to powerful and flexible policy control with appropriate granularity, while
still remaining manageable.
“On Improving Privacy and Security Through User-Informed Design.”
73/114
Chapter 6
Lessons Learned
If we recall Figure 1.1 from Chapter 1, as the work progressed we have looked
at various layers of the security stack. We started offat the very bottom with
the kernel security and showed how, given the history of bugs discovered in the
third-party kernel drivers, it is important to be careful in the fast product cycles of
smartphones. Next, we moved a more global work — how to evaluate the security
of a system. Finally, we designed a process to build tools with a new, user-informed
approach. In this model, instead of evaluating tools with user studies after they
were built, we involve the users in the design process. The questions remain: What
does it bring us? Are our tools in any way better or more intuitive?
6.1 Privacy Dashboard
The first product of the UID was the Privacy Dashboard that encapsulated a set
of PETs that everyday users expected to have on their mobile devices. After a
preliminary study where we identified their needs, we developed a prototype that
was a quite broad system to resolve the found issues. Next, we conducted another
study to obtain opinions about the product of our work.
To eliminate any biases and understand the present mental model of the par-
ticipants, we asked them a set of questions about using settings in general. In the
survey we used the seven-point Likert scale [130] to give them the opportunity to
express subjective responses, where: 1 = “Strongly disagree”, 2 = “Disagree”, 3 =
“Somehow disagree”, 4 = “Neither agree nor disagree”, 5 = “Somehow agree”, 6
= “Agree” and 7 = “Strongly agree”. As expected, the participants did not find
it hard to understand the categories and functions in the settings on their smart-
phones, with the median equal to 2. They were also confident in being able to find
a given setting in the app; again the median answer was equal to 2. However, we
were surprised to find that they also thought that it was easy to find help when
experiencing problems with the settings (median = 3).
Next, we wanted to see how intuitive the current design of the settings ap-
plication was, compared to the choices people made, when unconstrained by any
architecture. We asked them to assign the features of the Privacy Dashboard to
the categories that are normally available in FirefoxOS: Network & Connectivity,
Personalization, Account Management, Privacy & Security, Storage and Device.
Moreover, we asked them to judge which of the wireframes we presented to them
were “good” or “bad” design and express their comments. In the first phase, users
“On Improving Privacy and Security Through User-Informed Design.”
74/114 Chapter 6 Lessons Learned
Figure 6.1: Categories and Features. The figure presents the way participants
categorized each of the features included in the Privacy Dashboard. Not all of them
were considered privacy oriented
could assign each feature to more than one category; Figure 6.1 1shows the results
with the percentage of votes, rather than number of participants that chose a given
option. Interestingly, the RPP was put in the Privacy& Security section 30% of
the time, and in the Network & Connectivity section 25% of the time, which was
backed up with arguments that RPP requires connection to the network to enable
the remote connection. Backup was mostly assigned to the Storage, at 54% point,
but 12% still chose to place it in Privacy & Security. Both Adjustable Location
Accuracy (ALA) and Secondary-user Mode (SUM) were mostly seen as Privacy &
Security features. However, ALA was also categorized as Network & Connectivity,
because “it depends on the GPS and is populated over the network”, while SUM
was included in the Account Management in 31% of cases. The Guided Tour was
placed in the Device category in 60% of cases.
Finally, we moved to analyzing the ease of learning, satisfaction of performing
tasks and overall satisfaction with the four features: RPP, Backup, ALA and SUM.
On a seven-point Likert scale, the participants stated that overall it was easy to
learn, they were satisfied with performing the tasks and overall features; all the
1The figure comes from the original paper [122]c
!IEEE2011
“On Improving Privacy and Security Through User-Informed Design.”
6.1 Privacy Dashboard 75/114
Figure 6.2: Learnability, performance and satisfaction of Privacy Features. The
dots represent the outliers, the stars represent the extreme scores.
median scores were above 6. However, as shown in Figure 6.2 2, more participants
gave higher marks towards the ease of learning aspects, rather than the satisfaction
of performing the tasks and the overall satisfaction, which reflects on the ALA and
the SUM. When asked to justify their answers, we found that over-flexibility was
overwhelming to them and rather annoying. The initial prototype required the users
to adjust the settings for each app separately, which was time-consuming. The study
clearly showed that the approach of global settings with an exception list (or an opt-
out mechanism) is much more efficient and intuitive. Additionally, as an effect of
that study we added a search function on the exception list because people said they
knew in advance which applications they would like to choose and did not want to
browse through the whole list.
6.1.1 Guided Tour
After analyzing the general perception of the Privacy Dashboard we moved to the
individual features, starting with the Guided Tour. We asked people to go through it
and read the screens before diving into the Panel. The average time spent on reading
it was 3 minutes 45 seconds, with standard deviation of 1 minute 45 seconds. For the
survey, we used a seven-point scale again, this time with parings “Difficult/Easy”,
2The figure comes from the original paper [122]c
!IEEE2011
“On Improving Privacy and Security Through User-Informed Design.”
76/114 Chapter 6 Lessons Learned
Figure 6.3: Ease of use of the Guided Tour. Users were asked to state if it was “Easy
to understand”, “Helpful” and “Worth reading”. The dots represent the outliers, the
stars represent the extreme scores.
“Unhelpful/Helpful” and “Not worthy/Worthy”. Overall, as shown in Figure 6.3 3,
participants thought that the Guided Tour was easy to understand (median = 6),
helpful (median = 6), and worth reading (median = 6). In the free-form text section
they told us: “It was very intuitive”, “Pictures help as well”, and “Because I am
new to this function, I like background information and useful explanations”.
Additionally, we wanted to evaluate the impact on education about the privacy
threats. To do so, we asked the participants to express their position on four pre-
sented statements while filling out the background questionnaire, and then presented
them with very similar statements upon completing of the Guided Tour:
1. I am afraid about what happens to my information in my smartphone when I
cannot find it or it is broken.
2. I do not like lending my phone to others because I do not want anyone accessing
the information in my smartphone.
3. Providers of apps handle the information they collect about users in an im-
proper and distrustful way.
3The figure comes from the original paper [122]c
!IEEE2011
“On Improving Privacy and Security Through User-Informed Design.”
6.1 Privacy Dashboard 77/114
4. Smartphone users have lost control over how the data is collected by the app
providers.
5. I learned that my personal information in my phone will not be safe if I cannot
find my phone or it is broken.
6. I learned that my personal information in my phone will not be safe if I lend
my phone to someone.
7. I learned that my personal information in my phone can be misused if a third-
party app that I install gets it.
8. I learned that some of my sensitive personal information in my phone is being
obtained/sought by many apps that I installed.
As can be observed, the statements were formulated in a way that was supposed to
be similar but not identical, and they relate to the typical fears that users might have
when losing their device or having their private data leak. Participants were asked
in both cases to use a seven-point Likert scale to formulate the responses, anchored
with “Strongly disagree” and “Strongly agree”. When looking at the histograms we
noticed that the data was not normally distributed. Therefore we adopted Kendall’s
tau non-parametric test [70] to find the correlation between the two groups of an-
swers. The results showed that the first set of answers was positively strongly
correlated to the second one, except the ones given to statements 4 and 8. As shown
in Figure 6.4, there was a significant relationship between statements 1 and 5, state-
ments 2 and 6, statements 3 and 7 respectively. We concluded that the Guided
Tour succeeded in increasing users’ awareness of privacy. However, after applying
the Wilcoxon signed ranks test [147] on the set that had the significant positive
relationship, we found that the increase in the awareness was not as high as we
were hoping. Therefore, we decided to change and extend the Guided Tour. We
also realize that the formulation of the questions might have had an impact on the
results. After further research on the scale design [130]4, we decided to investigate
in future a greater diversity of questions.
6.1.2 Implications for Design of Privacy Dashboard
What we have learned from the user-informed design of the Privacy Dashboard
was that whenever we empower the users, we also need to make sure we educate
them well, so that they know how to navigate in the system. Moreover, we need
to make the solutions as intuitive as possible, also paying attention to good UIs,
and not just the hardened technical layer. Our study showed that gathering all
of the privacy settings under the umbrella of a single Privacy Dashboard app is
the best arrangement. Additionally, it is crucial to explain the individual features
themselves. In the original version, the Guided Tour did not provide as much help as
4The figure comes from the original paper [122]c
!IEEE2011
“On Improving Privacy and Security Through User-Informed Design.”
78/114 Chapter 6 Lessons Learned
Figure 6.4: Correlation between the answers given before (to statements 1 to 4) and
after going through the Guided Tour (statements 5 to 8). Compared were similar
statements: statement No. 1 and No. 5, No. 2 and No. 6, No. 3 and No. 7, No. 4
and No. 8. * Correlation is significant at the 0.05 level (2- tailed) ** Correlation is
significant at the 0.01 level (2-tailed)
we hoped, so we decided to extend the feature with three main focus areas. Instead
of just educating the user, the Guided Tour should also advertise and tutor. The
beginners particularly need more explanation on how to use the new elements and
to be guided to notice them. Finally, as much as it is important to give a broad
choice to the user, it is easy to overwhelm them with too many settings at once.
Thus a cascading model of unfolding possibilities, as the users explore the system
themselves (for instance through setting a global value and then adding exception
list) is much more user-friendly.
6.2 Permission Meter
The solution proposed in Section 5.2 offers the flexibility to zoom in and out to
take control over the privacy settings. The Privacy Meter and Privacy Notifications
give transparency to the users that they were asking for in the first and second
phases of the UID studies. The Privacy Meter can draw attention and provide
positive incentives to modify the usual tendency to ignore the privacy statements. It
should put the user at ease regarding unintended disclosure of sensitive information.
Moreover, every individual is in charge to determine their preference for information
sensitivity. The notification frequency can also be defined, so that there is less of
a chance for annoyance with overactive triggering. Convenience, ease of use and
flexibility for individual preference were the primary considerations and lessons we
learned throughout the process.
Developing the tool in a user-informed manner showed us several things. First,
we did not expect people to ask for such a tool. We were not aware that they would
choose to look into the settings and be curious to see what permissions are used
by which applications. However, as our initial studies showed, people feel out of
control, and such tools give them a sense that they are getting it back, especially
when permissions are explained in an easy-to-understand manner. Second, was the
“On Improving Privacy and Security Through User-Informed Design.”
6.3 Adjustable Location Accuracy (ALA) 79/114
Figure 6.5: Impact of the Privacy Alert on user behavior. We tested 80 people
and how their behavior changed after we started showing them alerts whenever
permissions of their choice were accessed by the applications. We observed that
the people changed the way they use apps and look for more privacy preserving
alternatives if possible.
question of how to build a good model for measuring privacy score. We still have
not succeeded at it and see this as the next challenge for our work. It is unclear
what weights should be put on each permission in order to verify the right algorithm.
However, a study conducted on a group of 80 people showed that an alert triggered
every time a sensitive permission is being accessed, has a positive influence on their
behavior. Users decreased the usage of applications that were requesting those
permissions and reported to seek alternatives that were more privacy-preserving, as
shown in Figure 6.5. It was up to the users to change the notification threshold, so it
is hard to interpret this as an escape from dysfunctional applications. Moreover, we
believe that even if they were simply upset with the level of notifications they were
receiving, the result was very positive: users switched to more privacy-preserving
solutions. An important design requirement in a feature like this is transparency
and trust. If implemented by the OS provider, who also controls the Market Place,
an enforcement mechanism for equal treatment of all apps should be put in place,
so that the ones from the OS provider do not get a better score than the ones from
outside.
6.3 Adjustable Location Accuracy (ALA)
Working with the users from the very beginning gave us a good understanding of
what people expect from a location-obfuscation solution. As with the other PETs,
“On Improving Privacy and Security Through User-Informed Design.”
80/114 Chapter 6 Lessons Learned
users expect privacy-preserving location-based services to run in the background and
somehow “intuitively guess” what they would want to reveal at any given moment.
Consent to share location data with an app heavily depends on the context, be it
spacial or temporal, as presented in [38]. Defining static policies does not follow
those requirements for the system. Thus, when granting access to location, not only
should we ask the question of “if” but also “in what circumstances”. We will now
describe the user studies that have been part of the UID process after we developed a
fully working solution. The contents of this section appears as in the original paper
“Hide, Don’t Seek: on User-Centric Location Accuracy” published at CENTRIC
2016 [119].
6.3.1 User Study Evaluation
The group consisted of 30 participants, 40% males and 60% females fairly equally
distributed between 17 and 60 years old. Their educational background varied:
36,7% graduated from a college, 30% were still in college, 20% had a higher education
level such as a master’s degree. The remaining group had either a high school
diploma, or finished their education one level before that. These show that there was
a bias towards higher education levels. The participants were distributed between
all kinds of jobs: students, graphic designers, secretaries, clerks, legal assistants,
project managers, etc. None of the participants used Firefox OS before, but all of
them previously had a smartphone.
Each participant was invited to the lab, where they were given the phone and
asked to use it as their primary device. They were not informed about the precise
purpose of the study, but we have told them that their behavior would be monitored.
Prior to the study, we obtained their approval and promised the results would be
anonymized and only used for research. We incorporated into the typical first-time-
use run some additional screens about the privacy features included in the modified
system. The Adjustable Location Accuracy was described with the set of screens
presented in Figure 6.6 5. We asked participants to use the Firefox OS phone for
a week. We monitored and made sure that they were using the devices during the
study. When the test period finished, we invited the participants back to the lab
and asked them to give their opinions on the following statements:
1. I am satisfied with the Adjustable Location Accuracy.
2. I feel that using the Adjustable Location Accuracy slows down my phone.
3. I feel that the Adjustable Location Accuracy is a feature that was hard to use.
4. How often did you set up the Adjustable Location Accuracy?
5. Name the setting that you found most useful.
5The figure comes from the original paper [122]c
!IEEE2011
“On Improving Privacy and Security Through User-Informed Design.”
6.3 Adjustable Location Accuracy (ALA) 81/114
For the first three questions, participants were asked to choose one out of seven
possible answers. We based the choice on the Likert scale: 1 = Strongly disagree,
2 = Disagree, 3 = Somehow disagree, 4 = Neither agree nor disagree, 5 = Somehow
agree, 6 = Agree, 7=- Strongly agree [16]. Question number 4 had the following
possible answers: 1 = I did not use the feature at all, 2 = I have tried to set up
the feature and resigned, 3 = I have set up the feature once, 4 = I have set up the
feature repeatedly. In this question, we asked the users to justify their answers in
free-text response. The last question was multiple choice (limited to 3) with the
following possibilities: 1 = I did not find Adjustable Location Accuracy useful at
all, 2 = No location, 3 = Precise Location, 4 = Custom Location, 5 = Adjustment,
6 = Per-application settings, 7 = Global setting, 8 = Privacy Profiles. We also left
space for free-text comments.
The first question was measuring the general satisfaction with the solution.
80% of the participants chose answers 6 and 7, which means they were highly satisfied
with the functions provided, 13% chose “Somehow agree”, and 7% were ambiguous
about it (answer 4). None of the users reported lack of satisfaction.
For the second question, we saw only answers that did not suggest any delays:
83% said that they strongly disagree with the statement, while 17% chose option 2.
With the third question we wanted to verify the perception of our tool. While
most users said that they did not find the solution hard (40% Strongly disagree,
26.7% Disagree, 26.7% Neither Agree nor Disagree, 6.6% Somehow agree), in the
comments section they often mentioned that the process of setting up the tool was
time-consuming.
However, when looking at the answers to the fourth question, we noticed that
most users set up the process only once (20 people). 8 participants chose answer
number 4, while only 2 participants did not use the feature at all. They argued they
“did not feel like it improved anything”. We did not verify if this was due to bad
explanation in the Guided Tour or a different privacy expectation.
Lastly, the users mostly chose the “Privacy Profiles”(83.3%), “Custom Lo-
cation” (76.7%), “Per application settings” (66.7%). These were followed by the
“Global Settings” (40%).
Overall we saw that the participants were happy to use the feature and did
not notice any performance problems. We have concluded that users are happy to
adapt PETs as long as they do not have to give up usability. Because PETs rarely
offer users more features or a better experience, the best way to drive adoption is
by making them both transparent and automatic. There is more to be done on the
topic of user studies and user-perception of location accuracy. We plan to deeply
investigate the topic, because the results to the last question were very interesting
to us. We feel that our solution would benefit from an automated setting based
on recommendations made by trusted people. However, this might not improve the
privacy, as [167] suggests.
“On Improving Privacy and Security Through User-Informed Design.”
82/114 Chapter 6 Lessons Learned
GT-3
GT-2
GT-4
GT-5
Figure 6.6: The wireframes presented to the participants of the user study per-
formed to evaluate the adaptation of our framework. At the beginning of the study
all participants took the Guided Tour that explained what the feature is about and
how can they set it. We have provided the users with both a short description of
main points of the feature as well as the reason why is it important to decrease the
precision of the location data provided to the applications.
6.3.2 Improvements
The location obfuscation does not include any overhead compared to the lack of the
feature, and it does not cause the apps to break. What is interesting is that while
some applications require the user to grant permission to access the geolocation data
from the device in “no location” mode, the app receives access but doesn’t get any
location data that is useful for tracking the user. This empowers the user to gain
access to the app without compromising their privacy. This is possible, because
the applications usually set up a callback to handle the location information once
it is obtained from the system. When the user selects “no location”, no data is
returned and no callbacks are made. If the application is designed to handle location-
information updates asynchronously, it usually continues to work and simply uses
the IP-based position. The application will typically have a hierarchy of geolocation
sources and the API based is highest in the rank. Thus with other settings, such
as “Adjustment” or “Custom Location”, the application will just accept the data
returned by the Geolocation API. To ensure higher protection we would need to add
some IP hiding.
The biggest problem of any location-obfuscation solution is the IP-based at-
tack. We propose to fight against it by using a Tor-based solution6. To do that,
some changes in the current implementation of Tor need to be made: the exit nodes
should comply with the region chosen by the user location., which decreases the
anonymity level provided by Tor. However, in the case of ALA we using Tor for
higher anonymity, but as a proxy to match the IP address with the location provided
6www.torproject.org
“On Improving Privacy and Security Through User-Informed Design.”
6.3 Adjustable Location Accuracy (ALA) 83/114
by ALA. Assume there are two web sites, A.com and B.com, that both use tracking
cookies/images from C.com. If the user visits A.com with Tor and location obfusca-
tion, and then B.com without Tor and location obfuscation, C.com’s tracking will see
two different IP addresses and locations for the user. If the user then visits a third
site with C.com tracking bits on it, C.com will see the same IP address and location
data it saw when the user visited B.com. C.com can develop a statistical model for
the location and IP address of the user that will discount the masked IP address and
fuzzed location reported when accessing A.com. This problem can be addressed by
developing a cookie-management method for the obfuscation techniques. Currently
we believe that the best way this attack can be defeated is by applying the cookie
policies found in the Tor Browser Bundle (TBB) and ensuring that all the trafficis
forwarded through Tor. We strongly encourage better investigation of how Tor tech-
nologies can be incorporated into industry solutions. While this would mean a huge
increase of traffic on the nodes, which requires better infrastructure backed with
funding, if used in enterprise setting, there would be an incentive for the companies
to pay for that infrastructure.
The W3C geolocation API takes into consideration the impact on user privacy
in Section 4 of the specification7. It states that “a conforming implementation
of [the] specification must provide a mechanism that protects the user’s privacy...”.
However, they do not specify how this should be done. Our implementation attempts
to further improve basic geolocation with user-selectable behavior of the geolocation
discovery and reporting mechanism. Additionally, we introduce the possibility of
setting a custom chosen, fixed position. We believe that it should be an enhancement
of the specification to introduce both the “no location” and “fixed position” choices.
We do not suggest working on the fuzzing mechanism, because this has not been
proven to be secure. For “custom location” there should be a requirement to allow
the user to set the position to a defined value. This would comply with the request of
respecting privacy that is not currently enforced. For “no location” we would like to
introduce a constant value. When a user selects the behavior in our implementation,
we do not execute any position callbacks to the client application. In some cases,
web apps interpret the lack of a callback as an error condition. This would not be
the case if a constant was present that would define what “no location” is. Then we
could execute the callback with that value and avoid the incorrect error condition.
We propose that “no location” be represented by a position value of NaN for the
coordinate latitude, longitude, and accuracy as well as a timestamp value of 0.
7http://dev.w3.org/geo/api/spec-source.html
“On Improving Privacy and Security Through User-Informed Design.”
85/114
Chapter 7
Beyond User-Informed Design (UID)
We learned a lot from the user-informed approach. It gave us a new perspective on
how to develop tools and improve existing solutions; how to think of users as our al-
lies not enemies. Because, let’s face it: often as technical experts and researchers we
neglect them as the “thing that we have to deal with" rather than “an enhancement
to our security stack". With the background of UID, there are more things that can
be done. In this chapter I discuss a few applications of the ideas that came from
our work. First, we decided to apply the model that was described in Section 3.2
to evaluate the state of the security of wearables and the IoT. Second, we used the
event-centric and context-aware authentication framework discussed in Section 5.3
to create a Mobile Device Management (MDM) solution.
7.1 Mobile Device Management (MDM)
Management of mobile devices and the creation of business-oriented solutions presents
unique challenges. Within an organization, different players have different security
concerns to ensure data confidentiality, integrity, availability and authenticity. On
an organizational level, the changing circumstances of device user and device-usage
context raise the need for the fine-grained management of discrete events. A wide
range of device types host and access data from the internal network. Each has dis-
tinct resources that need protection. The ubiquitous use of mobile phones introduces
new attack vectors to an organization’s data. In addition, the same machines are
used both for personal and business reasons. On the technical side, connectivity is
often intermittent and phones use multiple data networks. Also, mobile devices have
limited computing power, battery life, screen size and human-interface capabilities.
7.1.1 Implementation
Our framework addresses these issues by supporting, from conception to enforce-
ment, the design of policies that fit the organization’s human resources, technology,
environment and compliance requirements. To illustrate the transition from concept
to application, the process for implementation planning, device deployment and on-
going operation we have decided to implement our solution in practice, for the most
common authorization use case: MDM.
Planning the implementation includes defining the policy hierarchy and gov-
ernance of policy authors. Policy-hierarchy governance is the relationship between
“On Improving Privacy and Security Through User-Informed Design.”
86/114 Chapter 7 Beyond User-Informed Design (UID)
policy sets and authorizations in each policy set. At the highest level, the root-
policy owner sets up the policy hierarchy, as shown in Figure 5.13. Policy-hierarchy
governance integrates the definition of protected resources, the policies that control
them, and the individuals authorized to do so.
For example, in the case of a mobile phone, the telecommunication carrier has
interests that are distinct from those of the enterprise that owns the phone. Within
the enterprise, there may be corporate-level security concerns as well as issues that
need to be managed at a departmental level. The carrier policies might be concerned
with rogue applications and the integrity of the device itself and the network. At
enterprise and departmental levels, the concern may be with the exclusive use of
approved and vetted applications, sensitive documents, phone blacklists and the
restriction of certain activities to times, locations and specific user roles or group
memberships.
Policy author governance comprises defining roles, abilities per role and users,
some of whom are PDK users. In this MDM use-case example, we define the follow-
ing roles:
•Root-policy owner: Manages policy-authoring hierarchy and rights (including
delegation) for users lower down in the hierarchy tree. This hierarchy could
take into consideration the organization’s own governance structure.
•Policy author: Creates policy models by identifying protected resources and
the conditions under which they should be available. Uses the models to create,
verify and publish policies to be enforced on devices. Different policy authors
may be designated to manage policies for different protected resources.
•PDP manager: Oversees the PDP deployment.
•Data administrator: Configures the PIPs or databases where the data resides.
•Device administrator: Enrols mobile devices by deploying the PEPs to them
and making suitable PIP entries.
On the device, every controlled resource must be guarded by a PEP. On de-
vices capable of diverse types of events, there might be many PEPs. On simple
phones, there might only be one. PEPs may take the form of applications from
the market, applications developed in house, drivers installed by the OEM, or other
forms. The PEP implementation depends on the resource it protects. In any case,
the root-policy owner uses the PDK to define the other users and roles, and cre-
ate an environment in which they can work. The data manager confirms that the
necessary information is available in the PIP. The device administrator manages
the registration of devices and PEP deployment. When this is complete, the policy
author implements the policy design by:
1. Identifying the attributes used for policy evaluation, including the protected
resource, device types, users and any conditions determining policy scope such
as time and location.
“On Improving Privacy and Security Through User-Informed Design.”
7.1 Mobile Device Management (MDM) 87/114
2. Identifying the validity duration for which a policy decision can be cached on
the PEP and deciding which events are to be logged.
3. Configuring the policies and delegation of policy-authorship rights.
4. Testing the policy scope and results and, when ready, publishing the policy in
a deployed PDP where the policy becomes active.
At this point, PEPs are installed and communicating with an active PDP, so the
devices are managed according to the policy.
7.1.2 Lessons Learned
A comparison with other policy-management approaches such as traditional MDM,
firmware-based device isolation and APIs enabled by system providers, shows that
such approaches provide limited policy-design flexibility and lack awareness of the
context in which events occur when enforcing a policy. These solutions work jointly
with device APIs to provide management features such as application management
and network-usage authorization. They provide the ability to create application
whitelists and blacklists, restrict native applications and delegate application man-
agement to different administrators aiming for role-based access control. Some per-
mit IT administrators to apply controls based on parameters such as device type,
device ownership and employee role. As compared to our solution, these do not
provide a policy framework that permits the definition of fine-grained events that
are allowed or denied depending on the context in which the events occur; they are
not context-aware.
Most solutions are limited to those devices where they are supported. They
seek data security through absolute isolation of spaces that enable the coexistence
of a business environment hosting sensitive data, and a personal environment on the
same physical device. The business environment can be locked down to restrict the
authorized applications. The personal environment is open and unmanaged. This
approach treats the device as two logical devices with separate data, and applies the
previous solution to each. The same limitations apply as noted above; it is configured
statically and is not context-aware. The solution presented here has the advantage
of being context-aware, which allows for better management of business and leisure
time. It is not divided into two environments, making it more user-friendly.
Traditional MDM solutions rely upon the features enabled by OS manufactur-
ers that provide hooks to integrate devices to an organization’s MDM. These do
not provide an entire framework for policy management. Such solutions permit en-
abling and disabling features on a device. In contrast to our proposal this is a static
approach that does not take into account the real-time circumstances in which the
events occur. The context-aware policy framework described here not only permits
the definition of blacklisted or whitelisted applications, but also provides general-
ized resource-access control based on a wide set of conditions. These conditions may
include date or time, user sign-on status, device-network connection at the time of
“On Improving Privacy and Security Through User-Informed Design.”
88/114 Chapter 7 Beyond User-Informed Design (UID)
application execution, or other conditions the organization considers relevant. The
unrestricted specification of policy enables authors to decide the relevant context
and access-control scheme. The hierarchy and delegation allows stakeholders to
collaborate, and addresses the complexity of large systems. The real-time assess-
ment of events and the conditions in which events occur provides real-time policy
decision-making and enforcement.
“On Improving Privacy and Security Through User-Informed Design.”
89/114
Chapter 8
Conclusions
Today’s mobile devices have the ability to know almost every part of our daily
routine: when we wake up, what we eat, where we sleep, whom we meet and flirt
with, what we see and hear, our secrets. People cannot go long without their devices.
Contrary to what we often hear, the data is very encouraging: 97% of users agree
that privacy is a top concern on a mobile device. In fact, the vast majority think
it is at least important to know what type of information is being collected, and
to control how the personal information is shared, as is shown in Figure 8.1 which
was adapted from the Lookout Mobile Privacy IQ Study1. However, among this
group, only 52% actually reads any privacy policy on a mobile device. Let’s face
it, convenience and usability will always win; almost two thirds of people are more
willing to give up chocolate than the mobile Internet [150]. Finally, 75% of the apps
in the Android Play Store require at least one of the permissions that have impact
on the privacy: location, device ID, camera, contacts, other user accounts, call logs
or even microphone, messages and calendar [41].
The bottom line is that people do care. They also have no clue what is hap-
pening on their devices. If they are given a binary choice of privacy versus comfort,
they will choose the latter. It is unfair for us, the researchers to expect them to be
technically savvy and understand the technology. Why should they? There are more
mobile phones than people around the world. Among them, only a small percentage
are well educated. Those with an engineering degree are just a fraction of that.
Even having a degree does not mean anything. How many times have we failed to
make a secure connection through some encrypted channel and simply went back to
the plain-text e-mail or a phone call thinking that, sure, it is not secure, but what
are the odds that something bad will happen?
8.1 Summary of the Work Presented
In this work, I argued that no matter how much effort we put into protecting the
device, unless we start listening to the receiving side of our technology, the people, we
will not succeed in protecting user privacy. To this end I presented a comprehensive
argument consisting of several stages. First, I reviewed the literature and described
two real-life attacks. In one, we used the identical scene rendering mechanism and
performed a copy operation with a GPU on an embedded device. In the other, we
1https://www.lookout.com/resources/reports/mobile-privacy-iq
“On Improving Privacy and Security Through User-Informed Design.”
90/114 Chapter 8 Conclusions
Figure 8.1: Answers to the three questions evaluating how people feel about privacy.
The results of a study by Lookout show clearly that 97% of users agree that privacy
is a top concern on a mobile device.
abused some more flaws and were additionally able to read the data offa mobile
device. We showed, as many others before us, that keeping large parts of a driver
stack as closed source is, from a security perspective, no replacement for a good
kernel–user interface. Next, I moved from destroying to analyzing and building.
I provided a comprehensive understanding of the FirefoxOS security architecture
and pointed out the shortcomings in its security design. I then gave an outlook
to the possible future attacks that the system will face. Although the project has
been abandoned by Mozilla, we have seen some of our predictions coming true. I
moved on to build a model for evaluating the new systems. Evolution of the device’s
form factor, connectivity and the sensor technologies creates various attack vectors
specific to the device. When a shift was made from laptops to mobile devices, some
very specific elements played a role in defining the new threat model: the network
became a portable resource, the environment became extremely sensor-rich, and we
were able to correlate the personal data and the location tracking. Later, in my
work beyond the UIP we were able to apply that model to analyze the security of
wearables and the IoT. In summary, we found that together with more pervasiveness,
wearables introduce problems such as lack of data-storage diversification, payment
information, usability of small form factor, and a lack of transparency. Additionally,
when connected into the IoT, the threats are increased by multiplying the risks
and introducing new ones to the devices that previously did not suffer from them.
Finally, an observation was made that with the growing number of elements that
“On Improving Privacy and Security Through User-Informed Design.”
8.1 Summary of the Work Presented 91/114
form the network of things, the impact of any given attack grows.
All this work culminated in describing a new technique for developing privacy
tools: User-informed Design (UID), where we included our users in the initial stage
of considering what they might need, instead of treating them like experts who
evaluate what is already built. We realized that people do not usually have enough
experience and knowledge to provide sufficient feedback on already created tools.
It is wrong to expect them to help engineers figure out what is wrong. But they
can be introduced as everyday experts, telling us what they need, or rather allowing
us to evaluate the gap between their perception of how they think a mobile device
works and the way it actually works. We begun our work by doing a wide-scale
study to verify and judge the current state of privacy awareness of an average user.
As an outcome of this study we had an overview of what people expect, know
and hope for. Next, having defined the gap between users’ expectations and the
capabilities that the devices have, we designed and developed a prototype to meet
these requirements. Having the initial solution, we conducted another study to
analyze how the prototype fits the users’ needs. Based on that, we created a second
iteration of improvements and the final version of the tool. The feedback from the
users showed that the adaptation rate of the developed features was very high. It
also led us to change the presentation of the Location Accuracy and Secondary-user
mode as well as the layout of the Guided Tour. By using the UID, we were able
to get a better understanding of what people need, and make sure that our work
is not only of a high academic quality, but can also be introduced on a product
scale, and will be easily adapted. I believe that involving users in the research and
design phase, and then building solutions that meet their needs is crucial to bring
the academic world closer to the people.
Out of the first study came the Privacy Dashboard, a tool that addresses
four basic concerns of users: disclosure of their location, sharing data with people
who temporary use their phone, requirement of registering in a central data-base in
order to have a “Find My Device" feature, and the lack of choice regarding backup
methods. Looking at our users’ needs, we implemented and studied two of the ideas
that were part of the Privacy Dashboard: the Permission Control and Adjustable
Location Accuracy. We also created an Access Control mechanism that takes into
account the users’ context. We implemented a Mobile Device Management scheme
that illustrated the contextual awareness of the framework, because it enables control
of a device’s behavior while taking into account the conditions in which events occur.
The granularity of the policy scope can be determined by the business needs: a
policy can be very general for organization-wide applications, or individually-crafted
for special circumstances. The flexible and fine-grained management of discrete
events by the framework provides the ability to define, program, and enforce device-
management policies designed to meet the requirements of the stakeholders.
“On Improving Privacy and Security Through User-Informed Design.”
92/114 Chapter 8 Conclusions
8.2 Action Points for the Future
We identified several action points for academia and industry. The first is better
support of user-informed development of privacy tools in general, especially location
privacy-preserving solutions. We strongly believe that this is the only way to get
a higher adaptation rate of the PETs. The second is improving the current W3C
geolocation standard by introducing the “no location” solution. Additionally, the
geofencing document should be designed with greater care; its privacy section was
written as the last part. The biggest lesson learned from working with users on
PETs is seeing that they care about their data and protecting it, yet they seek
solutions that are transparent and intuitive. I believe that it is the role of the
standardization bodies and academia to work together creating standards that would
ensure exactly that. Naturally there is much to gain from such massive analysis of
personal information or big data [64], not least of which is that it allows the business
entities to offer better services through consumer-centric analysis. However, there is
a certain trade-offintroduced by giving away personal information: losing anonymity
which has the potential to initiate discrimination [10]. It is interesting to see how the
world changes. Even with millions of photos taken in 2005, only a small percentage
was uploaded online. At the end of 2014, almost 1.8 billion pictures are uploaded and
shared per day through Facebook, Flickr, Snapchat, Instagram and WhatsApp [86].
Of course, with good pattern-recognition techniques we should think about mass
surveillance (analysis of people in the background), discrimination [32], or even loan
evaluation. The line is very fine, though, as we all define the privacy threshold
differently. The solutions describing the planned usage of PII through Terms and
Conditions have little-to-no effect [132]. Therefore, it is important to touch on the
question of transparency. While security is a problem that can be boiled down to
meeting a certain standard, sometimes the best we can do in terms of privacy is
to be clear about what happens and when on the device. Despite all the research,
we are lacking mechanisms that inform users that their data is being collected and
uploaded in real time.
8.3 Future Research
It is my goal to explore what can be done to better understand how users make
their privacy decisions. If we think about buying a product, say a mobile phone,
the first step is to choose a device; will it be an Apple Phone, Android Phone or
something else? Then a model, manufacturer, memory size, color, and maybe some
more customization. The way these decisions are made–how the presented options
influence those decisions, is what behavioral economics is about. A famous paper by
Amos Tversky and Daniel Kahneman in 1979 [79] proposed prospect theory, showing
that decisions are rarely rational and calculated. They have argued that customers’
willingness to take risks is context-dependent. The term “bounded rationality” has
been proposed by Herbert Simon [145]. It is based on the idea that people only have
“On Improving Privacy and Security Through User-Informed Design.”
8.3 Future Research 93/114
so much cognitive power and time, and therefore are bound to work by rule of thumb.
They are not rational and tend to be overconfident, optimistic and extrapolate, as
a way to economize on the lack of brainpower [154].
I believe that similar techniques and theories could be leveraged into a field I
would like to call “behavioral security and privacy”. Most prominent research com-
bining economics and privacy has until now been done by a group led by Professor
Alessandro Acquisti. He focuses his research on the Economics of Privacy, namely:
what is the cost of privacy, what are the interactions with technologies, and whose
responsibility it is to protect user privacy? However Acquisti and some of his collab-
orators have also proposed bringing the lessons learned from behavioral economics
to the field of privacy. The works include: “What Can Behavioral Economics Teach
Us About Privacy?” [12], “Nudging Privacy. The Behavioral Economics of Personal
Information” [6], and “From the Economics to the Behavioral Economics of Privacy:
A Note” [7]. The idea is to investigate Kahneman’s explanation of brain work:
system 1 which is fast, automatic, frequent, emotional, stereotypical, subconscious,
and system 2 which is slow, effortful, infrequent, logical, calculating, conscious. It is
interesting that, in his book, Kahneman shows how the two ways of thinking bring
us to different conclusions given the same input. His concepts have had extreme im-
pact on market research. I would like to suggest applying them to the development
of privacy-preserving tools. “Thinking Fast, Thinking Slow” discusses the biases of
the human brain and why people struggle to make rational, statistics-based deci-
sions: because System 1 thinking involves associating new information with existing
patterns, or thoughts, rather than creating new patterns. We are prone to the an-
choring effect, which is a tendency to be distracted by irrelevant numbers, and to the
availability of information: whatever one can recall is the thing that is most likely
to be true, and by replacement of a more complex problem with a simplified one.
Additionally, we tend to be distracted by the context in which choices are presented,
known as the framing effect. Often, people forget that there are things they do not
have information about, and base their actions only on the things they do know —
they are overly optimistic and loss-averse. Finally, we follow the sunk-costs rule, by
which Kahneman means that we continue with things we have already invested in,
even when the perspective is less promising.
These rules of perceiving the world and decision-making models could be ap-
plied and evaluated when talking about privacy. Is it not intuitively correct that
users are overly optimistic about the safety of their data? How many times have
we heard “Why would they be after me? I have nothing to hide.”? Is it not intu-
itively correct that people make decisions about their privacy settings based on the
latest news in media? It is enough to look at the number of Tor-browser downloads
post-“Snowden”. I believe that this is the next step towards creating better, more
intuitive and interesting tools for protecting privacy. In a world full of choices, it
is a responsibility of technology to adapt to the people, not for the users to adapt
to the technology. The best solutions will not win just because they are most tech-
nically sound. They will win because they are striking the right balance. It is in
the interest of us all to find this balance in order to create truly secure devices —
“On Improving Privacy and Security Through User-Informed Design.”
LIST OF FIGURES 97/114
List of Figures
1.1 Device Security Stack .......................... 2
2.1 A typical System on Chip (SoC) that can be found in a smartphone 11
2.2 A typical graphics-processing pipeline (GPP) ............. 12
2.3 The attack vector ............................. 13
2.4 Android soft-keyboard with a currently active tooltip for the letter “j”. 14
2.5 This figure depicts the chain of events for each single key that is
logged. (a) After a key has been pressed on the soft-keyboard, a UMP
buffer is attached and the corresponding bitmap for the pressed key
is put into the buffer. (b) The buffer is attached by the GPU, which
then (c) uses it to render the full scene onto the display. Then in (d),
while the buffer created in (a) is still present, the keylogger attaches
it. (e) The Keylogger creates a MD5 sum of the buffer contents, sends
these out and releases the buffer. .................... 16
2.6 The payload as 4-by-4 texture padded with 6 words of 0 when inter-
preted as RGBA pixel format ...................... 17
3.1 The review process in FxOS ....................... 22
3.2 Comparison of security features on Android, iOS and FxOS ..... 27
3.3 Model for security evaluation of a new device. We base the analysis
on similarities with previously introduced devices and the assessment
spiral of identifying assets, threats, vulnerabilities, attacks, risks, de-
fenses and mitigations on various levels of the stack .......... 33
3.4 The cycle of creating a security enhanced solution. .......... 33
4.1 User-Informed Design (UID) process .................. 38
4.2 User-Centered Design (UCD) according to the ISO 13470 standard. . 39
4.3 Proposal of combining UCD and UID to enhance both ........ 40
4.4 The concept of the introductory study and the key results of each
step for the User-centric Design (UCD) process ............ 41
4.5 First two wireframes of the Guided Tour. Each panel consists of the
topic, a title and a short explanation. Instead of simply describing
the feature, the Guided Tour gives the consequences of the decision. 45
5.1 Flow of the Extended Location Settings ................ 50
5.2 Fixed location setting. The user can either choose from the predefined
list of regions and major cities, or input their own coordinates. The
location reported to the app will always be set to the defined values. 51
“On Improving Privacy and Security Through User-Informed Design.”
98/114 LIST OF FIGURES
5.3 Approximating the position with a grid algorithm ........... 52
5.4 An instance of the matrix showing theoretical structure of the solution. 55
5.5 An element of the matrix showing moderately detailed information . 56
5.6 Elements of the Privacy Dashboard and its Transparency Control
framework. Sliding bar allows us to define the permission priority
according to individual preference, while risk overview for an app
depicts overall privacy concern ..................... 56
5.7 The risk overview for an app depicts overall privacy concern ..... 57
5.8 Privacy risk bar for each of the permissions visualizes easily perceiv-
able consequence. ............................. 58
5.9 Framework Overview. The Policy Decision Kit (PDK) captures the
user’s design intent at Level 1, and generates an instance of the PDP
at Level 0. ................................. 60
5.10 Framework Level 0 Architecture. The Level 0 architecture consists of
four components: Policy Enforcement Point (PEP), Policy Decision
Point (PDP), Policy Information Point PIP), and the Fabric (depicted
by the double-headed arrow). ...................... 61
5.11 Policy enforcement. The Policy Enforcement Points (PEPs) are re-
mote agents located on the managed device that are triggered by
access requests. PEPs query the PDP for authorization and enforce
the response. ............................... 62
5.12 Policy Decision Point (PDP). The PDP is a server that issues re-
sponses to queries from PEPs. Its responses are based on PDP-
specific policies. PEPs and PDPs communicate through a two-way
channel. .................................. 63
5.13 Policy Ownership Tree. Policy stakeholders are arranged in a hier-
archy, each responsible for their own domain and allowed to delegate
authority to those beneath them. .................... 67
5.14 Policy Design Kit (PDK). The PDK is a policy life-cycle management
framework with a set of tools for the policy administrator to conduct
policy authoring, introspection of policy sets, PDP generation, sanity
checking and deployment. ........................ 68
6.1 Categories and Features. The figure presents the way participants
categorized each of the features included in the Privacy Dashboard.
Not all of them were considered privacy oriented ........... 74
6.2 Learnability, performance and satisfaction of Privacy Features. The
dots represent the outliers, the stars represent the extreme scores. . 75
6.3 Ease of use of the Guided Tour. Users were asked to state if it was
“Easy to understand”, “Helpful” and “Worth reading”. The dots rep-
resent the outliers, the stars represent the extreme scores. ...... 76
“On Improving Privacy and Security Through User-Informed Design.”
LIST OF FIGURES 99/114
6.4 Correlation between the answers given before (to statements 1 to
4) and after going through the Guided Tour (statements 5 to 8).
Compared were similar statements: statement No. 1 and No. 5, No.
2 and No. 6, No. 3 and No. 7, No. 4 and No. 8. * Correlation is
significant at the 0.05 level (2- tailed) ** Correlation is significant at
the 0.01 level (2-tailed) .......................... 78
6.5 Impact of the Privacy Alert on user behavior. We tested 80 people
and how their behavior changed after we started showing them alerts
whenever permissions of their choice were accessed by the applica-
tions. We observed that the people changed the way they use apps
and look for more privacy preserving alternatives if possible. ..... 79
6.6 The wireframes presented to the participants of the user study per-
formed to evaluate the adaptation of our framework. At the begin-
ning of the study all participants took the Guided Tour that explained
what the feature is about and how can they set it. We have provided
the users with both a short description of main points of the feature
as well as the reason why is it important to decrease the precision of
the location data provided to the applications. ............ 82
8.1 Answers to the three questions evaluating how people feel about pri-
vacy. The results of a study by Lookout show clearly that 97% of
users agree that privacy is a top concern on a mobile device. ..... 90
“On Improving Privacy and Security Through User-Informed Design.”
BIBLIOGRAPHY 101/114
Bibliography
[1] Directive on privacy and electronic communications. Technical report, Euro-
pean Parliament, 12 July 2002.
[2] Quantifying the effect of co-location information on location privacy. In
E. De Cristofaro and S. Murdoch, editors, Privacy Enhancing Technologies,
Lecture Notes in Computer Science. 2014.
[3] Reflective thoughts on the potential and challenges of wearable technology for
healthcare provision and medical education. International Journal of Infor-
mation Management, 35(5):521–526, Oct. 2015.
[4] Definition of Privacy. Oxford Dictionary, 2016.
[5] O. H. Abdelrahman and E. Gelenbe. Signalling Storms in 3G Mobile Networks.
IEEE international Conference on Communications (ICC), 2014.
[6] A. Acquisti. Nudging privacy: The behavioral economics of personal informa-
tion. IEEE Security Privacy, 7(6):82–85, Nov 2009.
[7] A. Acquisti. From the Economics to the Behavioral Economics of Privacy: A
Note, pages 23–26. Springer Berlin Heidelberg, Berlin, Heidelberg, 2010.
[8] A. Acquisti, I. Adjerid, and L. Brandimarte. Gone in 15 seconds: The limits
of privacy transparency and control. IEEE Security and Privacy, 11(4):72–74,
July 2013.
[9] A. Acquisti, L. Brandimarte, and G. Loewenstein. Privacy and human behav-
ior in the age of information. Science, 347(6221):509–514, 2015.
[10] A. Acquisti and C. M. Fong. An experiment in hiring discrimination via online
social networks. Social Science Research Network Working Paper Series, Apr.
2012.
[11] A. Acquisti and J. Grossklags. Privacy and rationality in individual decision
making. IEEE Security and Privacy, 3(1):26–33, Jan. 2005.
[12] A. Acquisti and J. Grossklags. What can behavioral economics teach us about
privacy?, 2006.
[13] I. Adjerid, A. Acquisti, L. Brandimarte, and G. Loewenstein. Sleights of
privacy: Framing, disclosures, and the limits of transparency. In Proceedings
of the Ninth Symposium on Usable Privacy and Security, SOUPS ’13, pages
9:1–9:11, New York, NY, USA, 2013. ACM.
“On Improving Privacy and Security Through User-Informed Design.”
102/114 BIBLIOGRAPHY
[14] D. Akhawe and A. P. Felt. Alice in warningland: A large-scale field study of
browser security warning effectiveness. In Proceedings of the 22Nd USENIX
Conference on Security.
[15] F. Alecu and P. Pocatilu. Enabling the ciphering indicator on android. Journal
of Mobile, Embedded and Distributed Systems, 6(2):52–60, 2014.
[16] I. Allen and C. A. Seaman. Likert scales and data analyses,. Quality Progress,
vol. 40, no. 7, 2007.
[17] Apple Inc. Use Siri on your iPhone, iPad, or iPod touch - Apple Support.
[18] J. Aron. Touchscreen keylogger captures details from your taps. New Scientist,
211(2825):21, 2011.
[19] P. Attfield, P. Chenard, M. Piekarska, J. Narvaez, M. Hendrick, and S. Curry.
Can i let you in? on event-centric context-aware authorization. In The
Eleventh International Conference on Systems, ICONS 2016, February 21 -
25, 2016 - Lisbon, Portugal, ISBN: 978-1-61208-451-0, 2016.
[20] K. W. Y. Au, Y. F. Zhou, Z. Huang, and D. Lie. Pscout: Analyzing the
android permission specification. In Proceedings of the 2012 ACM Conference
on Computer and Communications Security, CCS ’12, pages 217–228, New
York, NY, USA, 2012. ACM.
[21] A. J. Aviv, B. Sapp, M. Blaze, and J. M. Smith. Practicality of accelerometer
side channels on smartphones. In Proceedings of the 28th Annual Computer
Security Applications Conference, pages 41–50. ACM, 2012.
[22] C. P. Barnard. Hijacking user uploads to online persistent data repositories
for covert data exfiltration. Technical report, DTIC Document, 2010.
[23] S. J. Barnes. Mobile commerce. chapter The Wireless Application Proto-
col: Strategic Implications for Wireless Internet Services, pages 145–161. IGI
Global, Hershey, PA, USA, 2003.
[24] D. Barrera, H. G. Kayacik, P. C. van Oorschot, and A. Somayaji. A method-
ology for empirical analysis of permission-based security models and its appli-
cation to android. In Proceedings of the 17th ACM Conference on Computer
and Communications Security.
[25] M. Benisch, P. G. Kelley, N. Sadeh, and L. F. Cranor. Capturing location-
privacy preferences: Quantifying accuracy and user-burden tradeoffs. Personal
Ubiquitous Comput.
[26] A. R. Beresford, A. Rice, N. Skehin, and R. Sohan. Mockdroid: Trading
privacy for application functionality on smartphones. HotMobile ’11.
“On Improving Privacy and Security Through User-Informed Design.”
BIBLIOGRAPHY 103/114
[27] D. Biswas, I. Aad, and G. P. Perrucci. Privacy panel: Usable and quantifiable
mobile privacy. In Proceedings of the 2013 International Conference on Avail-
ability, Reliability and Security, ARES ’13, pages 218–223, Washington, DC,
USA, 2013. IEEE Computer Society.
[28] A. Boileau. Hit by a bus: Physical access attacks with firewire. Ruxcon, 2006.
[29] H. Bojinov, E. Bursztein, and D. Boneh. Xcs: Cross channel scripting and its
impact on web applications. In Proceedings of the 16th ACM Conference on
Computer and Communications Security.
[30] R. Borgaonkar. Dirty use of ussd codes in cellular network. https:
//www.troopers.de/wp-content/uploads/2012/12/TROOPERS13-Dirty_
use_of_USSD_codes_in_cellular-Ravi_Borgaonkor.pdf.
[31] R. Breuk and A. Spruyt. Integrating dma attacks in exploitation
frameworks. http://staff.science.uva.nl/~delaat/rp/2011-2012/p14/
report.pdf, 2012.
[32] T. Burghardt, A. Walter, E. Buchmann, and K. Böhm. Primo - towards
privacy aware image sharing. In Proceedings of the 2008 IEEE/WIC/ACM
International Conference on Web Intelligence and Intelligent Agent Technology
-Volume03, WI-IAT ’08, pages 21–24, Washington, DC, USA, 2008. IEEE
Computer Society.
[33] L. Cai and H. Chen. Touchlogger: Inferring keystrokes on touch screen from
smartphone motion. In Proceedings of the 6th USENIX Conference on Hot
Topics in Security, HotSec’11, pages 9–9, Berkeley, CA, USA, 2011. USENIX
Association.
[34] L. Cai and H. Chen. On the practicality of motion based keystroke inference
attack. In Trust and Trustworthy Computing, pages 273–290. Springer, 2012.
[35] J. Carlson. Gpus for mobile malware mitigation and more. Recon, 2012.
[36] B. D. Carrier and J. Grand. A hardware-based memory acquisition procedure
for digital investigations. Digit. Investig., 1(1):50–60, Feb. 2004.
[37] T. Chen, M. A. Kaafar, and R. Boreli. The where and when of finding new
friends: Analysis of a location-based social discovery network. In THE 7TH
INTERNATIONAL AAAI CONFERENCE ON WEBLOGS AND SOCIAL
MEDIA (ICWSM), 2013.
[38] S. Consolvo, I. E. Smith, T. Matthews, A. LaMarca, J. Tabert, and
P. Powledge. Location disclosure to social relations: Why, when, & what
people want to share. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems, CHI ’05, pages 81–90, New York, NY, USA,
2005. ACM.
“On Improving Privacy and Security Through User-Informed Design.”
104/114 BIBLIOGRAPHY
[39] M. Conti, V. T. N. Nguyen, and B. Crispo. Crepe: Context-related policy
enforcement for android. ISC’10.
[40] J. Danisevskis, M. Piekarska, and J.-P. Seifert. Dark side of the shader: Ad-
vanced gpu-aided payload delivery.
[41] Data Protection Commissioner. Global privacy sweep raises concerns about
mobile apps.
[42] R. Dayarathna. Discovering Constructs and Dimensions for Information Pri-
vacy Metrics. PhD thesis, Stockholm University, Department of Computer
and Systems Sciences, 2013. At the time of the doctoral defense, the following
paper was unpublished and had a status as follows: Paper 6: Accepted.
[43] D. DeFreez, B. Shastry, H. Chen, and J.-P. Seifert. A first look at firefox os
security.
[44] W. Diao, X. Liu, Z. Zhou, and K. Zhang. Your Voice Assistant is Mine: How
to Abuse Speakers to Steal Information and Control Your Phone. The 4th
Annual ACM CCS Workshop on Security and Privacy in Smartphones and
Mobile Devices, 2014.
[45] A. Distefano, A. Grillo, A. Lentini, and G. F. Italiano. Securemydroid: En-
forcing security in the mobile devices lifecycle. In Proceedings of the Sixth
Annual Workshop on Cyber Security and Information Intelligence Research.
[46] M. Dornseif. 0wn3d by an ipod: Firewire/1394 issues. PacSec, 2004.
[47] W. Enck. Defending users against smartphone apps: Techniques and future
directions. In Proceedings of the 7th International Conference on Information
Systems Security.
[48] W. Enck, D. Octeau, P. McDaniel, and S. Chaudhuri. A study of android ap-
plication security. In Proceedings of the 20th USENIX Conference on Security.
[49] W. H. Enck. Analysis Techniques for Mobile Operating System Security.PhD
thesis.
[50] A. Escudero-Pascual, T. Holleboom, and S. Fischer-Hubner. Privacy for loca-
tion data in mobile networks.
[51] A. P. Felt, E. Ha, S. Egelman, A. Haney, E. Chin, and D. Wagner. Android
permissions: User attention, comprehension, and behavior. In Proceedings of
the Eighth Symposium on Usable Privacy and Security.
[52] D. Ferraiolo, R. Chandramouli, R. Kuhn, and V. Hu. Extensible access control
markup language (xacml) and next generation access control (ngac). In Pro-
ceedings of the 2016 ACM International Workshop on Attribute Based Access
Control, ABAC ’16, pages 13–24, New York, NY, USA, 2016. ACM.
“On Improving Privacy and Security Through User-Informed Design.”
BIBLIOGRAPHY 105/114
[53] T. Fiebig, J. Danisevskis, and M. Piekarska. A metric for the evaluation
and comparison of keylogger performance. In 7th Workshop on Cyber Security
Experimentation and Test, CSET ’14, San Diego, CA, USA, August 18, 2014.,
2014.
[54] T. Fischer, A.-R. Sadeghi, and M. Winandy. A pattern for secure graphical
user interface systems. In Database and Expert Systems Application, 2009.
DEXA’09. 20th International Workshop on, pages 186–190. IEEE, 2009.
[55] J. Friedman and D. V. Hoffman. Protecting data on mobile devices: A taxon-
omy of security threats to mobile computing and review of applicable defenses.
Inf. Knowl. Syst. Manag.
[56] H. Fu and J. Lindqvist. General area or approximate location?: How people
understand location permissions. In Proceedings of the 13th Workshop on
Privacy in the Electronic Society, WPES ’14.
[57] J. Gao, E. Blasch, K. Pham, G. Chen, D. Shen, and Z. Wang. Automatic
vehicle license plate recognition with color component texture detection and
template matching. In SPIE Defense, Security, and Sensing, pages 87390Z–
87390Z. International Society for Optics and Photonics, 2013.
[58] S. L. Garfinkel and A. Shelat. Remembrance of data passed: A study of disk
sanitization practices. IEEE Security and Privacy, 2003.
[59] F. Giesen. A trip through the graphics pipeline. http://fgiesen.wordpress.
com/2011/07/09/a-trip-through-the-graphics-pipeline-2011-index/,
2011. Blog:The ryg blog.
[60] G. Gigerenzer and D. G. Goldstein. Reasoning the fast and frugal way: Models
of bounded rationality. Psychological Review, 103:650–669, 1996.
[61] N. Golde, K. Redon, and R. Borgaonkar. Weaponizing Femtocells: The Effect
of Rogue Devices on Mobile Telecommunications. In Proceedings of the 19th
Annual Network & Distributed System Security Symposium, Feb. 2012.
[62] N. Golde, K. Redon, and J.-P. Seifert. Let me answer that for you: Exploit-
ing broadcast information in cellular networks. In Proceedings of the 22Nd
USENIX Conference on Security.
[63] Google Inc. Android 1.6 Compatibility Definition Document. 2009.
[64] V. Gopalkrishnan, D. Steier, H. Lewis, and J. Guszcza. Big data, big busi-
ness: Bridging the gap. In Proceedings of the 1st International Workshop on
Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems,
Programming Models and Applications, BigMine ’12, pages 7–11, New York,
NY, USA, 2012. ACM.
“On Improving Privacy and Security Through User-Informed Design.”
106/114 BIBLIOGRAPHY
[65] M. C. Grace, Y. Zhou, Z. Wang, and X. Jiang. Systematic detection of capa-
bility leaks in stock android smartphones. In NDSS 2012.
[66] Y. Guo, L. Zhang, and X. Chen. Collaborative privacy management: Mo-
bile privacy beyond your own devices. In Proceedings of the ACM MobiCom
Workshop on Security and Privacy in Mobile Environments, SPME ’14, pages
25–30, New York, NY, USA, 2014. ACM.
[67] A. Hang, E. von Zezschwitz, A. De Luca, and H. Hussmann. Too much infor-
mation!: User attitudes towards smartphone sharing. In Proceedings of the 7th
Nordic Conference on Human-Computer Interaction: Making Sense Through
Design, NordiCHI ’12, pages 284–287, New York, NY, USA, 2012. ACM.
[68] M. Heiderich, J. Schwenk, T. Frosch, J. Magazinius, and E. Z. Yang. mxss at-
tacks: Attacking well-secured web-applications by using innerhtml mutations.
In Proceedings of the 2013 ACM SIGSAC Conference on Computer &
Communications Security.
[69] P. Hornyack, S. Han, J. Jung, S. Schechter, and D. Wetherall. These aren’t the
droids you’re looking for: Retrofitting android to protect data from imperious
applications. CCS ’11.
[70] J.-J. Hsieh and W.-C. Huang. Nonparametric estimation and test of con-
ditional kendall’s tau under semi-competing risks data and truncated data.
Journal of Applied Statistics, 42(7):1602–1616, 2015.
[71] L.-S. Huang, A. Moshchuk, H. J. Wang, S. Schechter, and C. Jackson. Click-
jacking: Attacks and defenses. In Proceedings of the 21st USENIX Conference
on Security Symposium.
[72] W. Hudson. User stories don’t help users: introducing persona stories. Inter-
actions, 20:50–53, 2013.
[73] Iso. ISO 9241-210:2010 - Ergonomics of human-system interaction – Part 210:
Human-centred design for interactive systems, 2010.
[74] J. Zdziarski. Identifying back doors, attack points, and surveillance
mechanisms in ios devices. ttp://www.zdziarski.com/blog/wp-content/
uploads/2014/08/Zdziarski-iOS-DI-2014.pdf.
[75] L. Jedrzejczyk, B. A. Price, A. K. Bandara, and B. Nuseibeh. On the impact of
real-time feedback on users’ behaviour in mobile location-sharing applications.
In Proceedings of the Sixth Symposium on Usable Privacy and Security, SOUPS
’10, pages 14:1–14:12, New York, NY, USA, 2010. ACM.
[76] L. K. John, A. Acquisti, and G. Loewenstein. Strangers on a plane: Context-
dependent willingness to divulge sensitive information. Journal of Consumer
Research, 37(5):858, 2011.
“On Improving Privacy and Security Through User-Informed Design.”
BIBLIOGRAPHY 107/114
[77] J. Jung, S. Han, and D. Wetherall. Short paper: Enhancing mobile applica-
tion permissions with runtime feedback and constraints. In Proceedings of the
Second ACM Workshop on Security and Privacy in Smartphones and Mobile
Devices, SPSM ’12, pages 45–50, New York, NY, USA, 2012. ACM.
[78] D. Kahneman and A. Tversky. Prospect Theory: An Analysis of Decision
under Risk. Econometrica, 47(2):263–91, March 1979.
[79] D. Kahneman and A. Tversky. Prospect theory: An analysis of decisions under
risk. Econometrica, pages 263–291, 1979.
[80] C. Kasmi and J. L. Esteves. Iemi threats for information security: Remote
command injection on modern smartphones. Electromagnetic Compatibility,
IEEE Transactions on, PP(99):1–4, 2015.
[81] D. Kedmey. Hackers leak explicit photos of more than 100 celebrities, 2014.
[82] M. J. Keith, S. C. Thompson, J. Hale, P. B. Lowry, and C. Greer. Information
disclosure on mobile devices: Re-examining privacy calculus with actual user
behavior. Int. J. Hum.-Comput. Stud., 71(12):1163–1173, Dec. 2013.
[83] P. G. Kelley, M. Benisch, L. F. Cranor, and N. Sadeh. When are users com-
fortable sharing locations with advertisers? In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, 2011.
[84] P. G. Kelley, L. F. Cranor, and N. Sadeh. Privacy as part of the app decision-
making process. In Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems, CHI ’13, pages 3393–3402, New York, NY, USA, 2013.
ACM.
[85] P. Kijudomsin, C. Pavaganun, S. Kaviya, W. Nacharee, and P. P. Yupapin.
Home intelligent based on privacy camera and home alarming systems. Pro-
cedia Engineering, 8:337 – 342, 2011. The 2nd International Science, Social
Science, Engineering and Energy Conference 2010 (I-SEEC 2010).
[86] Kleiner Perkins Caufield and Byers. Internet trends.
[87] J. Kong. Anonymous and Untraceable Communications in Mobile Wireless
Networks. PhD thesis, 2004. AAI3147729.
[88] D. F. Kune, J. Koelndorfer, N. Hopper, and Y. Kim. Location leaks on the
GSM Air Interface. Proceedings of the Network and Distributed System Secu-
rity Symposium, 2012.
[89] E. Ladakis, L. Koromilas, G. Vasiliadis, M. Polychonakis, and S. Ioannidis.
You can type, but you can’t hide: A stealthy gpu-based keylogger. In Pro-
ceedings of the European Workshop on System Security (EuroSec), 2013.
“On Improving Privacy and Security Through User-Informed Design.”
108/114 BIBLIOGRAPHY
[90] E. Ladakis, L. Koromilas, G. Vasiliadis, M. Polychronakis, and S. Ioannidis.
You can type, but you can’t hide: A stealthy gpu-based keylogger. In EU-
ROSEC - 2013 European Workshop on System Security, 2013.
[91] C. Li and J. Bernoff.Groundswell, Expanded and Revised Edition: Winning in
a World Transformed by Social Technologies. Harvard Business School Press,
Boston, MA, USA, 2011.
[92] J. Lin. Understanding and Capturing People’s Mobile App Privacy Preferences.
PhD thesis, Pittsburgh, PA, USA, 2013. AAI3577905.
[93] J. Lindström and C. Hanken. Security challenges and selected legal aspects
for wearable computing. J. Inf. Technol. Res., 5(1):68–87, Jan. 2012.
[94] A. Lineberry. Malicious code injection via /dev/mem. In Proceedings of Black-
hat Europe, 2009.
[95] D. Luebke and G. Humphreys. How gpus work. Computer, 40(2):126–130,
Feb. 2007.
[96] B. Lynch. Location and privacy: Where are we headed on data privacy day?
Technical report, Microsoft, 2011.
[97] C. Maartmann-Moe. Ftwautopwn. http://www.breaknenter.org/
projects/ftwautopwn/. source code.
[98] F. Maggi, A. Volpatto, S. Gasparini, G. Boracchi, and S. Zanero. A fast eaves-
dropping attack against touchscreens. In Information Assurance and Security
(IAS), 2011 7th International Conference on, pages 320–325. IEEE, 2011.
[99] D. Malandrino, A. Petta, V. Scarano, L. Serra, R. Spinelli, and B. Krishna-
murthy. Privacy awareness about information leakage: Who knows what about
me? In Proceedings of the 12th ACM Workshop on Workshop on Privacy in
the Electronic Society, WPES ’13, pages 279–284, New York, NY, USA, 2013.
ACM.
[100] K. McAllister. Writing kernel exploits. http://ugcs.net/~keegan/talks/
kernel-exploit/talk.pdf, 2012.
[101] E. McCallister, T. Grance, and K. A. Scarfone. Sp 800-122. guide to protecting
the confidentiality of personally identifiable information (pii). Technical report,
Gaithersburg, MD, United States, 2010.
[102] B. A. Mellers, A. Schwartz, and A. D. J. Cooke. Judgment and decision
making. Annu. Rev. Psychol., 49:447?77, 1998.
[103] Y. Michalevsky, D. Boneh, and G. Nakibly. Gyrophone: Recognizing Speech
from Gyroscope Signals. 23rd USENIX Security Symposium (USENIX Secu-
rity 14), 2014.
“On Improving Privacy and Security Through User-Informed Design.”
BIBLIOGRAPHY 109/114
[104] K. Micinski, P. Phelps, and J. S. Foster. An Empirical Study of Location
Truncation on Android. In Mobile Security Technologies (MoST) 2013.
[105] E. Miluzzo, A. Varshavsky, S. Balakrishnan, and R. R. Choudhury. Tapprints:
your finger taps have fingerprints. Proceedings of the 10th international con-
ference on Mobile systems, applications, and services - MobiSys ’12, 2012.
[106] N. Mommen and M. Piekarska. Towards Improving Privacy Awareness Re-
garding Apps’ Permissions on Linux Based Mobile OS. Springer International
Publishing, Cham, 2016.
[107] C. Mulliner, N. Golde, and J.-P. Seifert. Sms of death: From analyzing to
attacking mobile phones on a large scale. In Proceedings of the 20th USENIX
Conference on Security.
[108] A. Munshi, D. Ginsburg, and D. Shreiner. OpenGL(R) ES 2.0 Programming
Guide. Addison-Wesley Professional, 1 edition, 2008.
[109] B. J. Myers. Student Perceptions of Computer Anxiety: The Relationship of
Computer Attitude, Computer Experience, Age, Gender, and Socioeconomic
Status. PhD thesis, Vermillion, SD, USA, 2006. AAI3255100.
[110] Q. Ni, A. Trombetta, E. Bertino, and J. Lobo. Privacy-aware role based
access control. In Proceedings of the 12th ACM Symposium on Access Control
Models and Technologies, SACMAT ’07, pages 41–50, New York, NY, USA,
2007. ACM.
[111] Nokia Solutions and Networks. Understanding Smartphone Behavior in the
Network. Technical report, 2013.
[112] E. Owusu, J. Han, S. Das, A. Perrig, and J. Zhang. Accessory: password
inference using accelerometers on smartphones. In Proceedings of the Twelfth
Workshop on Mobile Computing Systems & Applications, page 9. ACM, 2012.
[113] A. V. Pandey, A. Manivannan, O. Nov, M. Satterthwaite, and E. Bertini. The
persuasive power of data visualization. IEEE Transactions on Visualization
and Computer Graphics, 20(12):2211–2220, Dec 2014.
[114] S. Park and H. A. Kautz. Privacy-preserving recognition of activities in daily
living from multi-view silhouettes and rfid-based training. In AI in Eldercare:
New Solutions to Old Problems, Papers from the 2008 AAAI Fall Symposium,
Arlington, Virginia, USA, November 7-9, 2008, pages 70–77, 2008.
[115] S. Patil and J. Lai. Who gets to know what when: Configuring privacy permis-
sions in an awareness application. In Proceedings of the SIGCHI Conference
on Human Factors in Computing Systems, CHI ’05, pages 101–110, New York,
NY, USA, 2005. ACM.
“On Improving Privacy and Security Through User-Informed Design.”
110/114 BIBLIOGRAPHY
[116] S. Patil, Y. Le Gall, A. J. Lee, and A. Kapadia. My privacy policy: Exploring
end-user specification of free-form location access rules. In Proceedings of the
16th International Conference on Financial Cryptography and Data Security.
[117] S. Patil, R. Schlegel, A. Kapadia, and A. J. Lee. Reflection or action?: How
feedback and control affect location sharing decisions. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems.
[118] D. R. Piegdon. Hacking in physically addressable memory - a proof of concept.
In Seminar of Advanced Exploitation Techniques, 2006.
[119] M. Piekarska. Hide, don’t seek: on user-centric location accuracy. In The Ninth
International Conference on Advances in Human-oriented and Personalized
Mechanisms, Technologies, and Services, CENTRIC’16, Rome, Italy, ISBN:
978-1-61208-503-6, 2016.
[120] M. Piekarska, S. Park, and A. Shaik. Back to the future: Evaluation model for
wearables based on the past experience. In The Fifth International Conference
on Communications, Computation, Networks and Technologies, INNOV’16,
Rome, Italy, ISBN: 978-1-61208-503-6, 2016,.
[121] M. Piekarska, B. Shastry, and R. Borgaonkar. What does the fox say? on
the security architecture of firefox OS. In Ninth International Conference
on Availability, Reliability and Security, ARES 2014, Fribourg, Switzerland,
September 8-12, 2014, pages 172–177, DOI: 10.1109/ARES.2014.30, 2014.
[122] M. Piekarska, Y. Zhou, D. Strohmeier, and A. Raake. Because we care: Pri-
vacy dashboard on firefox OS. CoRR, abs/1506.04105, 2015.
[123] M. Pilgrim. HTML5: Up and Running. O’Reilly Media, Inc., 1st edition,
2010.
[124] A. Pingley, W. Yu, N. Zhang, X. Fu, and W. Zhao. A context-aware scheme for
privacy-preserving location-based services. Comput. Netw., 56(11):2551–2568,
July 2012.
[125] M. Pirker and D. Slamanig. A framework for privacy-preserving mobile pay-
ment on security enhanced arm trustzone platforms. In 2012 IEEE 11th
International Conference on Trust, Security and Privacy in Computing and
Communications, pages 1155–1160, June 2012.
[126] K. P. N. Puttaswamy and B. Y. Zhao. Preserving privacy in location-based
mobile social applications. HotMobile ’10.
[127] P. A. Rauschnabel, A. Brem, and B. S. Ivens. Who will buy smart glasses?
Empirical results of two pre-market-entry studies on the role of personality
in individual awareness and intended adoption of Google Glass wearables.
Computers in Human Behavior, 49(C):635–647, Aug. 2015.
“On Improving Privacy and Security Through User-Informed Design.”
BIBLIOGRAPHY 111/114
[128] Research and Markets. Global smart wearables market forecast and opportu-
nities, 2020. Technical report, 2015.
[129] K. Rhee, D. Won, S.-W. Jang, S. Chae, and S. Park. Threat modeling of a mo-
bile device management system for secure smart work. Electronic Commerce
Research, 13(3):243–256, September 2013.
[130] J. Robertson. Likert-type scales, statistical methods, and effect sizes. Com-
mun. ACM, 55(5):6–7, May 2012.
[131] F. Rohrer, Y. Zhang, L. Chitkushev, and T. Zlateva. Dr baca: Dynamic role
based access control for android. ACSAC ’13.
[132] S. Rosen, Z. Qian, and Z. M. Mao. Appprofiler: A flexible method of exposing
privacy-related behavior in android applications to end users. In Proceedings
of the Third ACM Conference on Data and Application Security and Privacy,
CODASPY ’13, pages 221–232, New York, NY, USA, 2013. ACM.
[133] S. Rosenblatt. Why android won?t be getting app ops anytime soon. CNET.
[134] S. Sagiroglu and G. Canbek. Keyloggers. Technology and Society Magazine,
IEEE, 28(3):10–17, 2009.
[135] H. Saidi and A. Gehani. Mobile security: Challenges, lessons, and future
directions. Information Systems Security Association Journal.
[136] M. Sauter. Communication Systems for the Mobile Information Society. John
Wiley, 2006.
[137] R. Schlegel, A. Kapadia, and A. J. Lee. Eyeing your exposure: Quantifying
and controlling information sharing for improved privacy. In Proceedings of
the Seventh Symposium on Usable Privacy and Security, SOUPS ’11, pages
14:1–14:14, New York, NY, USA, 2011. ACM.
[138] R. Schlegel, K. Zhang, X.-y. Zhou, M. Intwala, A. Kapadia, and X. Wang.
Soundcomber: A stealthy and context-aware sound trojan for smartphones.
In NDSS, volume 11, pages 17–33, 2011.
[139] B. Schneier. Security pitfalls in cryptographic design. Information Manage-
ment & Computer Security, 6(3):133–137, 1998.
[140] R. Sevinsky. Funderbolt. adventures in thunder-
bolt dma attacks. https://media.blackhat.com/us-13/
US-13-Sevinsky-Funderbolt-Adventures-in-Thunderbolt-DMA-Attacks-Slides.
pdf, 2013.
[141] I. Shklovski, S. D. Mainwaring, H. H. Skúladóttir, and H. Borgthorsson. Leak-
iness and creepiness in app space: Perceptions of privacy and mobile app use.
“On Improving Privacy and Security Through User-Informed Design.”
112/114 BIBLIOGRAPHY
In Proceedings of the 32Nd Annual ACM Conference on Human Factors in
Computing Systems, CHI ’14, pages 2347–2356, New York, NY, USA, 2014.
ACM.
[142] R. Shokri, P. Papadimitratos, G. Theodorakopoulos, and J.-P. Hubaux. Col-
laborative location privacy. In Mobile Adhoc and Sensor Systems (MASS),
2011 IEEE 8th International Conference on.
[143] R. Shokri, G. Theodorakopoulos, P. Papadimitratos, E. Kazemi, and J.-P.
Hubaux. Hiding in the mobile crowd: Locationprivacy through collaboration.
Dependable and Secure Computing, IEEE Transactions on, 2014.
[144] R. Shokri, C. Troncoso, C. Diaz, J. Freudiger, and J.-P. Hubaux. Unraveling
an old cloak: K-anonymity for location privacy. In Proceedings of the 9th
Annual ACM Workshop on Privacy in the Electronic Society, WPES ’10.
[145] H. Simon. Rational decision making in business organizations. American
Economic Review, 69(4):493–513, 1979.
[146] L. Simon and R. Anderson. Pin skimmer: Inferring pins through the camera
and microphone. In Proceedings of the 3rd ACM workshop on Security and
privacy in smartphones and mobile devices. ACM, 2013.
[147] M. D. Smucker, J. Allan, and B. Carterette. A comparison of statistical signifi-
cance tests for information retrieval evaluation. In Proceedings of the Sixteenth
ACM Conference on Conference on Information and Knowledge Management,
CIKM ’07, pages 623–632, New York, NY, USA, 2007. ACM.
[148] P. Stewin and I. Bystrov. Understanding dma malware. In Proceedings of the
9th International Conference on Detection of Instrusions and Malware and
Vulnerability Assessment (DIMVA), 2012.
[149] TechCrunch. Sexual Activity Tracked By Fitbit Shows Up In Google Search
Results. 2011.
[150] TeleNav, Inc. Survey finds one-third of americans more willing to give up sex
than their mobile phones.
[151] The Common Criteria Recognition Agreement Members. Com-
mon Criteria for Information Technology Security Evaluation.
http://www.commoncriteriaportal.org/, Sept. 2006.
[152] J. Y. Tsai, P. Kelley, P. Drielsma, L. F. Cranor, J. Hong, and N. Sadeh. Who’s
viewed you?: The impact of feedback in a mobile location-sharing application.
In Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems, 2009.
“On Improving Privacy and Security Through User-Informed Design.”
BIBLIOGRAPHY 113/114
[153] J. Y. Tsai, P. G. Kelley, L. F. Cranor, and N. Sadeh. Location-sharing tech-
nologies: Privacy risks and controls. In A Journal of Law and Policy for the
Information Society 6(2), 2010.
[154] A. Tversky and D. Kahneman. Judgment under uncertainty: Heuristics and
biases. Science, 185(4157):1124–1131, Sept. 1974.
[155] S. Uellenbeck, M. Dürmuth, C. Wolf, and T. Holz. Quantifying the security
of graphical passwords: The case of android unlock patterns. In Proceedings
of the 2013 ACM SIGSAC Conference on Computer & Communications
Security.
[156] F. van den Broek and R. Wichers Schreur. Femtocell Security in Theory and
Practice, pages 183–198. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.
[157] G. Vasiliadis, M. Polychronakis, and S. Ioannidis. Gpu-assisted malware. In
Proceedings of the 5th International Conference on Malicious and Unwanted
Software (MALWARE), 2010.
[158] T. Vila, R. Greenstadt, and D. Molnar. Why we can’t be bothered to read
privacy policies models of privacy economics as a lemons market. In Proceed-
ings of the 5th International Conference on Electronic Commerce, ICEC ’03,
pages 403–407, New York, NY, USA, 2003. ACM.
[159] T. Wang, K. Lu, L. Lu, S. Chung, and W. Lee. Jekyll on ios: When benign
apps become evil. In Proceedings of the 22Nd USENIX Conference on Security.
[160] T. Watt. Mali GPUs in SmartTVs and Set-Top-
Boxes http://malideveloper.arm.com/learn-about-mali/
mali-gpus-in-smarttvs-and-set-top-boxes/,
accessed: 10.02.2014, 2012.
[161] C. Welch. Google’s android malware scanner detects only 15 percent of
malicious code in test (update). http://www.theverge.com/2012/12/10/
3751202/google-android-malware-scanner-test.
[162] J. D. Wendt and M. J. Guise. Trusted computing technologies, intelr trusted
execution technology. 2011.
[163] B. Wentz and J. Lazar. Are separate interfaces inherently unequal?: An evalu-
ation with blind users of the usability of two interfaces for a social networking
platform. In Proceedings of the 2011 iConference, iConference ’11, pages 91–
97, New York, NY, USA, 2011. ACM.
[164] M. Wernke, P. Skvortsov, F. Dürr, and K. Rothermel. A classification of
location privacy attacks and approaches. Personal Ubiquitous Comput.
[165] W. West and S. M. Pulimood. Analysis of privacy and security in html5 web
storage. J. Comput. Sci. Coll.
“On Improving Privacy and Security Through User-Informed Design.”
114/114 BIBLIOGRAPHY
[166] Wikipedia. Edison chen photo scandal — wikipedia, the free encyclopedia,
2015. [Online; accessed 23-October-2015].
[167] S. Wilson, J. Cranshaw, N. Sadeh, A. Acquisti, L. F. Cranor, J. Springfield,
S. Y. Jeong, and A. Balasubramanian. Privacy manipulation and acclimation
in a location sharing application. In Proceedings of the 2013 ACM Interna-
tional Joint Conference on Pervasive and Ubiquitous Computing.
[168] Z. Xu, K. Bai, and S. Zhu. Taplogger: Inferring user inputs on smartphone
touchscreens using on-board motion sensors. In Proceedings of the fifth ACM
conference on Security and Privacy in Wireless and Mobile Networks, pages
113–124. ACM, 2012.
[169] T. T. W. Yee and N. Thein. Leveraging access control mechanism of android
smartphone using context-related role-based access control model. In Net-
worked Computing and Advanced Information Management (NCM), 2011 7th
International Conference on.
[170] X. Zhang, J.-P. Seifert, and O. Aciiçmez. Seip: Simple and efficient integrity
protection for open mobile platforms. In Proceedings of the 12th International
Conference on Information and Communications Security.
[171] Y. Zhou, X. Zhang, X. Jiang, and V. W. Freeh. Taming information-stealing
smartphone applications (on android). TRUST’11.
“On Improving Privacy and Security Through User-Informed Design.”
