Jou nal o Machine Lea ning Resea ch 25 (2024) 1-44 Submi ed 12/21; Re ised 2/24; Published 3/24
Di e en ially P i a e Me hods o Managing
Model Unce ain y in Linea Reg ession Models
V´ıc o Pe˜na ic o .pena.piza [email p o ec ed]
Depa men d’Es ad´ıs ica i In es igaci´o Ope a i a
Uni e si a Poli `ecnica de Ca alunya
Ba celona, Spain
And ´es F. Ba ien os [email p o ec ed]
Depa men o S a is ics
Flo ida S a e Uni e si y
Tallahassee, FL 32306, USA
Edi o : Mo i z Ha d
Abs ac
In his wo k, we p opose di e en ially p i a e me hods o hypo hesis es ing, model a e -
aging, and model selec ion o no mal linea models. We p opose Bayesian me hods based
on mix u es o g-p io s and non-Bayesian me hods based on likelihood- a io s a is ics and
in o ma ion c i e ia. The p ocedu es a e asymp o ically consis en and s aigh o wa d o
implemen wi h exis ing so wa e. We ocus on p ac ical issues such as adjus ing c i i-
cal alues so ha hypo hesis es s ha e adequa e ype I e o a es and quan i ying he
unce ain y in oduced by he p i acy-ensu ing mechanisms.
Keywo ds: Con iden ial Da a, Reg ession, Bayesian Me hods, In o ma ion C i e ia
1. In oduc ion
Di e en ial p i acy (Dwo k e al., 2006) is a o mal amewo k o quan i ying he p i acy
o andomized algo i hms. I s heo e ical p ope ies ha e been s udied ex ensi ely (see
e.g. Dwo k e al. (2014)) and i has been adop ed by companies such as Google, Apple,
and Mic oso (Ga inkel e al., 2018), as well as ins i u ions like he U.S. Census Bu eau
(Abowd, 2018).
In his a icle, we de elop di e en ially p i a e me hods o no mal linea models. We
p opose di e en ially p i a e hypo hesis es s o compa ing nes ed models (in Sec ion 4)
as well as me hods o model a e aging and selec ion (in Sec ion 5). We conside Bayesian
me hods based on mix u es o g-p io s (Liang e al., 2008) and non-Bayesian me hods ha
a e buil upon likelihood- a io s a is ics and in o ma ion c i e ia.
In Bayesian hypo hesis es ing and model selec ion, p io dis ibu ions mus be chosen
ca e ully because hei e ec does no anish as he sample size g ows (Baya i e al., 2012).
Ou wo k is based on mix u es o g-p io s because, when combined wi h igh -Haa p io s
on he common pa ame e s, hey sa is y a lis o appealing c i e ia p oposed in Baya i e al.
(2012). They a e also con enien ly implemen ed in he Rpackage lib a y(BAS) (Clyde,
2020).
c
2024 V´ıc o Pe˜na, And ´es F. Ba ien os.
License: CC-BY 4.0, see h ps://c ea i ecommons.o g/licenses/by/4.0/. A ibu ion equi emen s a e p o ided
a h p://jml .o g/pape s/ 25/21-1536.h ml.
Pe˜
na and Ba ien os
F om a non-Bayesian pe spec i e, we wo k wi h likelihood- a io es s and in o ma ion
c i e ia. We make his choice because o hei heo e ical p ope ies, in ui i e appeal, and
ease o use. The class o in o ma ion c i e ia we conside includes he Akaike In o ma-
ion C i e ion (AIC; Akaike (1974)) and he Bayesian In o ma ion C i e ion (BIC; Schwa z
(1978)), among o he s. Al hough we ocus on no mal linea models, in o ma ion c i e ia a e
use ul in model a e aging and selec ion p oblems o mo e complex models. We ecommend
he monog aph Claeskens e al. (2008) o an ex ensi e o e iew o he app oach.
We en o ce di e en ial p i acy wi h well-es ablished echniques. Fo hypo hesis es ing,
we use he subsample and agg ega e echnique (Nissim e al., 2007; Smi h, 2011), which
consis s in spli ing he da a in o subg oups and eleasing pe u bed a e ages. Fo model
a e aging and selec ion, we use su icien -s a is ic pe u ba ion, which consis s in eleasing
a noisy e sion o a su icien s a is ic (see, o example, McShe y and Mi ono (2009); Vu
and Sla ko ic (2009) and Be ns ein and Sheldon (2019)).
1.1 Rela ed Wo k
The e is a g owing li e a u e on di e en ially p i a e me hods o linea eg ession models.
Fo example, Ami ai and Rei e (2018) es ima e quan iles and pos e io ail p obabili ies o
coe icien s, Ba ien os e al. (2019) es he signi icance o indi idual eg ession coe icien s,
and Fe ando e al. (2022) p opose me hods o poin and in e al es ima ion ha can be
applied o he no mal linea model. Lei e al. (2018) conside he p oblem o model selec ion
based on in o ma ion c i e ia, bu do no conside model a e aging o Bayesian app oaches,
and Be ns ein and Sheldon (2019) p opose a me hod o sampling om pos e io dis ibu-
ions on eg ession coe icien s, bu do no conside hypo hesis es ing, model a e aging o
selec ion.
Ou me hods o hypo hesis es ing in ol e da a-spli ing and censo ing. Bo h ope a-
ions can induce bias in he ou pu s. E ans e al. (2020) conside s he e ec s o censo ing in
es ima es ha a e asymp o ically no mal and p oposes s a egies o co ec he bias induced
by censo ing. Co ing on e al. (2021) uses bags o li le boo s aps (Kleine e al., 2012)
o ind unbiased es ima es and alid con idence in e als. Al e na i ely, Fe ando e al.
(2022) use he pa ame ic boo s ap o bias co ec ion. In hypo hesis es ing, he bias
induced by da a-spli ing and censo ing leads o inapp op ia e c i ical alues o ejec ing
null hypo heses. We add ess his issue by simula ing he dis ibu ion o he di e en ially
p i a e es s a is ics unde he null hypo hesis and hen ind co ec ed c i ical alues ha
a e adequa ely calib a ed.
In gene al, ou Bayesian me hodology d aws om he objec i e Bayesian li e a u e o
hypo hesis es ing, a e aging, and selec ion, especially om Liang e al. (2008) and Baya i
e al. (2012). In hese e e ences, he classes o p io s we conside he e a e p oposed a e
showing ha hey sa is y a lis o concep ually appealing c i e ia.
1.2 Main Con ibu ions
In Sec ion 4, we a gue ha wo king on a loga i hmic scale is na u al o de ining di e en ially
p i a e Bayes ac o s. We show ha he me hods a e asymp o ically consis en unde
egula i y condi ions ha a e simila o he ones needed o consis ency when he e a e no
p i acy cons ain s. In Sec ion 4.2, we desc ibe a simple p ocedu e o quan i y he e ec s
2
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
o he p i acy-ensu ing mechanisms. Then, in Sec ion 4.3, we s udy he e ec s o censo ing
and da a-spli ing.
In Sec ion 5, we use su icien -s a is ic pe u ba ion o de ine me hods o model a -
e aging and selec ion. I we ook a nai e app oach o he p oblem, he a iance o he
pe u ba ion e m would g ow exponen ially in he numbe o p edic o s. Wi h su icien -
s a is ic pe u ba ion, he a iance o he pe u ba ion e m g ows quad a ically in he
numbe o p edic o s. The me hods o model selec ion a e consis en unde condi ions
ha a e simila o hose needed o consis ency wi hou p i acy cons ain s. In Sec ion 5.1,
we p opose a s a egy ha quan i ies he unce ain y in oduced by he mechanisms ha
is analogous o he one pu sued in Sec ion 4.2.
F om a p ac ical poin o iew, we gi e guidelines o maximizing he s a is ical u ili y o
he me hods in ini e samples. We illus a e he pe o mance o ou me hods in Sec ions 4.4
and 5.2. We include addi ional esul s om he simula ion s udies in he Appendix.
The p oo s o he P oposi ions s a ed in he main ex can be ound in he Appendix A.
1.3 No a ion
Ex ema appea o en in Sec ion 4 because he me hods a e based on censo ed s a is ics.
Ou no a ion o hem is a∨b= max(a, b) and a∧b= min(a, b). The censo ed s a is ics
a e o he o m Tc= (T∨a)∧b: ha is, Tc=awhene e T≤aand Tc=bwhene e
T≥b(wi h he unde s anding ha a<b).
We use he ollowing no a ion o p obabili y dis ibu ions. The p- a ia e no mal dis i-
bu ion wi h mean µand co a iance ma ix Σ is Np(µ, Σ), he Laplace dis ibu ion wi h loca-
ion pa ame e µand scale pa ame e bis L1(µ, b), and he (p×p)-dimensional Wisha dis-
ibu ion wi h deg ee o eedom n>p−1 and posi i e-de ini e scale ma ix Vis Wp(n, V ).
In he case o ma ices, all o hem a e assumed o be ull- ank unless o he wise s a ed.
Ou no a ion o basic ma ix ope a ions and special ma ices is as ollows. The ma ix
anspose o Ais A0, he pe pendicula p ojec ion ope a o on o he column space o Ais
PA=A(A0A)−1A0, and he uppe Cholesky ac o o Ais A1/2. The (n×n)−dimensional
ze o ma ix is 0n×nand he (n×n)-dimensional iden i y ma ix is In. Fo ec o s, he
usual p-no m on
R
dis k·kp. The ze o ec o is 0n= (0,0, ... n),0)0and he ec o o ones is
1n= (1,1, ... n),1)0. The expec a ion o a andom a iable Xis E(X) and he a iance is
Va (X).
2. B ie Re iew o Di e en ial P i acy
Di e en ial p i acy (Dwo k e al., 2006, 2014) is a p obabilis ic p ope y o andomized
algo i hms. In ui i ely, di e en ial p i acy limi s how much we can lea n abou indi idual
en ies in a da a se .
In he li e a u e, andomized algo i hms ha ensu e di e en ial p i acy a e e e ed o
as mechanisms. Concep ually, mechanisms a e unc ions M ha ake da a Das inpu s
and ou pu a andom M(D) ha is, in some sense, p i a e. In he de ini ion o di e en ial
p i acy, a key no ion is ha o neighbo ing da a se s. Two da a se s Dand ˜
Da e neighbo s,
which is deno ed by D∼˜
D, i hey only di e in one ow.
3
Pe˜
na and Ba ien os
De ini ion 1 ((ε, δ)-di e en ial p i acy) Le ε > 0and 0≤δ≤1. A mechanism M
sa is ies (ε, δ)-di e en ial p i acy i , o all D∼˜
Dany M-measu able se S, we ha e
P[M(D)∈S]≤exp(ε)P[M(˜
D)∈S] + δ.
When δ= 0, we e ie e ε-di e en ial p i acy, which is he mos popula o maliza ion
o p i acy in he li e a u e. When εis small, he p i acy o Minc eases; in such cases,
he p obabili y dis ibu ions o M(D) and M(˜
D) a e o ced o be simila , so he ou pu
o Mis no e y sensi i e o small changes in D. On he o he hand, as εinc eases, he
dis ibu ions o M(D) and M(˜
D) a e allowed o be mo e di e en , which dec eases he
p i acy o he ou pu . When 0 < δ ≤1, De ini ion 1 allows M(D) o elease ou pu s ha
lead o a “high p i acy loss” wi h p obabili y δ(see Sec ion 2 in Dwo k e al. (2014) o
de ails).
In his a icle, M(D) a e pe u bed e sions o con iden ial s a is ics T(D). The scale
o he pe u ba ion depends on he global sensi i i y o T(D), a concep we de ine below.
De ini ion 2 (Global sensi i i y) Le Tbe a s a is ic mapping da a o
R
d. The global
sensi i i y o Tis de ined as ∆p= supD∼˜
DkT(D)−T(˜
D)kp.
A key p ope y o di e en ial p i acy is ha ans o ma ions o (ε, δ)-di e en ially p i-
a e s a is ics a e (ε, δ)-di e en ially p i a e. In he li e a u e, his is known as he pos -
p ocessing p ope y o di e en ial p i acy. We use his p ope y equen ly; o example,
we use i o ind pos e io p obabili ies o hypo heses gi en Bayes ac o s.
3. B ie Re iew o Bayes Fac o s and In o ma ion C i e ia
In his sec ion, we e iew basic ac s abou Bayes ac o s and in o ma ion c i e ia ha
a e ele an o ou pu poses. This is no mean o be a comp ehensi e e iew; we e e
he eade o Be ge and Pe icchi (2001), Liang e al. (2008), and Baya i e al. (2012)
o u he backg ound on Bayesian me hods and Claeskens e al. (2008) o in o ma ion
c i e ia.
3.1 Hypo hesis Tes ing
In Sec ion 4, we in oduce di e en ially p i a e me hods o hypo hesis es ing. We co e
Bayesian app oaches ha a e based on Bayes ac o s and non-Bayesian app oaches ha a e
based on likelihood- a io es s and in o ma ion c i e ia. Now, we e iew some co e concep s
in Bayesian and non-Bayesian es ing ha a e help ul o con ex ualizing ou wo k.
Le y= (y1, ... , yn) be a ec o collec ing independen and iden ically dis ibu ed obse -
a ions om a s a is ical model wi h sampling densi y (y|θ) = Qn
i=1 (yi|θ), o θ∈Θ.
Ou goal is es ing H0:θ∈Θ0agains H1:θ∈Θ1, whe e Θ0and Θ1a e disjoin subse s
o Θ.
In he Bayesian pa adigm, all unknowns ha e p obabili y dis ibu ions associa ed o
hem, including he hypo heses H0and H1and he pa ame e θ. Be o e obse ing any
da a, he unce ain y in he hypo heses is e lec ed in he p io p obabili ies P(H0) and
P(H1) = 1 −P(H0). The unce ain y in θis usually exp essed condi ionally h ough he
p io dis ibu ions π(θ|H0) and π(θ|H1). Upon obse ing da a, he unce ain y abou θ
4
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
and he hypo heses is upda ed in he pos e io dis ibu ion, which is simply he condi ional
dis ibu ion o hese unknowns gi en he da a.
A key quan i y in Bayesian hypo hesis es ing is he Bayes ac o , which is de ined as
B10 =Zθ∈Θ1
π(θ|H1) (y|θ)ν(dθ)/Zθ∈Θ0
π(θ|H0) (y|θ)ν(dθ)
o a domina ing measu e ν(·).
Bayes ac o s a e impo an in Bayesian es ing o se e al easons. Fo example, i
o10 =P(H1)/P(H0) a e he p io odds o he hypo heses, hen he pos e io p obabili y o
H1is
P(H1|y) = o10B10/(1 + o10B10),
which depends on he da a only h ough he Bayes ac o B10. Bayes ac o s can be mo i-
a ed as a ios o in eg a ed likelihoods ha quan i y he e idence in a o o H1 ela i e
o H0(Be ge e al., 1999). The e ha e been e o s o ca ego ize he s eng h o e idence
in a o o agains H1based on he magni ude o B10 alone. Pe haps he mos popula
ca ego iza ion is “Je eys’ scale o e idence” (Je eys (1939), see Table 1).
Table 1: Je eys’ scale o e idence o Bayes ac o s (Je eys, 1939)
Bayes ac o In e p e a ion
B10 <1H0suppo ed
1< B10 <101/2E idence agains H0, bu no wo h mo e han a ba e men ion
101/2< B10 <10 E idence agains H0subs an ial
10 < B10 <103/2E idence agains H0s ong
103/2< B10 <102E idence agains H0 e y s ong
B10 >102E idence agains H0decisi e
Loga i hms o Bayes ac o s a e na u ally symme ic abou ze o. To see his, assume
ha H0and H1a e equally likely a p io i. Then, he pos e io p obabili y o H1is he
s anda d logis ic unc ion in log B10: ha is, P(H1|D) = 1/[1+exp(−log B10)]. Changing
he sign o log B10 leads o he complemen 1 −P(H1|D), and P(H1|D)=1/2 i and
only i log B10 = 0.
In Bayesian hypo hesis es ing, he p io s π(θ|H0) and π(θ|H1) mus be chosen
ca e ully. Vague p io s on θ, which a e commonplace in es ima ion p oblems, can lead
o pos e io p obabili ies ha o e whelmingly suppo H0no ma e wha he da a a e.
This phenomenon is known in he li e a u e as Lindley’s pa adox (see Lindley (1957) and
Robe (2014)), and i can occu when H1 ep esen s a la ge se han H0. In no mal linea
eg ession p oblems, mix u es o g-p io s ha e been s udied ca e ully in Liang e al. (2008)
and Baya i e al. (2012). They a oid Lindley’s pa adox by ha ing a ixed scale ma ix
and hey ha e o he desi able p ope ies such as in a iance o Bayes ac o s wi h espec o
changes o measu emen uni s and la ge-sample consis ency. In Sec ions 4 and 5, we wo k
wi h p io s wi hin his class.
F om a non-Bayesian pe spec i e, we p opose wo king wi h likelihood- a io es s and
in o ma ion c i e ia. The likelihood a io o H1 o H0is de ined as
5
Pe˜
na and Ba ien os
Λ10 = max
θ∈Θ1
(y|θ)/max
θ∈Θ0
(y|θ).
The likelihood a io Λ10 is simila o he Bayes ac o B10, bu ins ead o a e aging he
likelihoods wi h weigh s gi en by π(θ|H1) and π(θ|H0), he likelihoods a e maximized
unde H1and H0. Likelihood a io es s a e s anda d wi hin he ield o s a is ics and hei
p ope ies a e well-cha ac e ized (see, o example, Lehmann and Romano (2005)). Unde
mild condi ions, he ans o med log-likelihood a io 2 log Λ10 is asymp o ically dis ibu ed
as chi-squa ed unde H0and consis en unde H1.
Finally, we de ine he class o in o ma ion c i e ia
I10 =n−ρ/2Λ10,
which encompasses AIC o ρ= 2p/ log n, BIC o ρ=p, and he likelihood a io s a is ic
Λ10 o ρ= 0. Fo a ixed ρ, he in o ma ion c i e ion I10 can be in e p e ed as a penalized
likelihood a io, whe e ρac s as a penal y o model complexi y. Fo he no mal linea
model, likelihood- a io es s and AIC ail o be consis en unde H0; BIC, on he o he
hand, is consis en unde H0and H1. We e e he eade o Chap e 4 in he monog aph
Claeskens e al. (2008) o a mo e gene al e sion o his esul and a discussion on ela ed
issues.
3.2 Model A e aging and Selec ion
In Sec ion 5, we ocus on model a e aging and selec ion. The con ex is a eg ession p oblem
whe e he e is an ou come a iable Y∈Rnand pp edic o s collec ed in a design ma ix
X∈Rn×p. We do no know which a iables in X, i any, should be included in ou model
o Ygi en X. This ype o unce ain y is o en e e ed o as model unce ain y. Fo an
in oduc ion o he opic wi h a s ong Bayesian la o , we ecommend D ape (1995).
Model unce ain y can be pa ame e ized h ough a bina y ec o γ∈ {0,1}p ha indi-
ca es ac i e p edic o s: γi= 0 i he i h p edic o is no ac i e and γi= 1 i i is. Concep-
ually, we assume ha he e is a ue model gene a ing he da a iden i ied by T∈ {0,1}p.
Gi en ini e da a D, we a e do no know wha he u h (T) is. F om a Bayesian pe spec i e,
we can pu a p io on γand ind pos e io p obabili ies o quan i y his unce ain y; om
a non-Bayesian pe spec i e, we can ind a poin es ima e o γo a e age ou unce ain y
o e i wi h ules inspi ed by Bayesian p ocedu es.
Fo each model, which we iden i y by i s ac i e p edic o s in γ∈ {0,1}p, we can compu e
a Bayes ac o o in o ma ion c i e ion ela i e o he null model, which does no include
any ac i e p edic o s. We deno e hese null-based Bayes ac o s and in o ma ion c i e ia
Bγ0and Iγ0, espec i ely. Wi h hese, we can pe o m model selec ion a e maximizing
o e γo we can ind model-a e aged es ima es wi h weigh s p opo ional o Bγ0o Iγ0.
4. Hypo hesis Tes ing
In his sec ion, we desc ibe di e en ially p i a e me hods o es ing a null hypo hesis H0
agains an al e na i e H1. The me hodology desc ibed he e can be applied in gene al, bu in
Sec ion 4.1 we ocus on nes ed linea eg ession models, o which we ha e ound heo e ical
gua an ees.
6
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
Ou me hods a e based on he subsample and agg ega e echnique (Nissim e al., 2007).
I consis s in spli ing he da a in o Mdisjoin subg oups, compu ing s a is ics wi hin he
subg oups, and a e aging he esul s. The ou pu is made di e en ially p i a e by adding
a pe u ba ion e m η. The a iance o ηis inc easing in he global sensi i i y (i.e., he
ange) o he s a is ics in ol ed.
We apply he subsample and agg ega e echnique as ollows. Fi s , we spli he da a
in o Mdisjoin subg oups o sample size b1, b2, ... , bM(PM
i=1 bi=n) and compu e censo ed
s a is ics Tc
i= (Ti∨L)∧U∈[L, U] o i∈ {1,2, ... , M}, whe e Tia e aw s a is ics
compu ed om con iden ial da a. A e censo ing he Ti, we know ha hey ha e global
sensi i i y ∆ = U−L. Finally, we elease he noisy a e age PM
i=1 Tc
i/M +η, whe e ηis
a andom pe u ba ion e m ha ensu es (ε, δ)-di e en ial p i acy. I δ= 0, hen η∼
L1(0,∆/(Mε)); i 0 < δ ≤1, hen η∼N1(0, a∆2/(2Mε)), whe e ais a cons an ha can
be compu ed wi h Algo i hm 1 in Balle and Wang (2018).
The con iden ial s a is ics Tia e loga i hms o Bayes ac o s o loga i hms o in o ma ion
c i e ia. We jus i y wo king on a loga i hmic scale o Bayes ac o s in he subsequen
pa ag aphs. A simila a gumen can be used o jus i y his choice o in o ma ion c i e ia.
Le Bc
10,i be he censo ed Bayes ac o in he i h subg oup. Nai ely, one could elease he
noisy a e age PM
i=1 Bc
10,i/M +η. Un o una ely, ha app oach has undesi able p ope ies.
Unde bo h he Laplace and analy ic Gaussian mechanisms, ηis suppo ed on
R
and
symme ic a ound ze o, whe eas B10 is always non-nega i e and shows equal suppo o H0
and H1when B10 = 1. I he e a e no p i acy cons ain s, Bayes ac o s sa is y B01 =B−1
10 ,
bu in gene al PM
i=1 Bc
01,i/M +η6= (PM
i=1 Bc
10,i/M +η)−1.
Al e na i ely, we p opose wo king on a loga i hmic scale, de ining
log ˜
B10 =
M
X
i=1
log Bc
10,i/M +η, log Bc
10,i = (log B10,i ∨L)∧U.
Loga i hms o Bayes ac o s a e suppo ed on
R
and, as we a gued in Sec ion 3, hey ha e
a na u al symme y a ound ze o, so i is sensible o add a ze o mean pe u ba ion e m
on ha scale. A e exponen ia ing, we ob ain a geome ic mean o censo ed Bayes ac o s
wi h a mul iplica i e pe u ba ion:
˜
B10 = exp(η) M
Y
i=1
Bc
10,i!1/M
.
Since he dis ibu ion o ηis symme ic, ( ˜
B10)−1is equal in dis ibu ion o ˜
B01, and i is
exac ly equal o ˜
B01 when η= 0 (i.e., when he e a e no p i acy cons ain s). Geome ic
means o Bayes ac o s ha e appea ed in he objec i e Bayesian li e a u e in geome ic
in insic Bayes ac o s (Be ge and Pe icchi, 1996).
The s a is ic ˜
B10 is based on censo ed Bayes ac o s ha a e suppo ed on [L, U]. How-
e e , he suppo o ˜
B10 is no [L, U] a e in oducing he pe u ba ion e m η. This issue
can be sol ed by censo ing ˜
B10, de ining
B∗
10 = ( ˜
B10 ∨L∗)∧U∗,(1)
7
Pe˜
na and Ba ien os
whe e L∗= exp(L) and U∗= exp(U). Wi h B∗
10, we can de ine he di e en ially p i a e
pos e io p obabili y o H1gi en Das P∗(H1|D) = [1 −P(H0)]B∗
10/{P(H0) + [1 −
P(H0)]B∗
10}.
Following he same easoning, we can de ine a di e en ially p i a e in o ma ion c i e ion
I∗
10 = (˜
I10 ∨L∗)∧U∗(2)
˜
I10 = exp(η) M
Y
i=1
Ic
10,i!1/M
Ic
10,i = (b−ρ/2
iΛ10,i ∨L)∧U.
4.1 Nes ed Linea Reg ession Models
Conside he no mal linea model
Y=X0β0+Xβ +σW, W ∼Nn(0n, In),
whe e X0∈
R
n×p0and X∈
R
n×pa e ull- ank and n>p+p0. In his sec ion, we p esen
di e en ially p i a e me hods o es ing H0:β= 0pagains H1:β6= 0p. The se o
p edic o s in X0is common o H0and H1and i can be, o ins ance, an in e cep 1n.
Fo he Bayesian me hods, we de ine ou p io s a e epa ame e izing he model. We
ew i e he model as Y=X0ψ+V β +σW o V= (In−PX0)Xand ψ=β0+
(X0
0X0)−1X0
0Xβ. In his pa ame e iza ion, he common p edic o s X0a e o hogonal o V,
which is speci ic o H1. I X0is an in e cep 1n, he epa ame e iza ion simply cen e s he
p edic o s in X.
Ou p io speci ica ion is
π(ψ, σ2)∝1/σ2, π(β|σ2, H1) = Z∞
0
Np(β|0p, gσ2(V0V)−1)π(g)ν(dg).
whe e νis an app op ia e domina ing measu e. We allow he p io measu e on g o depend
on nand p, bu no on Y.
The p io on (ψ, σ2) is he igh -Haa p io o his p oblem, which is imp ope , whe eas
β|σ2, H1is a mix u e o g-p io s. This class o p io s has s ong heo e ical suppo (see
Liang e al. (2008) and Baya i e al. (2012) o de ails). Fo example, i leads o Bayes
ac o s and pos e io p obabili ies o hypo heses ha a e in a ian wi h espec o in e ible
linea ans o ma ions o he design ma ix V, such as changes o uni s. This p ope y would
no be sa is ied i we had chosen a diagonal co a iance ma ix o β|g, σ2, H1.
The p io dis ibu ion o β0and σ2is imp ope , so he ma ginal dis ibu ions can only
be de ined up o a bi a y mul iplica i e cons an s. We use he same cons an s o bo h H0
and H1, which is jus i ied by he p inciple o in a iance (Be ge e al., 1998) and p edic i e
ma ching a gumen s (Baya i e al., 2012).
When he da a a e no con iden ial, we can epo he Bayes ac o (Liang e al., 2008):
B10 =Z∞
0
(g+ 1)(n−p−p0)/2[1 + g(1 −R2)]−(n−p0)/2π(g)ν(dg),(3)
8
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
whe e R2=Y0PVY/Y 0(In−PX0)Yo , om a non-Bayesian pe spec i e, we can epo he
in o ma ion c i e ion
I10 =n−ρ/2Λ10 =n−ρ/2(1 −R2)−n/2.(4)
I he e a e p i acy cons ain s, B10 o I10 canno be eleased di ec ly. We p opose
applying he subsample and agg ega e echnique o elease di e en ially p i a e e sions o
B10 and I10. Tha is, we spli up he da a se in o Mdisjoin subg oups o sample sizes
b1, b2, ... , bM(PM
i=1 bi=n), de ine p io s πi(gi) and penal ies ρi o i∈ {1,2, ... , M}, and
elease B∗
10 and I∗
10 as de ined in Equa ions (1) and (2), espec i ely.
P oposi ion 1 below s a es ha B∗
10 and I∗
10 a e consis en unde some egula i y condi-
ions. Consis ency does no ollow om Liang e al. (2008) o a ious easons. Fi s , he e
is a g owing numbe o subg oups and he sample sizes wi hin he subg oups g ow o in in-
i y. Ano he di e ence is ha he Bayes ac o s a e censo ed and he e is a pe u ba ion
e m η. Ou esul does no ollow di ec ly om Smi h (2011), ei he . Smi h (2011) s udies
he asymp o ic beha io o a e ages PM
i=1 Ti/M eleased wi h he subsample and agg ega e
me hod. I T∗
iis he “ ue alue” o Ti, Lemma 6 in Smi h (2011) assumes ha √bi(Ti−T∗
i)
is asymp o ically no mal, E(Ti)−T∗
i∈O(1/bi), and E{[√bi(Ti−T∗
i)]3} ∈ O(1).Fu he -
mo e, he esul s in Smi h (2011) a e o independen and iden ically dis ibu ed da a. In
ou case, i is unclea whe he he asymp o ic condi ions a e sa is ied (i hey a e, i would
equi e p oo ), and we would need addi ional assump ions on, a leas , he design ma ices,
he p io s πi(gi), and he penal ies ρi. Ins ead o e i ying he condi ions in Smi h (2011),
ou p oo uses a union bound and ail inequali ies o R2.
P oposi ion 1 Unde he egula i y condi ions lis ed below, B∗
10 and I∗
10 a e consis en:
unde H0,B∗
10 →P0and I∗
10 →P0; unde H1,(B∗
10)−1→P0and (I∗
10)−1→P0.
1. Well-speci ied model: The aw con iden ial da a a e gene a ed om he no mal
linea model desc ibed in his sec ion.
2. G ow h o Mand bi:limn→∞ M=∞and limn→∞ in i∈1:Mbi=∞. I p≥2,
limn→∞ supi∈1:MM/bi= 0 and, i p= 1,limn→∞ supi∈1:MMplog bi/bi= 0.
3. Censo ing limi s: limn→∞ L=−∞,limn→∞ U=∞.
4. P i acy pa ame e s: The p i acy pa ame e s εand δa e such ha limn→∞(U−
L)/(Mε) = limn→∞ a(U−L)2/(Mε) = 0.
5. Design ma ices: Unde H1,limn→∞ in i∈1:Mβ0
TX0
iXiβT/(σ2
Tbi)>0, whe e βTand
σ2
Ta e he ixed ue alues o βand σ2, espec i ely.
6. P io s on giand penal ies ρi:
lim
n→∞ sup
i∈1:MZ∞
0
bp/2
i(gi+ 1)−p/2πi(gi)ν(dgi)<∞, ρi≤max(p, log bi)
lim
n→∞ in
i∈1:MZ∞
bi
bp/2
i(gi+ 1)−p/2πi(gi)ν(dgi)>0, ρi≥p.
9
Pe˜
na and Ba ien os
whe e βγ∈
R
|γ|×1is a ec o including he βjsuch ha γj= 1 and Vγ∈
R
n×|γ|is a ma ix
wi h he ac i e a iables in γ. Jus as we did in Sec ion 4, we pa ame e ize he model so
ha X0and Vγa e o hogonal. Fo con enience, we deno e he null model whe e none o
he a iables a e ac i e as γ= 0. The ma ix X0 ep esen s a se o p edic o s we a e su e
o include in ou model. Usually, X0con ains an in e cep 1n, bu i can be emp y as well.
I X0is emp y, he me hods in his sec ion would be de ined in analogous manne a e
eplacing In−PX0by In.
F om a Bayesian pe spec i e, ou p io speci ica ion on he eg ession coe icien s βγis
he same we had in Sec ion 4: we pu mix u es o g-p io s on βγ|σ2, γ and he igh -Haa
p io π(ψ, σ2)∝1/σ2on he common pa ame e s. I γis no he null model, he exp ession
o he non-p i a e Bayes ac o o model γ o he null model, deno ed Bγ0, is iden ical o
he one in Equa ion (3) a e subs i u ing pby |γ|and R2by R2
γ=Y0PVγY/Y 0(I−PX0)Y.
I γis he null model, we ha e Bγ0= 1.
Gi en a p io dis ibu ion π(γ) on γ, he pos e io p obabili y o he model iden i ied
by γgi en he da a Dis de ined as
P(γ|D) = P(γ)Bγ0/X
˜γ∈{0,1}p
P(˜γ)B˜γ0,
which depends on he da a only h ough he Bayes ac o s Bγ0. We assume ha he p io
P(γ) can depend on p, bu no on no he design ma ix. Mos common choices o P(γ),
like a uni o m p io P(γ) = 2−po he hie a chical uni o m p io ecommended by Sco
and Be ge (2010) sa is y he condi ion.
F om a non-Bayesian pe spec i e, he in o ma ion c i e ia I∗
γ0a e as in Equa ion 4 a e
subs i u ing R2by R2
γand ρby ρ|γ|, whe e ρ|γ|is an inc easing unc ion in |γ|such as
ρ|γ|=|γ|.
We could use he me hod in Sec ion 4.1 o elease di e en ially p i a e e sions o all he
Bγ0o Iγ0. Howe e , i we eleased hose 2ps a is ics, he a iance o he p i a e s a is ics
would inc ease exponen ially in p. Ins ead, we p opose wo king wi h a pe u bed e sion o
a su icien s a is ic whose dimension inc eases quad a ically in p.
Le Z= (I−PX0)Y∈
R
n×1be he null-cen e ed ou come a iable and V∈
R
n×pbe
he design ma ix o he ull model ha includes all pp edic o s, which we collec in a da a
ma ix D= [V;Z]. Assuming Z0Z > 0 almos su ely, we de ine
G=D0D=V0V V 0Z
Z0V Z0Z=V0V V 0Y
Y0V Y 0(I−PX0)Y.
The G am ma ix Gis a su icien s a is ic o he no mal linea model (see, o example,
Sebe and Lee (2012)). As a consequence, all o he R2
γ,Bγ0, and Iγ0can be cons uc ed
by aking app op ia e subse s o G.
We p opose eleasing a noisy e sion o he su icien s a is ic G, a echnique known in he
di e en ial p i acy li e a u e as su icien -s a is ic pe u ba ion (see, o example, McShe y
and Mi ono (2009); Vu and Sla ko ic (2009) and Be ns ein and Sheldon (2019)).
We cons uc a di e en ially p i a e e sion o Gby adding a andom pe u ba ion e m,
de ining G∗=G+E, whe e Eis a andom pe u ba ion ma ix ha ensu es di e en ial p i-
acy. To ensu e ε-di e en ial p i acy, we use he Laplace mechanism. Fo (ε, δ)-di e en ial
16
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
p i acy wi h 0 <ε<1 and 0 < δ ≤1, we use Algo i hm 2 in She e (2019), which we e e
o as he Wisha mechanism.
To es ablish he pa ame e s o he dis ibu ion o E, we assume ha he e a e lowe
and uppe bounds o he da a: ha is, he e a e land usuch ha each en y dij in Dis
wi hin he in e al [l, u]. Since he esponse and p edic o s a e cen e ed, l < 0 and u > 0.
The en ies o G=D0Da e o he o m Pn
i=1 dijdij0. He e, d1jis he i s ow and j h
column o D. I we eplace his en y by ano he one, say ˜
d1j0, hen he maximum absolu e
di e ence in he en ies o Gis |d1jd1j0−˜
d1j˜
d1j0|. When compu ing he sensi i i y ∆1, we
only conside he en ies whe e j≥j0because Gis symme ic. The e o e, he sensi i i y
∆1can be uppe -bounded as ollows:
∆1= sup
D∼
e
DkD0D−˜
D0˜
Dk1
= sup
D∼
e
DX
j≥j0|d1jd1j0−˜
d1j˜
d1j0|
=X
j≥j0
sup
D∼
e
D|d1jd1j0−˜
d1j˜
d1j0|
≤(p+ 1)(p+ 2) sup
d11, d110∈(u,l)|d11d110|
≤(p+ 1)(p+ 2)(l2∨u2).
Fo he Laplace mechanism, we de ine he pe u ba ion e m Eas a symme ic andom
ma ix o he o m
E=
e11 e12 ... e1,p+1
e12 e22 ... e2,p+1
.
.
..
.
.....
.
.
e1,p+1 e2,p+1 ... ep+1,p+1
,
whe e ejj0iid
∼L1(0,∆1/ε), o j≥j0, and ej0j=ejj0. Fo he Wisha mechanism (i.e.,
Algo i hm 2 in She e , 2019), we need a uni o m bound on he Euclidean no m o he ows
o D. Gi en he assump ion l < dij < u, we can use p(p+ 1)(l2∨u2) as a uni o m bound.
In his case, gi en he bound, we de ine he andom pe u ba ion as E=M−E(M), whe e
M∼Wp+1(k, (p+ 1)(l2∨u2)Ip+1) and k=bp+ 1 + 28 log(4/δ)/ε2cwi h 0 < ε < 1 and
0< δ ≤1.
Fo bo h mechanisms, he a iances o he en ies in Ecan be high, especially o
small alues o εo δ. This, in u n, can lead o ou pu s ha o e es ima e he numbe o
ac i e p edic o s. To a oid his issue, we p opose pos -p ocessing G∗in wo ways: ha d-
h esholding o -diagonal elemen s as in Bickel e al. (2008) and adding a cons an o he
diagonal elemen s.
We p opose h esholding he o -diagonal en ies o G∗a eλ, he λ- h pe cen ile o Eij.
Mo e o mally, we de ine G∗∗ o be he ma ix wi h ypical elemen G∗
ij
1
(i=jo |G∗
ij| ≥
eλ). This choice can be jus i ied as ollows: i an o -diagonal en y o G∗
ij =Gij +Eij is
no an ex eme alue in he dis ibu ion o Eij, i is likely ha G∗
ij is essen ially Eij and
Gij is nea ly ze o.
Adding a cons an o he diagonal elemen s o G∗can be seen as a idge- ype o
egula iza ion ha can educe he a iabili y o he ou pu . I can also be used o gua an ee
17
Pe˜
na and Ba ien os
ha he ou pu is posi i e-de ini e because, a e adding he Laplace pe u ba ion e ms, G∗
may no be posi i e-de ini e. In ac , he ha d- h esholded ma ix G∗∗ need no be posi i e-
de ini e e en i G∗is posi i e-de ini e (Bickel e al., 2008). Gi en a symme ic ma ix A,
which he e can be ei he G∗o G∗∗, he ma ix A =A+ Ip+1 is posi i e-de ini e as long
as > −eigmin(A), whe e eigmin(·) is a unc ion e u ning he minimum eigen alue o a
ma ix.
In essence, we p opose eleasing a di e en ially p i a e G am ma ix G∗, which we hen
ans o m o ob ain a be e es ima e o G. Then, we use he o mulas we would use i we
had access o Ga e plugging in i s di e en ially p i a e es ima e.
In ou applica ions (in Sec ion 5.2), we conside me hods ha ha d- h eshold G∗and
me hods ha do no . Tha is, we compa e me hods ha a e based on G∗∗
=G∗∗+ Ip+1 o
me hods based on G∗
=G∗+ Ip+1, whe e G∗is no ha d- h esholded. In ou expe ience,
ha d- h esholding is help ul when he g ound u h is spa se, bu i can be de imen al
when mos p edic o s a e ac i e. We jus i y his a gumen in Sec ion 5.2.
P oposi ion 2 es ablishes model-selec ion consis ency unde some assump ions. Mo e
p ecisely, we show ha he di e en ially p i a e Bayes ac o o any model γ o he ue
model, which can be exp essed as B∗
γT =B∗
γ0/B∗
T0, con e ges o ze o in p obabili y o
any γ6=T. The di e en ially p i a e in o ma ion c i e ia I∗
γT a e also consis en unde he
assump ions lis ed below. We p o ed he esul by cha ac e izing he asymp o ic beha io
o G∗
,G∗∗
, and R2,∗
γ, and hen bounding he Bayes ac o s and in o ma ion c i e ia abo e
and below. Jus as we had in Sec ion 4, he p oo co e s εand (ε, δ) di e en ially p i a e
me hods. The p oo does no ollow om Liang e al. (2008) o se e al easons, one o
hem being ha we a e no assuming ha he esponse gi en he co a ia es is no mal, since
we assume ha he da a a e bounded.
P oposi ion 2 Le T∈ {0,1}pbe a ec o indexing he uly ac i e p edic o s. As n→ ∞,
and unde he egula i y condi ions lis ed below, B∗
γT →P0and I∗
γT →P0 o any γ6=T.
1. Boundedness: The da a Da e wi hin he in e al [l, u] o ini e land u.
2. Reg ession mean and a iance: E(Y|X0, V ) = X0ψ+VTβTand Va (Y|
X0, V ) = σ2
TIn, whe e VT∈
R
n×pTis a ma ix ha con ains he uly ac i e p e-
dic o s.
3. P i acy pa ame e s: The p i acy pa ame e s εand δa e such ha he ma ix
pe u ba ion e m E/n →P0.
4. Regula iza ion pa ame e s: λis ixed and is so ha limn→∞ /n = 0.
5. Design ma ices: limn→∞ V0V/n =S1, whe e S1is symme ic and posi i e-de ini e.
6. P io s on gand penal ies ρ: o all 1≤ |γ| ≤ p, he p io π(g)sa is ies
lim
n→∞Z∞
0
np/2(g+ 1)−|γ|/2π(g)ν(dg)<∞
lim
n→∞Z∞
n
np/2(g+ 1)−|γ|/2π(g)ν(dg)>0,
18
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
o om a non-Bayesian pe spec i e, ρ|γ|is inc easing in |γ|and sa is ies |γ| ≤ ρ|γ|≤
|γ|∨log n.
The amewo k he e is di e en o he one in Sec ion 4. We assume ha he con iden ial
da a a e bounded (Assump ion 1) and do no assume ha he dis ibu ion o he ou come
gi en he p edic o s is no mal. In o he wo ds, Bayes ac o s and in o ma ion c i e ia a e
misspeci ied beyond he addi ion o he pe u ba ion e m E. None heless, P oposi ion 2
shows ha he me hods a e consis en . Ou se up is also dis inc o he one adop ed in Lei
e al. (2018), whe e i is simul aneously assumed ha he esponse is no mal (Assump ion 1
in Lei e al. (2018)) and bounded (Assump ion 4). Assump ion 2 equi es ha he eg ession
mean be well-speci ied and ha he co a iance o he esponse gi en he p edic o s be
sphe ical. Assump ion 3 o ces he pe u ba ion ma ix E o be so ha E/n →P0. As we
had in Sec ion 4, i su ices o le εand δbe ixed o i o hold. Assump ion 4 imposes
condi ions on he egula iza ion pa ame e s. The case o a non- h esholded ma ix G∗
is
included as λ= 0. Assump ion 5 is simila o he egula i y condi ion on design ma ices
in P oposi ion 1. Finally, Assump ion 6 is essen ially he same as he assump ions on he
p io s and penal ies in 2. Jus as we had in Sec ion 4, he di e en ially p i a e Bayes ac o s
wi h Zellne ’s g-p io wi h g=n, he obus p io , Zellne -Siow, and BIC a e all consis en ,
and so is BIC.
In he common scena io whe e X0is an in e cep 1n, he me hods desc ibed in his
sec ion can be con enien ly implemen ed in Rwi h he bas.lm unc ion in lib a y(BAS)
(Clyde, 2020). Gi en a se o p edic o s and an ou come a iable, he bas.lm unc ion
enume a es Bayes ac o s o small o mode a e pand samples om he model space o la ge
p. The unc ion ou pu s o he s a is ics o in e es such as pos e io inclusion p obabili ies
and model-a e aged es ima es. To use bas.lm o ou p oblem, we need o gene a e a
syn he ic da a se D(con aining bo h cen e ed p edic o s and ou come) whose su icien
s a is ic D0Dis equal o a ixed G am ma ix G,which can be G∗
o G∗∗
. P oposi ion 3
below shows how o ob ain such a syn he ic da a se D= [V;Z] gi en a ma ix G.
P oposi ion 3 Le U ∈
R
n×(p+1) be a ull- ank ma ix and de ine M= (In−1n10
n/n)U.
Gi en a ma ix G, we can gene a e a syn he ic da a se D= [V;Z]wi h he o mula D=
M(M0M)−1/2G1/2. The syn he ic da a se Dsa is ies he iden i ies D0D=G,V01n= 0p,
and Z01n= 0.
P oposi ion 3 gua an ees ha he ou pu s we ob ain om unning bas.lm on he syn-
he ic da a Da e iden ical o wha we would ind by aking subse s o Gdi ec ly. In he
p oposi ion, he ma ix Uis a bi a y; in p ac ice, i s en ies can be simula ed by sampling
independen ly om he uni o m dis ibu ion.
5.1 Quan i ying he Unce ain y In oduced by he Mechanism
Wi h he non- h esholded me hods, he p i a e s a is ics a e o he o m G∗
=G+E+ Ip+1.
Since and he dis ibu ion o Ea e bo h known, we can de ine a con idence se o he non-
p i a e Ggi en G∗
. Wi h such a se , i is possible o ind con idence egions o summa ies
o in e es T(G) like leas -squa es es ima es o inclusion p obabili ies.
To de ine a 1−αcon idence se o G, we i s ind E1−αsuch ha P(E∈ E1−α)=1−α.
Fo a ma ix no m k·k, de ine E1−α={E:kE−E(E)k ≤ q1−α}whe e P[kE−E(E)k ≤
19
Pe˜
na and Ba ien os
q1−α] = 1 −α. Then, ˜
C1−α={G∗
− Ip+1 −E:E∈ E1−α}is a 1 −αcon idence se o G.
Since Gis symme ic and posi i e-de ini e, we can in e sec ˜
C1−αwi h he se o symme ic
posi i e-de ini e ma ices S++ o de ine a 1−αcon idence egion C1−α=˜
C1−α∩S++ whose
olume is a mos ha o ˜
C1−α. The con idence se C1−αcan be ans o med in o T(C1−α)
o p oduce con idence se s o summa ies o in e es T(G).
We can app oxima e T(C1−α) wi h a ejec ion sample . Fi s , simula e E1, E2, ... , Ensim
om he app op ia e mechanism and compu e kEi−E(E)k o i∈ {1,2, ... , nsim}. Then,
app oxima e he ((1 −α)×100) h pe cen ile q1−αwi h i s empi ical e sion ˆq1−α= in {q:
Pnsim
i=1
1
(kEi−E(E)k ≤ q)/nsim ≥1−α}and de ine ˆ
E1−α={Ei:kEi−E(E)k ≤ ˆq1−α, i =
1, . . . , nsim}. A e ha , we can ind T(ˆ
C1−α); ha is, compu e T(G∗
− Ip+1 −Ei) o
Ei∈ˆ
E1−αand only keep hose such ha G∗
− Ip+1 −Ei∈ S++. This compu a ional
s a egy o cons uc T(C1−α) is an applica ion o he pa ame ic boo s ap: he quan i y
o be in e ed is he unknown con iden ial summa y T(G), and he only sou ce o andomness
is he noise injec ed in o G o make i di e en ially p i a e.
In gene al, he con idence se need no be an in e al, bu we can summa ize he con i-
dence se wi h a his og am. To do so, we de ine he bins o he his og am as Bk= [ k−1, k)
wi h min{T(ˆ
C1−α)}= 0< 1< . . . < K−1< K= max{T(ˆ
C1−α)}and hei co -
esponding ela i e equencies #Bkusing he numbe o elemen s o T(ˆ
C1−α) ha all
in Bk,k= 1, . . . , K. We deno e he his og am summa izing T(ˆ
C1−α) as His (T, ˆ
C1−α) =
{(Bk,#Bk)}K
k=1. We can also epo his og ams in a densi y scale; ha is, His (T, ˆ
C1−α) =
{(Bk, dk)}K
k=1, whe e
dk=#Bk
|T(ˆ
C1−α)|( k− k−1)
and |T(ˆ
C1−α)|deno es he ca dinali y o T(ˆ
C1−α). While His (T, ˆ
C1−α) displays he dis i-
bu ion o he es ima o o T(G) cons ained o T(ˆ
C1−α), i s suppo , deno ed by [ 0, K],
co esponds o an 1 −αcon idence in e al. This is because 0= min T(ˆ
C1−α) and
K= max T(ˆ
C1−α). The e o e, analys s can di ec ly epo [ 0, K] as he con idence in-
e al, i desi ed. Addi ionally, analys s can use he his og am o assess whe he [ 0, K]
p o ides a good ep esen a ion o he se T(ˆ
C1−α). They can also conside i T(ˆ
C1−α) would
be be e ep esen ed by he union o non-adjacen in e als.
5.2 Empi ical E alua ions
We e alua e he pe o mance o he me hods desc ibed in his sec ion in a simula ion s udy
and an applica ion. The simula ion s udy is simila o he one in Liang e al. (2008),
whe eas he eal da a se is a subse o he Ma ch 2000 Cu en Popula ion Su ey ha
was analyzed in Ba ien os e al. (2019).
We implemen he me hods wi h he Rpackage BAS (Clyde, 2020). In he simula ion
s udy, we ound Bayes ac o s wi h he Zellne -Siow p io (ZS) and in o ma ion c i e ia
wi h BIC. The p io dis ibu ion on he model space π(γ) is he hie a chical uni o m p io
p oposed in Sco and Be ge (2010). F om a non-Bayesian pe spec i e, π(γ) ac s as a
unc ion ha weighs he in o ma ion c i e ia. The esul s wi h Zellne -Siow and BIC a e
almos iden ical. We epo he ou pu s based on he o me he e and show he esul s wi h
he la e in Appendix C.
20
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
We compa e he esul s we ob ain by ha d- h esholding and no h esholding he G am
ma ix G∗. In all cases, we add a egula iza ion pa ame e o he diagonal en ies o G∗.
Fo he Laplace mechanism, we se o be he 99- h pe cen ile o eigmin(E), which we ind
ia simula ion. Fo he Wisha mechanism, we use he analy ical exp ession in Rema k 2
o She e (2019).
5.2.1 Simula ion S udy
We simula e da a om a no mal linea model wi h pp edic o s, whe e pis se o 2, 6,
o 9. The sample size n(in housands) a ies om 5 o 10,000. The numbe o ac i e
p edic o s in he ue model |T|depends on he alue o pand anges om 0 (null model
is ue) o p( ull model is ue). Speci ically, i p= 2, we se |T| ∈ {0,1,2}; i p= 6, we
se |T|∈{0,3,6}; and i p= 9, we se |T|∈{0,4,9}. The p edic o s a e independen ly
d awn om he uni o m dis ibu ion on (−2,2). Following Has ie e al. (2017), we de ine
he signal- o-noise a io (SNR) as he a iance o he eg ession mean (which is andom,
since we a e simula ing p edic o s and β) di ided by σ2. In ou simula ions, we assume
ha he in e cep is ze o and βis a p-dimensional ec o equal o b[1,...,1]0. We use
op imiza ion o ind σ2and bsuch ha SNR = 0.5 and he esponse alls wi hin (−2,2)
wi h high p obabili y. Fo each combina ion o |T|and n, we simula e 1,000 da a se s.
All he da a se s we simula ed a e such ha he esponse alls in (−2,2). We conside
ε∈ {0.5,0.9}and, in he case o he Wisha mechanism, we se δ= 1/n.
We assess he pe o mance o he me hods by acking Mon e Ca lo a e ages o p edic-
i e mean squa ed e o s and he pos e io p obabili y o he ue model. We de ine he
p edic i e mean squa ed e o as PMSE = n−1kVTβT−V β∗k2
2,whe e VTis a design ma ix
con aining uly ac i e p edic o s, βTis he ue alue o β, and β∗is he di e en ially
p i a e model-a e aged pos e io expec a ion.
Figu e 3 displays PMSEs o di e en alues o n,p,ε, and |T|. As expec ed, he
PMSEs o bo h p i a e and non-p i a e app oaches dec ease as he sample size inc eases.
We obse e ha he PMSEs o he di e en ially p i a e me hods a e smalle when εis 0.9
compa ed o when εis 0.5, and hey a e always highe han he non-p i a e PMSEs. This
is expec ed, as la ge alues o εshould lead o g ea e s a is ical u ili y. In mos cases,
he me hods based on he Laplace mechanism ha e a lowe PMSE han hose based on
he Wisha mechanism. Howe e , he Wisha mechanism seems o pe o m be e in he
case when pis ei he 6 o 9, |T|is ze o, and he sample size is small. Al hough his is
sligh ly less e iden in he igu e, upon close inspec ion, we can see ha me hods based
on ha d- h esholding end o ha e a sligh ly lowe PMSE when |T|is ze o bu cease o be
ad an ageous when |T|is la ge and he sample size is small.
Figu e 4 displays he pos e io p obabili ies o he ue model, o he same alues o n,
p,ε, and |T| ha we used in Figu e 3. The esul s a e consis en wi h wha we obse ed
o PMSEs. We also obse e ha , al hough all p obabili ies inc ease as he sample size
inc eases, he a e a which hey inc ease depends on p. Fo highe alues o p, he a e o
inc ease is lowe because he compu a ional complexi y o he p oblem inc eases wi h he
dimension o G. The numbe o en ies in Ginc eases quad a ically in p, and so does he
a iance o he pe u ba ion e m added o ensu e di e en ial p i acy. This ac a ec s he
con e gence a e o he pos e io p obabili ies.
21
Pe˜
na and Ba ien os
p = 9
|T| = 0
p = 9
|T| = 4
p = 9
|T| = 9
p = 6
|T| = 0
p = 6
|T| = 3
p = 6
|T| = 6
p = 2
|T| = 0
p = 2
|T| = 1
p = 2
|T| = 2
10 100 1000 10000 10 100 1000 10000 10 100 1000 10000
1e−12
1e−08
1e−04
1e−12
1e−08
1e−04
1e−12
1e−08
1e−04
n ( housands)
PMSE wi h Bayesian model a e aging
Me hod/Mechanism
G (no DP)
Lap Non− h esh (DP)
Wish Non− h esh (DP)
Lap Th esh (DP)
Wish Th esh (DP)
ε
0.5
0.9
Figu e 3: Simula ion s udy: Sample size (x-axis) agains log(PMSE) (y-axis) wi h Zellne -
Siow p io .
5.2.2 Applica ion: Cu en Popula ion Su ey
The da a se includes n= 49,436 heads o households wi h non-nega i e incomes. We
conside 6 p edic o s: age in yea s (β1), age squa ed (β2), ma i al s a us (β3), sex (β4),
educa ion (β5), and ace (β6). All p edic o s a e nume ic o bina y excep o educa ion,
which is an o dinal a iable. To educe he numbe o coe icien s in he model, we ea
educa ion as nume ic, anging om 1 ( o less han 1s g ade) o 16 ( o doc o al deg ee).
The bina y p edic o s a e: ma i al s a us (1: ci ilian spouse p esen ; 0: o he wise), sex (1:
male; 0: emale), and ace (1: whi e; 0: o he wise). The esponse a iable is income.
In his applica ion, he non-p i a e inclusion p obabili ies a e all close o one. To p o ide
a mo e challenging benchma k o ou me hods, we pe mu e he ows o ma i al s a us and
educa ion in he design ma ix o a i icially make he inclusion p obabili ies o β3and
β5close o ze o. The p edic o s and he esponse a e cen e ed and escaled o he in e al
(−0.5,0.5).
Figu e 5 displays he pos e io expec ed alues o β1,β3, and β4wi h he Zellne -Siow
p io and ε= 0.9.
We use he his og ams desc ibed in Sec ion 5.1 o de ine app oxima e 95% con idence
se s o T(G) = E(βj|G). Ou choice o ma ix no m is he F obenius no m. Speci ically, we
un ou p ocedu e 250 imes and, o each un and a ixed collec ion o bins B1,...,BK, we
summa ize each T(ˆ
C1−α) wi h i s co esponding his og am His (T, ˆ
C0.95) = {(Bk, dk)}K
k=1.
22
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
p = 9
|T| = 0
p = 9
|T| = 4
p = 9
|T| = 9
p = 6
|T| = 0
p = 6
|T| = 3
p = 6
|T| = 6
p = 2
|T| = 0
p = 2
|T| = 1
p = 2
|T| = 2
10 100 1000 10000 10 100 1000 10000 10 100 1000 10000
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
n ( housands)
Pos e io p obabili y ue model
Me hod/Mechanism
G (no DP)
Lap Non− h esh (DP)
Wish Non− h esh (DP)
Lap Th esh (DP)
Wish Th esh (DP)
ε
0.5
0.9
Figu e 4: Simula ion s udy: Sample size (x-axis) agains pos e io p obabili y o he ue
model (y-axis) wi h Zellne -Siow p io .
I we le d(l)
1, . . . , d(l)
Kbe he densi ies alues o he his og am associa ed wi h he l- h un,
l= 1,...,250, we de ine a e age his og ams as
His T(·) = E(βj| ·),ˆ
C0.95=( Bk, dk=1
250
250
X
l=1
d(l)
k!)K
k=1
, j ∈ {1,3,4}.
The esul s a e displayed in Figu e 5. In all cases, he di e en ially p i a e me hods
a e close o he non-p i a e answe s. We can also see ha he his og ams a e use ul o
quan i ying he unce ain y in oduced by he mechanism, since hei sp ead inc eases when
he h esholded and non- h esholded me hods do no ag ee in hei es ima es.
5.3 Guidelines
Th esholded me hods end o pe o m bes when he ue numbe o p edic o s is small.
On he o he hand, when mos p edic o s a e ac i e, non- h esholded me hods end o ou -
pe o m h esholded me hods. The oo cause behind his phenomenon is ha h esholding
sh inks he elemen s in G∗, which p omo es spa si y.
In p ac ice, we ecommend ha use s un analyses wi h bo h h esholded and non-
h esholded me hods. This can be done wi hou a ec ing he p i acy budge o he analys
23
Pe˜
na and Ba ien os
pos e io expec ed alue o β1
densi y
−2 −1 0 1 2
0
1
2
3
4
Laplace mechanism
pos e io expec ed alue o β3
densi y
−0.010 0.000 0.010
0
500
1000
1500
2000
Laplace mechanism
pos e io expec ed alue o β4
densi y
0.00 0.01 0.02 0.03 0.04
0
20
40
60
80
Laplace mechanism
pos e io expec ed alue o β1
densi y
−2 −1 0 1 2
0
1
2
3
4
Wisha mechanism
pos e io expec ed alue o β3
densi y
−0.010 0.000 0.010
0
500
1000
1500
2000
Wisha mechanism
pos e io expec ed alue o β4
densi y
0.00 0.01 0.02 0.03 0.04
0
20
40
60
80
Wisha mechanism
Figu e 5: Cu en popula ion su ey: Pos e io expec a ions o coe icien s β1, β3, and β4
wi h Zellne -Siow p io and ε= 0.9. The e ical solid lines a e he non-p i a e
pos e io expec a ions, whe eas he dashed lines and do ed lines a e a e aged
pos e io expec a ions es ima ed wi h non- h esholded me hods and h esholded
me hods, espec i ely.
because bo h G∗
and he h esholded ma ix G∗∗
a e pos -p ocessed e sions o he same
di e en ially p i a e ma ix G∗.
Finally, we ecommend epo ing con idence se s whene e possible, since we ind hem
o be a aluable ool o quan i ying he unce ain y in oduced by he mechanisms.
6. Conclusions and Fu u e Wo k
In his a icle, we p oposed di e en ially p i a e me hods o hypo hesis es ing, model
a e aging, and model selec ion o no mal linea models. Unde egula i y condi ions, he
me hods a e consis en . The egula i y condi ions we ha e imposed a e simila o he
condi ions used in he li e a u e o es ablishing consis ency o non-di e en ially p i a e
me hods.
Ou me hods o hypo hesis es ing a e based on da a-spli ing and censo ing s a is ics.
We ha e s udied he e ec s o hese ope a ions on he pe o mance o he me hods. In
he case o da a-spli ing, inc easing he numbe o subse s educes he a iance o he
di e en ially p i a e s a is ics, bu i adds bias. In he case o censo ing, mo e s ingen
24
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
censo ing educes he a iance, bu i can lead o subs an ial bias i he ue, con iden ial
s a is ic lies ou side he uncenso ed ange.
The me hods we p oposed o model a e aging and selec ion a e based on a pe u bed
su icien s a is ic. I we suspec ha he g ound u h is spa se, we ecommend ha d-
h esholding he pe u bed su icien s a is ic; howe e , i mos p edic o s a e ac i e, ha d-
h esholding can lead o unde i ing.
The me hodology p oposed he e could be ex ended in a numbe o ways. I would be
use ul o ex end he me hods o gene alized linea models h ough he amewo k p oposed
in Li and Clyde (2018). The implemen a ion o hypo hesis es ing is s aigh o wa d,
bu ou me hods o model unce ain y, which a e based on su icien s a is ics, canno be
applied di ec ly. This obs acle can be o e come using app oxima e su icien s a is ics, as
p oposed in Huggins e al. (2017). This app oach has been used success ully in es ima ion
p oblems unde di e en ial p i acy cons ain s in Kulka ni e al. (2021). I would also be
in e es ing o ex end he me hods o su i al models because heal h eco ds a e con iden ial.
In his case, he ex ension could adap he amewo k p oposed in Cas ellanos e al. (2021).
In ou wo k, we ha e used o - he-shel echniques o es ablishing di e en ial p i acy.
While we obse e ha ou p oposals can be use ul in p ac ice, i migh be possible o design
mo e e icien mechanisms ha a e speci ically ailo ed o he asks we conside ed. This is
ano he in e es ing a enue o u u e esea ch.
Acknowledgmen s
The au ho s would like o hank he eedback om wo anonymous e iewe s ha g ea ly
imp o ed he p esen a ion and con en s o he a icle. The esea ch o he second au ho was
suppo ed by he Na ional Science Founda ion Na ional Cen e o Science and Enginee ing
S a is ics [49100420C0002 and 49100422C0008] and he Tes Resou ce Managemen Cen e
(TRMC) wi hin he O ice o he Sec e a y o De ense (OSD), con ac #FA807518D0002.
25
Pe˜
na and Ba ien os
We can show ha Q/(n−p0)→Pσ2
Twi h he Hanson-W igh inequali y, which in u n
implies ha Q/n →Pσ2
T. We ha e E[Q/(n−p0)] = σ2
T. De ine W=Y−E(Y). Then,
E(W) = 0nand W2
ia e uni o mly bounded since bo h Yand E(Y) a e by Assump ion 1.
The e o e, we can pick a ini e cons an Ksa is ying E[exp(W2
i/K2)] <2.
Applying he Hanson-W igh inequali y, o any gi en > 0,
P[|Q/(n−p0)−σ2
T|> ]≤2e−2c min[(K2k(I−PX0)/(n−p0)kF)−1,(k(I−PX0)/(n−p0)kop)−1]→n→∞ 0,
because k(I−PX0)/(n−p0)kop =k(I−PX0)/(n−p0)kF= 1/√n−p0→0 as n→ ∞. This
implies Q/n →Pσ2
Tand Z0Z/n →Pσ2
T+β0
TS3βT.
We ha e shown ha
G/n →PG/n =S1S2βT
β0
TS0
2σ2
T+β0
TS3βT=G∞.
The e o e, we ha e G∗/n →PG∞.
In he case o he non- h esholded ma ix G∗
, we ha e es ablished ha G∗
/n =G∗/n +
/nIp+1 →PG∞since /n →0 by Assump ion 4. In he case o G∗∗
, he e is an indica o
ha can ha d- h eshold o -diagonal elemen s. Le G∗∗
ij /n =G∗
ij/n
1
(i=jo |G∗
ij|/n ≥
eλ/n) be he (i, j)- h en y o G∗∗. On he one hand, limn→∞ eλ/n = 0 because he a iance
o Eij is ini e and does no depend on n. Fo i=j he indica o is equal o 1. Fo i6=j:
E[
1
(|G∗
ij|/n ≥eλ/n)] = P[|G∗
ij|/n ≥eλ/n]→1
Va [
1
(|G∗
ij|/n ≥eλ/n] = P[|G∗
ij|/n ≥eλ/n]−P[|G∗
ij|/n ≥eλ/n]2→0
1
(|G∗
ij|/n ≥eλ/n)→P1.
By Slu zky’s lemma, we ha e ha G∗∗
ij /n →G∞,ij o all i, j. This is enough o show ha
G∗/n →G∞. Finally, since G∗∗
=G∗∗ + Ip+1 and /n →0 by Assump ion 4, we ha e
G∗∗
/n →PG∞, as equi ed.
P oposi ion 6 (Con e gence o noisy R2) Gi en G∗
o G∗∗
and a model-indexing ec o
γ∈ {0,1}p, we can cons uc a di e en ially p i a e e sion o R2
γ, deno ed R2,∗
γ. Le T∈
{0,1}pbe he index o he ue model. Unde he egula i y condi ions s a ed in P oposi ion 5
and assuming ha Z0Zis no equal o ze o almos su ely, he ollowing a e ue:
1. I γdoes no nes T(i.e. i he e exis s isuch ha Ti= 1 and γi= 0), hen
R2,∗
γ→R2
γ,∞and R2,∗
T→R2
T,∞wi h R2
γ,∞< R2
T,∞.
2. I he T= 0pis he null model, hen nR2,∗
γis in Op(1).
3. I γnes s T(i.e. i Ti= 1 implies γi= 1), hen [(1 −R2,∗
T)/(1 −R2,∗
γ)](n−p0)/2is in
Op(1).
32
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
P oo We p o e he h ee s a emen s sepa a ely.
P oo o s a emen 1. The p oo s o R2
γ→PR2
γ,∞and R2,∗
γ→PR2
γ,∞a e a di ec
consequence o P oposi ion 5. No e ha
R2
γ=Z0Vγ(V0
γVγ)−1V0
γZ
Z0Z=Z0PVγZ/n
Z0Z/n .
On he one hand, n/Z0Z→P1/(σ2
T+β0
TS3βT). Then, V0
γZa e sub ec o s o V0Z, so
V0
γZ/n con e ges o a sub ec o o S2βT. Simila ly, V0
γVγ/n con e ges o a subma ix o S1
and, since we assume ha Vis ull- ank, V0
γVγis in e ible o any γ. This implies ha
R2
γcon e ges in p obabili y o some cons an R2
γ,∞, which has o be be ween ze o and one
because Vγ(V0
γVγ)−1Vγis a p ojec ion ma ix and Z0PXZ≤Z0Z o any p ojec ion ma ix
PX.
The con e gence o he noisy R2,∗
γ o R2
γ,∞can be es ablished a e no ing ha R2,∗
γcan
be cons uc ed by aking subma ices o G∗
o G∗∗
and mul iplying and di iding e ms as
needed. We can in oke P oposi ion 5 and Slu zky’s lemma and conclude ha R2,∗
γ→PR2
γ,∞,
as equi ed.
I emains o show ha i γdoes no nes he ue model T,R2
γ,∞< R2
T,∞. To see his,
no e ha
lim
n→∞
E(Z0PVγZ/n) = lim
n→∞β0
TX0
TPVγXTβT/n, lim
n→∞Va (Z0PVγZ/n) = 0,
and β0
TX0
TPVγXTβT< β0
TX0
TXTβT=β0
TX0
TPVTXTβT, which implies R2
γ,∞< R2
T,∞.
P oo o s a emen 2. I Tis he null model, we show ha nR2,∗
γis in Op(1). I is
use ul o w i e
nR2,∗
γ=
1
√n(Z0V)∗
γh(V0V)∗
γ
ni−1(V0Z)∗
γ1
√n
(Z0Z)∗/n .
By P oposi ion 5, we know ha he denomina o con e ges in p obabili y o a cons an . I
is enough o show ha he nume a o is in Op(1). The ma ix (V0V)∗
γ/n−1is in Op(1) by
P oposi ion 5. I emains o show ha (V0Z)∗
γ/√nis in Op(1). Le w∗= (V0Z) + E2and
kλ he app op ia e h eshold o he o -diagonal elemen s o G∗∗. Then,
k1
√n(V0Z)∗
γk2≤1
nkw∗
1
(|w∗|> kλ)k2
≤1
nkw∗k2
≤1
nkV0Zk2+1
nkE2k2.
I su ices o show ha bo h he e o E2/√nand he non-p i a e V0Z/√na e in Op(1).
Each o he Eij in Ehas a ini e a iance ha does no depend on n. The e o e, each
Eij/√nhas a a iance ha goes o ze o and, since E2has pelemen s, kE2/√nkis in Op(1).
I only emains o show ha V0Z/√nis in Op(1), which we show using he Hanson-W igh
inequali y.
33
Pe˜
na and Ba ien os
Fi s , no e ha kV0Z/√nk2=Z0V V 0Z/n. Then, since we a e assuming ha he ue
model is he null model:
E(Z0V V 0Z/n) = σ2
n (V0V)→n→∞ c > 0.
The limi is a posi i e cons an because V0V/n con e ges o a symme ic posi i e-de ini e
ma ix by Assump ion 5, which also implies limn→∞kV V 0/nkF<∞and limn→∞kV V 0/nkop <
∞.F om he e, we can apply he Hanson-W igh inequali y o es ablish ha |Z0V V 0Z/n −
E(Z0V V 0Z/n)|is in Op(1) and, since E(Z0V V 0Z/n) con e ges o a cons an , we conclude
ha Z0V V 0Z/n is in Op(1), as equi ed.
P oo o s a emen 3. Finally, we show ha i γnes s T, [(1−R2,∗
T)/(1−R2,∗
γ)](n−p0)/2
is in Op(1). Conside he non-p i a e [(1 −R2
T)/(1 −R2
γ)](n−p0)/2. We can w i e
1−R2
T
1−R2
γ(n−p0)/2
=1 + 2
n−p0
Z0(PVγ−PVT)Z
2Z0(In−PVγ)Z/(n−p0)(n−p0)/2
≤exp Z0(PVγ−PVT)Z
2Z0(In−PVγ)Z/(n−p0).
The denomina o Z0(In−PVγ)Z/(n−p0) con e ges o a cons an . This ac ollows di ec ly
gi en he asymp o ic beha io o Z0PVγZ/n and Z0Z/n we jus desc ibed in he p oo o he
i s s a emen . The same is ue o he p i a e e sion o he s a is ic, using he a gumen
we used o R2,∗
γ.
The nume a o Z0(PVγ−PVT)Zis in Op(1), which can be shown using he Hanson-
W igh inequali y. The expec a ion is E[Z0(PVγ−PVT)Z] = σ2(|γ| − |T|). Le Qγ=
Z0(PVγ−PVT)Z−σ2(|γ|−|T|). Applying he Hanson-W igh inequali y
P(Qγ> M)≤2 exp −cM
K2min M
K2(|γ|−|T|),1,
whe e Kis a cons an ha can be chosen in a simila way as we did in P oposi ion 5. The
igh -hand side can be made a bi a ily close o ze o by inc easing M, which es ablishes
ha Qγand Z0(PVγ−PVT)Za e in Op(1). The same is ue o he p i a e e sion o he
s a is ic, using an a gumen which is simila o he one we used o showing ha nR2,∗
γis in
Op(1) when he ue model is he null model. Bo h (V0Z)∗
γ/√nand (V0Z)∗
T/√ncon e ge
o hei p i a e e sions (V0Z)γ/√nand (V0Z)T/√nbecause he e o goes o ze o and
he indica o con e ges (see P oposi ion 5 and he ea lie p oo o nR2,∗
γ o mo e de ailed
e sions o hese a gumen s). The p i a e (V0V)∗
γalso con e ge o hei non-p i a e e sions
(see e.g. P oposi ion 5). The e o e, he p i a e (Z0(PVγ−PVT)Z)∗has he same asymp o ic
beha io as he p i a e Z0(PVγ−PVT)Z, which we ha e shown o be in Op(1). This comple es
he p oo .
A.4 P oo o P oposi ion 2 in main ex
We conside wo cases: one whe e he ue model is he null model and ano he one whe e
he ue model is no he null model.
34
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
T ue model is he null model: Le γbe a model ha is no he null model. Then,
we ha e
B∗
γ0≤exp n−p0
2
R2,∗
γ
1−R2,∗
γ!Z∞
0
(g+ 1)−|γ|/2π(g)ν(dg).
The in eg al con e ges o ze o by Assump ion 6 and he exponen ial e m is in Op(1)
because, by P oposi ion 6, we know ha nR2,∗
γis in Op(1). The e o e, o any model γ
which is no he null model, B∗
γ0→P0. A simila a gumen wo ks o in o ma ion c i e ia.
In such case,
I∗
γ0≤n−ρ|γ|/2exp n
2
R2,∗
γ
1−R2,∗
γ!→P0.
T ue model is no he null model: We s udy he asymp o ic beha io o B∗
γT , whe e
Tis he ue model and γis a model ha is no he ue model. We spli his case in o
wo subcases: one whe e γnes s he ue model, and ano he one whe e γdoes no nes
he ue model.
Fi s , no e ha , since we a e wo king wi h null-based Bayes ac o s,
B∗
γT =B∗
γ0(1 −R2,∗
T)(n−p0)/2
B∗
T0(1 −R2,∗
T)(n−p0)/2
we will bound he nume a o and denomina o sepa a ely and pu ou bounds oge he .
Fi s , we bound he nume a o :
B∗
γ0(1 −R2,∗
T)(n−p0)/2=Z∞
0
(g+ 1)−|γ|/2"1 + g(1 −R2,∗
T)−R2,∗
T
1 + g(1 −R2,∗
γ)#(n−p0)/2
π(g)ν(dg)
≤ 1−R2,∗
T
1−R2,∗
γ!(n−p0)/2Z∞
0
(g+ 1)−|γ|/2π(g)ν(dg).
Then, we bound he denomina o :
B∗
T0(1 −R2,∗
T)(n−p0)/2≥Z∞
n
(g+ 1)−|T|/2"1−R2,∗
T
1 + g(1 −R2,∗
T)#(n−p0)/2
π(g)ν(dg)
≥"1−R2,∗
T
1 + n(1 −R2,∗
T)#(n−p0)/2Z∞
n
(g+ 1)−|T|/2π(g)ν(dg).
Pu ing he bounds oge he and using Assump ion 6:
B∗
γT ≤ 1−R2,∗
T
1−R2,∗
γ!(n−p0)/2 1−R2,∗
T
1 + n(1 −R2,∗
T)!−(n−p0)/2R∞
0(g+ 1)−|γ|/2π(g)ν(dg)
R∞
n(g+ 1)−|T|/2π(g)ν(dg)
.n(|T|−|γ|)/2 1−R2,∗
T
1−R2,∗
γ!(n−p0)/2
exp(R2,∗
γ/(1 −R2,∗
γ)).
35
Pe˜
na and Ba ien os
When γnes s T,n(|T|−|γ|)/2goes o ze o and he emaining e ms a e in Op(1) by P oposi-
ion 6, so B∗
γT con e ges o ze o in p obabili y. When γdoes no nes T, we show ha
An=n(|T|−|γ|)/2[(1 −R2,∗
T)/(1 −R2,∗
γ)](n−p0)/2→P0
Le Rn= (1 −R2,∗
T)/(1 −R2,∗
γ), which con e ges in p obabili y o a cons an less han one.
Taking loga i hms
log nn(|T|−|γ|)/2R(n−p0)/2
no=n−p0
2[(|T|−|γ|) log n/(n−p0) + log Rn],
which di e ges o −∞ in p obabili y, so An→P0 The emaining e m in he uppe bound
o B∗
γT is in Op(1). The e o e, we ha e shown ha B∗
γT →P0.
The p oo o in o ma ion c i e ia is essen ially he same, bu we do no need o bound
in eg als. Indeed,
I∗
γT =n(ρ|T|−ρ|γ|)/2 1−R2,∗
T
1−R2,∗
γ!n/2
,
and we can use he same a gumen s we used o B∗
γT o show ha I∗
γT is consis en .
A.5 P oo o P oposi ion 3 in main ex
Le U ∈
R
n×(p+1) be a ull- ank ma ix and de ine M= (In−PX0)U. Gi en a G am ma ix
G, we can gene a e a syn he ic da a se D= [V;Z] wi h he o mula D=M(M0M)−1/2G1/2.
In o he wo ds, we ha e D0D=G:
D0D=G1/2M(M0M)−1/2M0M(M0M)−1/2G1/2=G1/2G1/2=G.
The syn he ic da a a e also cen e ed ( he same way ha Vand Za e cen e ed in ou
cons uc ion o D). This is ue because Dis p e-mul iplied by In−PX0, so i is o hogonal
o he span o X0= 1n.
Appendix B. E ec s o censo ing and da a-spli ing: High School and
Beyond Su ey
In his sec ion, we e isi he High School and Beyond Su ey da a se (Sec ion 4.4) o s udy
he e ec s o se ing di e en censo ing limi s. Fo conc e eness, we es ic ou a en ion
o he es o H02 agains H12; ha is, he hypo hesis es whe e we wish o know i ead
sco es a e p edic i e o ma h sco es when science sco es a e al eady a co a ia e in he
model. The esul s o he es o H01 agains H11 a e simila . The simula ion se up is he
same as desc ibed in Sec ion B. The only changes a e he censo ing limi s.
In Figu e 6, we display he esul s o he di e en ially p i a e pos e io p obabili y o
H12 o di e en alues o εand M. We censo he pos e io p obabili y o H12 a [0.35,0.65],
[0.25,0.75], [0.01,0.99], and [0.001,0.999]. The ue, non-p i a e pos e io p obabili y o
H12 is nea 1. Fo all censo ing limi s, inc easing he numbe o subg oups dec eases he
a iance o he ou pu , bu i induces a bias ha sh inks he p obabili y o 0.5. Mo e
s ingen censo ing limi s educe he a iance as well, bu come a he cos o po en ial
36
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
bias: o ins ance, censo ing he pos e io p obabili y a [0.35,0.65] is clea ly oo s ingen ,
since he ue pos e io p obabili y is much highe han he uppe limi .
In Figu e 7, we display he analogous esul o likelihood a ios. In his case, he lowe
censo ing limi is se o L= 0, which is a na u al lowe bound o likelihood a ios. The g ay
+ and Ma e calib a ed c i ical alues o ejec ion o H02 a signi icance a le els 0.01 and
0.05, espec i ely. The uppe censo ing limi s Ua e se o 1, 2, 7, and 10. We a i e a he
same conclusions we eached wi h he Bayesian analysis. The ue, non-p i a e likelihood
a io is abo e 10, and i we censo a a much lowe alue (such as 1 o 2) he es ails o
be powe ul. The es pe o ms bes i he lowe bound is 10, in which case he es is qui e
powe ul, especially o M≥5 and δ≥0.25.
P(H12| D) in [0.001, 0.999]
M = 2
P(H12| D) in [0.001, 0.999]
M = 5
P(H12| D) in [0.001, 0.999]
M = 10
P(H12| D) in [0.01, 0.99]
M = 2
P(H12| D) in [0.01, 0.99]
M = 5
P(H12| D) in [0.01, 0.99]
M = 10
P(H12| D) in [0.25, 0.75]
M = 2
P(H12| D) in [0.25, 0.75]
M = 5
P(H12| D) in [0.25, 0.75]
M = 10
P(H12| D) in [0.35, 0.65]
M = 2
P(H12| D) in [0.35, 0.65]
M = 5
P(H12| D) in [0.35, 0.65]
M = 10
0.25 0.50 1.00 2.00 4.00 8.00 0.25 0.50 1.00 2.00 4.00 8.00 0.25 0.50 1.00 2.00 4.00 8.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
..
P(H12| D)
Figu e 6: Dis ibu ion o P∗(H12 |D) as a unc ion o ε,M, and censo ing limi s. The lowe
endpoin o he e o ba s is he i s qua ile o he dis ibu ion, he midpoin is
he median, and he uppe endpoin is he hi d qua ile. The dashed lines a e
he non-p i a e pos e io p obabili ies.
37
Pe˜
na and Ba ien os
U = 10
M = 2
U = 10
M = 5
U = 10
M = 10
U = 7
M = 2
U = 7
M = 5
U = 7
M = 10
U = 2
M = 2
U = 2
M = 5
U = 2
M = 10
U = 1
M = 2
U = 1
M = 5
U = 1
M = 10
0.01 0.25 0.50 0.75 0.01 0.25 0.50 0.75 0.01 0.25 0.50 0.75
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
δ
2log*Λ10,2
Figu e 7: Dis ibu ion o 2 log Λ∗
10,1and 2 log Λ∗
10,2as a unc ion o δ,M, and censo ing
uppe limi U. The g ay + and Ma e co ec ed c i ical alues a he 0.01 and
0.05 signi icance le els, espec i ely.
Appendix C. Addi ional Plo s o Simula ion S udy
In his sec ion, we include addi ional plo s o he simula ion s udy in Sec ion 5.2. We show
esul s wi h BIC combined wi h leas -squa es es ima es, as well as esul s o addi ional
alues o ε ha we e no included in he main ex . The in e p e a ion o he plo s is he
same: h esholded me hods pe o m bes when |T|is small, and non- h esholded me hods
pe o m bes when |T|is la ge.
38
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
p = 9
|T| = 0
p = 9
|T| = 4
p = 9
|T| = 9
p = 6
|T| = 0
p = 6
|T| = 3
p = 6
|T| = 6
p = 2
|T| = 0
p = 2
|T| = 1
p = 2
|T| = 2
10 100 1000 10000 10 100 1000 10000 10 100 1000 10000
1e−11
1e−07
1e−03
1e−11
1e−07
1e−03
1e−11
1e−07
1e−03
n ( housands)
PMSE wi h Bayesian model a e aging
Me hod/Mechanism
G (no DP)
Lap Non− h esh (DP)
Wish Non− h esh (DP)
Lap Th esh (DP)
Wish Th esh (DP)
ε
0.5
0.9
Figu e 8: Simula ion s udy: Sample size (x-axis) agains log(PMSE) (y-axis) wi h BIC
p io .
39
Pe˜
na and Ba ien os
p = 9
|T| = 0
p = 9
|T| = 4
p = 9
|T| = 9
p = 6
|T| = 0
p = 6
|T| = 3
p = 6
|T| = 6
p = 2
|T| = 0
p = 2
|T| = 1
p = 2
|T| = 2
10 100 1000 10000 10 100 1000 10000 10 100 1000 10000
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
n ( housands)
Pos e io p obabili y ue model
Me hod/Mechanism
G (no DP)
Lap Non− h esh (DP)
Wish Non− h esh (DP)
Lap Th esh (DP)
Wish Th esh (DP)
ε
0.5
0.9
Figu e 9: Simula ion s udy: Sample size (x-axis) agains pos e io p obabili y o he ue
model (y-axis) wi h BIC p io .
40
Di e en ially p i a e me hods o managing model unce ain y in linea eg ession
Re e ences
John M Abowd. The US census bu eau adop s di e en ial p i acy. In P oceedings o he
24 h ACM SIGKDD In e na ional Con e ence on Knowledge Disco e y & Da a Mining,
pages 2867–2867, 2018.
Mil on Ab amowi z, I ene A S egun, e al. Handbook o ma hema ical unc ions, olume 55.
Do e New Yo k, 1964.
Hi o ugu Akaike. A new look a he s a is ical model iden i ica ion. IEEE T ansac ions on
Au oma ic Con ol, 19(6):716–723, 1974.
Gilad Ami ai and Je ome Rei e . Di e en ially p i a e pos e io summa ies o linea e-
g ession coe icien s. Jou nal o P i acy and Con iden iali y, 8(1), 2018.
Bo ja Balle and Yu-Xiang Wang. Imp o ing he Gaussian mechanism o di e en ial p i acy:
Analy ical calib a ion and op imal denoising. In In e na ional Con e ence on Machine
Lea ning, pages 394–403. PMLR, 2018.
And ´es F Ba ien os, Je ome P Rei e , Ashwin Machana ajjhala, and Yan Chen. Di e en-
ially p i a e signi icance es s o eg ession coe icien s. Jou nal o Compu a ional and
G aphical S a is ics, 28(2):440–453, 2019.
Ma ia J Baya i, James O Be ge , Anabel Fo e, Gonzalo Ga c´ıa-Dona o, e al. C i e ia o
Bayesian model choice wi h applica ion o a iable selec ion. The Annals o S a is ics,
40(3):1550–1577, 2012.
James O Be ge and Luis R Pe icchi. The in insic Bayes ac o o model selec ion and
p edic ion. Jou nal o he Ame ican S a is ical Associa ion, 91(433):109–122, 1996.
James O Be ge and Luis R Pe icchi. Objec i e Bayesian me hods o model selec ion:
In oduc ion and compa ison. Lec u e No es-Monog aph Se ies, pages 135–207, 2001.
James O Be ge , Luis R Pe icchi, and Julia A Va sha sky. Bayes ac o s and ma ginal
dis ibu ions in in a ian si ua ions. Sankhy¯a: The Indian Jou nal o S a is ics, Se ies
A, pages 307–321, 1998.
James O Be ge , B une o Liseo, and Robe L Wolpe . In eg a ed likelihood me hods o
elimina ing nuisance pa ame e s. S a is ical science, pages 1–22, 1999.
Ga e Be ns ein and Daniel R Sheldon. Di e en ially p i a e Bayesian linea eg ession.
Ad ances in Neu al In o ma ion P ocessing Sys ems, 32:525–535, 2019.
Pe e J Bickel, Eliza e a Le ina, e al. Co a iance egula iza ion by h esholding. The
Annals o S a is ics, 36(6):2577–2604, 2008.
Lucien Bi g´e. An al e na i e poin o iew on Lepski’s me hod. Lec u e No es-Monog aph
Se ies, pages 113–133, 2001.
41
