Search Mailing List Archives
[liberationtech] DuckDuckGo vs Startpage
mikeperry at torproject.org
Mon Jun 24 23:26:06 PDT 2013
> On 06/24/2013 10:00 PM, Mike Perry wrote:
> > IxQuick has so far successfully negotiated with Google against
> > outright banning us. Google sees a spike in IxQuick traffic every
> > time we increase StartPage's prominence in TBB, and this does not
> > go unnoticed by Google.
> > Unfortunately, Google's knee-jerk reaction to each increase so far
> > is to argue harder in favor of banning all Tor users from both
> > Startpage and Google, so we'll have to wait and see how this plays
> > out...
> > Backchannel like that (and direct-channel refusals to work with
> > Tor) really makes you wonder about Google's commitment to privacy
> > and the freedom of access to information.
> Very interesting. I don't know the backchannel relationships but I'd
> guess Google's decision to allow or not allow Tor users doesn't depend
> on the levels of traffic they get from StartPage from TBB front page.
Well, that's not exactly how it works directly, but the effect is the
same. I was simplifying the explanation for the purposes of brevity,
and because I was basically a 3rd party to this pressure who was not
present during the actual negotiation.
However, near as I can tell, the actual mechanism of the pressure is
both economic and service-level. Google isn't transparent about what it
pays for ad revenue and what it allows for API key volume, and they
simply pay less ad revenue and/or ban your API key if they don't like
your query flow for whatever reason. They also call you up and start
asking questions if your volume suddenly increases, and sometimes just
shut off your API key at random (and when they do this, StartPage has to
ban Tor users, which has happened each time we've featured them in a
more easily accessible way in TBB so far).
Google is also unwilling to work with us to deploy rate limiting
solutions, even if Tor were to develop them for them. I've tried
numerous times through multiple channels over the past 5 (five!) years
now to get some level of agreement to support various alternative and
less intrusive rate limiting mechanisms based on proof of work, blind
signatures, and other schemes instead of SMS and Captcha, so that Tor
could turn around and try to find a sponsor to build it, but the only
response we can get is "Abuse rate limiting is hard, and Google is the
best in the world at it! You can't mess with success!"
It is very frustrating, but I also feel like if we stop trying to use
any flavor of Google results entirely, we lose the ability to signal to
them how many people care about Tor.
> >> Just trying to rationally explain it.
> > I would not rationally use the hidden service version in lieu of
> > https by default.
> > As I alluded to through my questioning of the https backend link to
> > Bing, the transit path from Tor to DDG is not the weakest link in
> > an already-https search engine.
> Okay, so this seems to be the sticking point? Using the !g bang syntax
> they route Google requests through DDG (so you can search Google if
> you want, even though they don't seem to rely on Google for their own
> index). Is that reroute different than what Ixquick does? I don't
> know. For the index itself, I wasn't able to find anything on the
> technical connection between DDG and their index sources.
g! is just a redirect. There is no privacy there.
> Apparently the founder of DDG is interested in getting an external
> audit, so this might be the type of issue that could solve? He was
> looking for external audit recommendations as of two days ago (
> ). I'd ping him @yegg or yegg at alum.mit.edu with some recs.
Sure. I don't think this stuff is rocket science. There are probably
several people on this list that could help him figure out how to make
stuff end-to-end encrypted for front end and backend, excluding his
actual servers, and help him certify and promote that claim.
I am after a bigger monster, though.
> > Further, claims that the performance is the same or similar are
> > not rigorous.
> > Hidden service circuits require ~4X as many Tor router traversals
> > as normal Tor exit circuits to set up, and unlike normal Tor exit
> > circuits, they are often *not* prebuilt. Once they are set up, they
> > still require 2X as many Tor router traversals end-to-end as normal
> > circuits. You could easily circle the globe several times to issue
> > a single search query.
> > And all this is to use the Tor hidden service's 80bit-secure hash
> > instead of an https cert, along with all of the other issues with
> > Tor Hidden Services that have accumulated over the past decade due
> > to the lack of time for maintenance on Tor's part? I am not
> > convinced.
> This is good to know -- don't promote hidden service versions of
> websites (including DDG) when they have an https version, as hidden
> services are broken as of now.
Right. However, hidden services are still useful in narrow
circumstances, even as janky as they are. I think their most compelling
usecase is as fully internal TCP-style application endpoints, not as
authentication mechanisms for services that already exist on the
surveilled Internet, and use it for their communications.
In other words, once the query that you issue to a service such as a
search engine over a secure front end is leaking out the back end via
plaintext, NSA compromise, or datasharing deals, the end-to-end benefits
of using hidden services instead of https go out the window, especially
given the performance costs.
More information about the liberationtech