Scrapy: Run Using TOR and Multiple Agents Part 2

http://pkmishra.github.io/blog/2013/04/16/scrapy-run-using-tor-and-multiple-agents-part-2-ubuntu/

As discussed in last post this post is about running the same things on ubuntu. Again I am going to assume that you already have scrapy installed on your system. To install Tor

• Add tor repository to your ubuntu repository list (See official documentation for more detail)

1

echo "deb  http://deb.torproject.org/torproject.org quantal main" | sudo tee -a /etc/apt/sources.list

Here quantal is the release name for Ubuntu 12.10 release. If you are running any other version of ubuntu change it accordingly. + add gpg keys to sign the package

1
2

gpg --keyserver keys.gnupg.net --recv 886DDD89
gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add -

• Now referesh the package and install tor

1
2
3

apt-get update
apt-get install deb.torproject.org-keyring
apt-get install tor

• Install Polipo install using apt-get install polipo and change the configuration as described in last post. Everything else remains same i.e. by default polipo is using 8123 and Tor 9050 so you don’t have to modify anything.
• By default Tor and polipo will be installed as service. So after changing configuration you must restart polipo service

1
2

using sudo /etc/init.d/polipo restart
(you can use same comand to start/stop both the services)

• Now check whether tor is working fine or not by running

1

curl --proxy http://localhost:8123 https://check.torproject.org/ | grep "Congratulations. This browser is configured to use Tor"

You can see you will get this text displayed if everything is working. You are all set now. Hope this helps