Saturday, November 22, 2008

Setting up local Ubuntu & Debian mirror using apt-mirror

Prologue: Our institute was moving to it's new campus. For a central Govt. run research institute, the huge land requirements forced it to be located quite far off from the city. It is a remote place around 50km from Kolkata, WB, India. Leased line internet connectivity is there in the Institute, but there was a "rumour" that internet connectivity will not be provided in the student's hostel. There is a large number of Ubuntu and Debian GNU/Linux users among the students. Whether the rumour was true or not, I could not have taken any chance. I had a spare 60GB Seagate USB HDD. It was big enough to hold the main, universe, multiverse and restricted packages for Intrepid and the main, contrib and non-free packages for Lenny. In addition to that I also mirrored a few other extra stuffs. We had a about a month to go, before we left for the new campus, so I set out for finishing the job before that. Ofcourse the biggest hurdle was bandwidth. We had a 2Mbit/s ADSL link in the hostel. So I knew it was going to take a few days to finish the download.
There are several ways to make a offline copy of ubuntu or debian repo. I found apt-mirror most easiest to use. I have documented the steps here, just in case someone else finds it useful.
i) The first job is ofcourse to install apt-mirror, it is there in the repo, so this step is trivial
sudo apt-get install apt-mirror

ii) The next step is to tell apt-mirror, whats stuffs to mirror. The configuration files for apt-mirror is in /etc/apt/ubumirror.list and /etc/apt/debmirror.list
Here are the two files:

ubumirror.list
### ubumirror.list
############# config ##################
#
set base_path /home/sambit/stjet/mirror
#
# if you change the base path you must create the directories below with write privlages
#
# set mirror_path $base_path/mirror
# set skel_path $base_path/skel
# set var_path $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch
set nthreads 20
set _tilde 0
#
############# end config ##############
#Packages
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid main restricted universe multiverse
#Updates
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-updates main restricted universe multiverse
#Security updates
deb http://security.ubuntu.com/ubuntu/ intrepid-security universe main multiverse restricted
#Proposed
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-proposed restricted main multiverse universe
#Backports
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-backports restricted main multiverse universe
#Openoffice3
deb http://ppa.launchpad.net/openoffice-pkgs/ubuntu intrepid main
#Partner repo
deb http://archive.canonical.com/ubuntu intrepid partner
#Medibuntu packages
deb http://packages.medibuntu.org/ intrepid free non-free
clean http://ftp.twaren.net/Linux/Ubuntu/ubuntu/


debmirror.list

### debmirror.list
############# config ##################
#
set base_path /home/sambit/stjet/mirror/debmirror
#
# if you change the base path you must create the directories below with write privlages
#
# set mirror_path $base_path/mirror
# set skel_path $base_path/skel
# set var_path $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch
set nthreads 20
set _tilde 0
#
############# end config ##############
deb http://ftp.twaren.net/debian/ lenny main contrib non-free
deb http://ftp.twaren.net/debian/ lenny-proposed-updates main contrib non-free
clean http://ftp.twaren.net/debian


My 60GB SeagateHDD inside Transcend StoreJet USB enclosure was mounted at /home/sambit/stjet , so the base path was /home/sambit/stjet/mirror , which meant all the stuffs would be downloaded to the mirror directory inside the disk.
The deb lines tells apt-mirror what to mirror. I had selected main restricted universe multiverse for intrepid, as well all the updates including security, proposed and backports. A few additional repos were also selected. ftp.twaren.net is located in Taiwan, and from my experience I have seen it gives the best download speed in my location. Unfortuntely in India bandwidth is sitll not as easily available as it is in other countries. The official ubuntu mirror in India is at the Indian Institute of Technology, Madras, but the download speed from it is quite slow compared to the Taiwanese mirror.
The last line clean http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ and clean http://ftp.twaren.net/debian tells which directory to check for cleaning up disk space if possible.
iii) To start mirroring type

sudo apt-mirror /etc/apt/ubumirror.list

and
sudo apt-mirror /etc/apt/debmirror.list

It will show how many GBs have to be downloaded. In my case it was around 25GB for the ubuntu mirror and around 20GB for the Debian mirror. That would have required a couple of days with the 2Mbit/s connection. I used screen for the purpose. The advantage is that the virtual console can be detached after running the command so that the download goes on in the background. I could connect to the detached console any time later using screen -R, to view the progress.
iv) The packages are to be served over http. So a webserver is required for this. Use either the mighty Apache2 or the lighty Lighttpd. Apache2 was already installed in one of my old PIV. So I didn't have to do anything extra for this. All I need was to create symlinks in the /var/www/ubuntu and /var/www/debian directory, so that it points to the packages.

ln -s /home/sambit/stjet/mirror/mirror/ftp.twaren.net/Linux/Ubuntu/ubuntu /var/www/ubuntu/ubuntu
ln -s /home/sambit/stjet/mirror/mirror/packages.medibuntu.org /var/www/ubuntu/medibuntu
ln -s /home/sambit/stjet/mirror/mirror/ppa.launchpad.net/openoffice-pkgs/ubuntu /var/www/ubuntu/ubuoofficeppa
ln -s /home/sambit/stjet/mirror/mirror/archive.canonical.com/ubuntu /var/www/ubuntu/ubupartner
ln -s /home/sambit/stjet/mirror/mirror/security.ubuntu.com/ubuntu/ /var/www/ubuntu/ubusec
ln -s /home/sambit/stjet/mirror/debmirror/mirror/ftp.twaren.net/debian/ /var/www/debian

v)The hostname for the machine is mars.iiserk.net (don't try to access it, it is behind a firewall and not accessible from the internet). I now have my own Ubuntu Intrepid mirror at http://mars.iiserk.net/ubuntu
vi) The last step is to modify the /etc/apt/sources.list for the client machines, so that they pull down packages from mars.iiserk.net.
Here is my copy for my laptop running Ubuntu Intrepid:

##Local Mirror##
##Stuffs##
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid main restricted universe multiverse
deb http://mars.iiserk.net/ubuntu/ubupartner/ intrepid partner
deb http://mars.iiserk.net/ubuntu/ubuoofficeppa intrepid main
deb http://mars.iiserk.net/ubuntu/medibuntu intrepid free non-free
##Updates##
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid-updates main restricted universe multiverse
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid-proposed restricted main multiverse universe
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid-backports restricted main multiverse universe
deb http://mars.iiserk.net/ubuntu/ubusec/ intrepid-security main universe multiverse restricted
####################################################
##Internet##
##Sources##
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid main restricted universe multiverse
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-updates main restricted universe multiverse
deb-src http://archive.canonical.com/ubuntu intrepid partner
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-security main restricted
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-security universe
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-security multiverse
##Others##
#deb http://www.remastersys.klikit-linux.com/repository remastersys/

I didn't mirror the sources, so they were to be pulled directly from the internet, when required. The other packages would be fetched from the my local mirror.
It took me around 5 days to fetch 45GB of data.
To keep the mirror updated, I added to to lines to crontab

2 2 * * * /usr/bin/apt-mirror /etc/apt/debmirror.list >> /var/spool/apt-mirror/var/cron.log
55 2 * * * /usr/bin/apt-mirror /etc/apt/ubumirror.list >> /var/spool/apt-mirror/var/cron.log

These will check for updates every night at 0202 hrs for debian mirror and 0255hrs for ubuntu mirror.

No comments: