Skip to main content

Setting up local Ubuntu & Debian mirror using apt-mirror

Prologue: Our institute was moving to it's new campus. For a central Govt. run research institute, the huge land requirements forced it to be located quite far off from the city. It is a remote place around 50km from Kolkata, WB, India. Leased line internet connectivity is there in the Institute, but there was a "rumour" that internet connectivity will not be provided in the student's hostel. There is a large number of Ubuntu and Debian GNU/Linux users among the students. Whether the rumour was true or not, I could not have taken any chance. I had a spare 60GB Seagate USB HDD. It was big enough to hold the main, universe, multiverse and restricted packages for Intrepid and the main, contrib and non-free packages for Lenny. In addition to that I also mirrored a few other extra stuffs. We had a about a month to go, before we left for the new campus, so I set out for finishing the job before that. Ofcourse the biggest hurdle was bandwidth. We had a 2Mbit/s ADSL link in the hostel. So I knew it was going to take a few days to finish the download.
There are several ways to make a offline copy of ubuntu or debian repo. I found apt-mirror most easiest to use. I have documented the steps here, just in case someone else finds it useful.
i) The first job is ofcourse to install apt-mirror, it is there in the repo, so this step is trivial
sudo apt-get install apt-mirror

ii) The next step is to tell apt-mirror, whats stuffs to mirror. The configuration files for apt-mirror is in /etc/apt/ubumirror.list and /etc/apt/debmirror.list
Here are the two files:

ubumirror.list
### ubumirror.list
############# config ##################
#
set base_path    /home/sambit/stjet/mirror
#
# if you change the base path you must create the directories below with write privlages
#
# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  
set nthreads     20
set _tilde 0
#
############# end config ##############
#Packages
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid main restricted universe multiverse
#Updates
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-updates main restricted universe multiverse
#Security updates
deb http://security.ubuntu.com/ubuntu/ intrepid-security universe main multiverse restricted
#Proposed
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-proposed restricted main multiverse universe
#Backports
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-backports restricted main multiverse universe
#Openoffice3
deb http://ppa.launchpad.net/openoffice-pkgs/ubuntu intrepid main
#Partner repo
deb http://archive.canonical.com/ubuntu intrepid partner
#Medibuntu packages
deb http://packages.medibuntu.org/ intrepid free non-free
clean http://ftp.twaren.net/Linux/Ubuntu/ubuntu/


debmirror.list

### debmirror.list
############# config ##################
#
set base_path    /home/sambit/stjet/mirror/debmirror
#
# if you change the base path you must create the directories below with write privlages
#
# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  
set nthreads     20
set _tilde 0
#
############# end config ##############
deb http://ftp.twaren.net/debian/ lenny main contrib non-free
deb http://ftp.twaren.net/debian/ lenny-proposed-updates main contrib non-free
clean http://ftp.twaren.net/debian


My 60GB SeagateHDD inside Transcend StoreJet USB enclosure was mounted at /home/sambit/stjet , so the base path was /home/sambit/stjet/mirror , which meant all the stuffs would be downloaded to the mirror directory inside the disk.
The deb lines tells apt-mirror what to mirror. I had selected main restricted universe multiverse for intrepid, as well all the updates including security, proposed and backports. A few additional repos were also selected. ftp.twaren.net is located in Taiwan, and from my experience I have seen it gives the best download speed in my location. Unfortuntely in India bandwidth is sitll not as easily available as it is in other countries. The official ubuntu mirror in India is at the Indian Institute of Technology, Madras, but the download speed from it is quite slow compared to the Taiwanese mirror.
The last line clean http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ and clean http://ftp.twaren.net/debian tells which directory to check for cleaning up disk space if possible.
iii) To start mirroring type

sudo apt-mirror /etc/apt/ubumirror.list

and
sudo apt-mirror /etc/apt/debmirror.list

It will show how many GBs have to be downloaded. In my case it was around 25GB for the ubuntu mirror and around 20GB for the Debian mirror. That would have required a couple of days with the 2Mbit/s connection. I used screen for the purpose. The advantage is that the virtual console can be detached after running the command so that the download goes on in the background. I could connect to the detached console any time later using screen -R, to view the progress.
iv) The packages are to be served over http. So a webserver is required for this. Use either the mighty Apache2 or the lighty Lighttpd. Apache2 was already installed in one of my old PIV. So I didn't have to do anything extra for this. All I need was to create symlinks in the /var/www/ubuntu and /var/www/debian directory, so that it points to the packages.

ln -s /home/sambit/stjet/mirror/mirror/ftp.twaren.net/Linux/Ubuntu/ubuntu /var/www/ubuntu/ubuntu
ln -s /home/sambit/stjet/mirror/mirror/packages.medibuntu.org /var/www/ubuntu/medibuntu
ln -s /home/sambit/stjet/mirror/mirror/ppa.launchpad.net/openoffice-pkgs/ubuntu /var/www/ubuntu/ubuoofficeppa
ln -s /home/sambit/stjet/mirror/mirror/archive.canonical.com/ubuntu /var/www/ubuntu/ubupartner
ln -s /home/sambit/stjet/mirror/mirror/security.ubuntu.com/ubuntu/ /var/www/ubuntu/ubusec
ln -s /home/sambit/stjet/mirror/debmirror/mirror/ftp.twaren.net/debian/ /var/www/debian

v)The hostname for the machine is mars.iiserk.net (don't try to access it, it is behind a firewall and not accessible from the internet). I now have my own Ubuntu Intrepid mirror at http://mars.iiserk.net/ubuntu
vi) The last step is to modify the /etc/apt/sources.list for the client machines, so that they pull down packages from mars.iiserk.net.
Here is my copy for my laptop running Ubuntu Intrepid:

##Local Mirror##
##Stuffs##
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid main restricted universe multiverse
deb http://mars.iiserk.net/ubuntu/ubupartner/ intrepid partner
deb http://mars.iiserk.net/ubuntu/ubuoofficeppa intrepid main
deb http://mars.iiserk.net/ubuntu/medibuntu intrepid free non-free
##Updates##
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid-updates main restricted universe multiverse
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid-proposed restricted main multiverse universe
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid-backports restricted main multiverse universe
deb http://mars.iiserk.net/ubuntu/ubusec/ intrepid-security main universe multiverse restricted
####################################################
##Internet##
##Sources##
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid main restricted universe multiverse
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-updates main restricted universe multiverse
deb-src http://archive.canonical.com/ubuntu intrepid partner
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-security main restricted
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-security universe
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-security multiverse
##Others##
#deb http://www.remastersys.klikit-linux.com/repository remastersys/

I didn't mirror the sources, so they were to be pulled directly from the internet, when required. The other packages would be fetched from the my local mirror.
It took me around 5 days to fetch 45GB of data.
To keep the mirror updated, I added to to lines to crontab

2 2     * * *   /usr/bin/apt-mirror /etc/apt/debmirror.list >> /var/spool/apt-mirror/var/cron.log
55 2    * * *   /usr/bin/apt-mirror /etc/apt/ubumirror.list >> /var/spool/apt-mirror/var/cron.log

These will check for updates every night at 0202 hrs for debian mirror and 0255hrs for ubuntu mirror.
Labels:

Comments

Post a Comment

Popular posts from this blog

Making inactve USB Hard Disk spin down automatically in Linux.

I have a 400GB Seagate IDE HDD connected to Mars, our hostel's file-server using an USB enclosure. The USB enclosure is a cheap "Made in China" product. Consequently it has some special "features". One such notable "feature" is that the disk is kept spinning by the controller even if there has been no disk I/O for a long time. I have three other USB disks connected to the same machine, a 1TB Seagate FreeAgent Desk External Drive, a 500GB Maxtor Basics External Drive and a 2.5" 60GB Fujitsu SATA Disk inside a Transcend USB enclosure. All of these spin down themselves if there has been no I/O for sometime. Keeping the hard disk spinning unnecessarily for ever, not only wastes power but also overheats the drive, thereby reducing its life. I tried noflushd, which is supposed to force idle hard disks to spin down, but found it to be of no help. USB enclosure generally work by performing an SCSI emulation over USB. sdparm is an utility which can be use
6 comments
Read more

Force an application to use VPN, using iptables in Linux

Enforcing an application, for example a torrent client like Transmission , to always use the VPN interface or any particular network interface for that matter, is trivially simple using iptables on Debian, Ubuntu or any other GNU/Linux distro. Personally, I am running Debian Sid on the Raspberry Pi . Occasionally I use it for downloading files ( legal stuff, seriously, believe me :D  ) using Transmission Bittorrent client over a VPN connection. Sometimes it happens that the VPN connection fails and doesn't reconnect for whatever reason and Transmission continues pulling stuff directly over my internet connection, which I would like to avoid. Fortunately it is very straightforward to enforce rules based on application owner UID. Transmission runs under the owner debian-transmission in Debian (use htop to check this) and the following two lines of iptables ensures that any process with owner having UID, debian-transmission , will not use any other network interface apart from the
9 comments
Read more

Rendering LaTeX in Blogger.

Rendering LaTeX in Blogger is pretty easy thanks to the JavaScript LaTeX equation render engine from http://www.yourequations.com . To enable LaTeX rendering go to the Blogger Dashboard --> Layout --> Edit HTML . Then add the line <script> type="text/javascript" src="http://tex.yourequations.com/"></script> <script type="text/javascript" src="http://tex.yourequations.com/"></script> just before <body/> . Half the job is now done. Then, for example to render: \int_{0}^{\pi}\frac{x^{4}\left(1-x\right)^{4}}{1+x^{2}}dx =\frac{22}{7}-\pi Use the code: <pre lang="eq.latex"> \int_{0}^{1}\frac{x^{4}\left(1-x\right)^{4}}{1+x^{2}}dx =\frac{22}{7}-\pi </pre> The LaTeX code will now be displayed as: \int_{0}^{1}\frac{x^{4}\left(1-x\right)^{4}}{1+x^{2}}dx=\frac{22}{7}-\pi Ofcourse Javascript needs to be enabled in the Browser for the renderer to work.
9 comments
Read more