Skip to main content

Setting up local Ubuntu & Debian mirror using apt-mirror

Prologue: Our institute was moving to it's new campus. For a central Govt. run research institute, the huge land requirements forced it to be located quite far off from the city. It is a remote place around 50km from Kolkata, WB, India. Leased line internet connectivity is there in the Institute, but there was a "rumour" that internet connectivity will not be provided in the student's hostel. There is a large number of Ubuntu and Debian GNU/Linux users among the students. Whether the rumour was true or not, I could not have taken any chance. I had a spare 60GB Seagate USB HDD. It was big enough to hold the main, universe, multiverse and restricted packages for Intrepid and the main, contrib and non-free packages for Lenny. In addition to that I also mirrored a few other extra stuffs. We had a about a month to go, before we left for the new campus, so I set out for finishing the job before that. Ofcourse the biggest hurdle was bandwidth. We had a 2Mbit/s ADSL link in the hostel. So I knew it was going to take a few days to finish the download.
There are several ways to make a offline copy of ubuntu or debian repo. I found apt-mirror most easiest to use. I have documented the steps here, just in case someone else finds it useful.
i) The first job is ofcourse to install apt-mirror, it is there in the repo, so this step is trivial
sudo apt-get install apt-mirror

ii) The next step is to tell apt-mirror, whats stuffs to mirror. The configuration files for apt-mirror is in /etc/apt/ubumirror.list and /etc/apt/debmirror.list
Here are the two files:

ubumirror.list
### ubumirror.list
############# config ##################
#
set base_path /home/sambit/stjet/mirror
#
# if you change the base path you must create the directories below with write privlages
#
# set mirror_path $base_path/mirror
# set skel_path $base_path/skel
# set var_path $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch
set nthreads 20
set _tilde 0
#
############# end config ##############
#Packages
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid main restricted universe multiverse
#Updates
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-updates main restricted universe multiverse
#Security updates
deb http://security.ubuntu.com/ubuntu/ intrepid-security universe main multiverse restricted
#Proposed
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-proposed restricted main multiverse universe
#Backports
deb http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-backports restricted main multiverse universe
#Openoffice3
deb http://ppa.launchpad.net/openoffice-pkgs/ubuntu intrepid main
#Partner repo
deb http://archive.canonical.com/ubuntu intrepid partner
#Medibuntu packages
deb http://packages.medibuntu.org/ intrepid free non-free
clean http://ftp.twaren.net/Linux/Ubuntu/ubuntu/


debmirror.list

### debmirror.list
############# config ##################
#
set base_path /home/sambit/stjet/mirror/debmirror
#
# if you change the base path you must create the directories below with write privlages
#
# set mirror_path $base_path/mirror
# set skel_path $base_path/skel
# set var_path $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch
set nthreads 20
set _tilde 0
#
############# end config ##############
deb http://ftp.twaren.net/debian/ lenny main contrib non-free
deb http://ftp.twaren.net/debian/ lenny-proposed-updates main contrib non-free
clean http://ftp.twaren.net/debian


My 60GB SeagateHDD inside Transcend StoreJet USB enclosure was mounted at /home/sambit/stjet , so the base path was /home/sambit/stjet/mirror , which meant all the stuffs would be downloaded to the mirror directory inside the disk.
The deb lines tells apt-mirror what to mirror. I had selected main restricted universe multiverse for intrepid, as well all the updates including security, proposed and backports. A few additional repos were also selected. ftp.twaren.net is located in Taiwan, and from my experience I have seen it gives the best download speed in my location. Unfortuntely in India bandwidth is sitll not as easily available as it is in other countries. The official ubuntu mirror in India is at the Indian Institute of Technology, Madras, but the download speed from it is quite slow compared to the Taiwanese mirror.
The last line clean http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ and clean http://ftp.twaren.net/debian tells which directory to check for cleaning up disk space if possible.
iii) To start mirroring type

sudo apt-mirror /etc/apt/ubumirror.list

and
sudo apt-mirror /etc/apt/debmirror.list

It will show how many GBs have to be downloaded. In my case it was around 25GB for the ubuntu mirror and around 20GB for the Debian mirror. That would have required a couple of days with the 2Mbit/s connection. I used screen for the purpose. The advantage is that the virtual console can be detached after running the command so that the download goes on in the background. I could connect to the detached console any time later using screen -R, to view the progress.
iv) The packages are to be served over http. So a webserver is required for this. Use either the mighty Apache2 or the lighty Lighttpd. Apache2 was already installed in one of my old PIV. So I didn't have to do anything extra for this. All I need was to create symlinks in the /var/www/ubuntu and /var/www/debian directory, so that it points to the packages.

ln -s /home/sambit/stjet/mirror/mirror/ftp.twaren.net/Linux/Ubuntu/ubuntu /var/www/ubuntu/ubuntu
ln -s /home/sambit/stjet/mirror/mirror/packages.medibuntu.org /var/www/ubuntu/medibuntu
ln -s /home/sambit/stjet/mirror/mirror/ppa.launchpad.net/openoffice-pkgs/ubuntu /var/www/ubuntu/ubuoofficeppa
ln -s /home/sambit/stjet/mirror/mirror/archive.canonical.com/ubuntu /var/www/ubuntu/ubupartner
ln -s /home/sambit/stjet/mirror/mirror/security.ubuntu.com/ubuntu/ /var/www/ubuntu/ubusec
ln -s /home/sambit/stjet/mirror/debmirror/mirror/ftp.twaren.net/debian/ /var/www/debian

v)The hostname for the machine is mars.iiserk.net (don't try to access it, it is behind a firewall and not accessible from the internet). I now have my own Ubuntu Intrepid mirror at http://mars.iiserk.net/ubuntu
vi) The last step is to modify the /etc/apt/sources.list for the client machines, so that they pull down packages from mars.iiserk.net.
Here is my copy for my laptop running Ubuntu Intrepid:

##Local Mirror##
##Stuffs##
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid main restricted universe multiverse
deb http://mars.iiserk.net/ubuntu/ubupartner/ intrepid partner
deb http://mars.iiserk.net/ubuntu/ubuoofficeppa intrepid main
deb http://mars.iiserk.net/ubuntu/medibuntu intrepid free non-free
##Updates##
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid-updates main restricted universe multiverse
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid-proposed restricted main multiverse universe
deb http://mars.iiserk.net/ubuntu/ubuntu/ intrepid-backports restricted main multiverse universe
deb http://mars.iiserk.net/ubuntu/ubusec/ intrepid-security main universe multiverse restricted
####################################################
##Internet##
##Sources##
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid main restricted universe multiverse
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-updates main restricted universe multiverse
deb-src http://archive.canonical.com/ubuntu intrepid partner
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-security main restricted
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-security universe
deb-src http://ftp.twaren.net/Linux/Ubuntu/ubuntu/ intrepid-security multiverse
##Others##
#deb http://www.remastersys.klikit-linux.com/repository remastersys/

I didn't mirror the sources, so they were to be pulled directly from the internet, when required. The other packages would be fetched from the my local mirror.
It took me around 5 days to fetch 45GB of data.
To keep the mirror updated, I added to to lines to crontab

2 2 * * * /usr/bin/apt-mirror /etc/apt/debmirror.list >> /var/spool/apt-mirror/var/cron.log
55 2 * * * /usr/bin/apt-mirror /etc/apt/ubumirror.list >> /var/spool/apt-mirror/var/cron.log

These will check for updates every night at 0202 hrs for debian mirror and 0255hrs for ubuntu mirror.

Comments

Popular posts from this blog

Force an application to use VPN, using iptables in Linux

Enforcing an application, for example a torrent client like Transmission, to always use the VPN interface or any particular network interface for that matter, is trivially simple using iptables on Debian, Ubuntu or any other GNU/Linux distro.
Personally, I am running Debian Sid on the Raspberry Pi. Occasionally I use it for downloading files ( legal stuff, seriously, believe me :D  ) using Transmission Bittorrent client over a VPN connection. Sometimes it happens that the VPN connection fails and doesn't reconnect for whatever reason and Transmission continues pulling stuff directly over my internet connection, which I would like to avoid. Fortunately it is very straightforward to enforce rules based on application owner UID. Transmission runs under the owner debian-transmission in Debian (use htop to check this) and the following two lines of iptables ensures that any process with owner having UID, debian-transmission, will not use any other network interface apart from the Open…

Some more fun with SSH port forwarding and socks proxy

Few days ago I made the following post:

Prologue: Our Institute has several nice Dual Core Machines deployed for the students. Unfortunately the machines are behind a NAT with no port forwarded for external SSH access. Student's hostel is a bit far off from the computational centre. As such if someone felt the need of accessing the machines during non-office hours, it was a wee bit difficult. The sysadmin would not have agreed to forward any ports. Something had to be done....

SSH has a very useful feature - Remote and Local Port Forwarding. We have an old rickety PIII running Ubuntu 8.04.1 in the Hostel, it is connected to the net and is accessible via SSH from the internet. Using a tiny little shell script running on one of the machines in the Institute, I managed to make the old PIII an intermediate gateway for gaining SSH access to the Institute's machines from anywhere in the internet. The script is of few lines, but nevertheless powerful enough to serve our purpose.

#!/bin/…

Raspberry Pi -- Installing Samba (Windows Share) File server

Having successfully run Debian Wheezy on my Raspberry Pi, I went forward with my initial idea of setting up a low cost power efficient file server for accessing my external hard disks from my Windows7 desktop, HP-Mini running Ubuntu and Mac Mini running OS X Lion (yeah I do like bragging about my machines :D ).

This turned out to be pretty straight forward.

As expected, the external Seagate USB disk immediately got recognized and appeared as /dev/sda
[ 579.948350] usb 1-1.2: New USB device found, idVendor=0bc2, idProduct=3001 [ 579.948384] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 579.948405] usb 1-1.2: Product: FreeAgent [ 579.948421] usb 1-1.2: Manufacturer: Seagate [ 579.948447] usb 1-1.2: SerialNumber: 2GEX323R [ 579.967638] scsi0 : usb-storage 1-1.2:1.0 [ 580.970520] scsi 0:0:0:0: Direct-Access Seagate FreeAgent 102D PQ: 0 ANSI: 4 [ 589.142942] sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 589.144669] sd…