Posted by: Odzangba | December 24, 2007

Use Apt-mirror to Create Your Own Ubuntu Mirror

Setting up your own Ubuntu mirror is quite easy thanks to a nifty little tool called apt-mirror. And it’s very rewarding too… super fast installations and the convenience of not needing an internet connection when you want to install most applications. We’ll need a few things to get the job done though – apt-mirror, a high speed internet connection and some 45GB of disk space. It’s more than likely that you won’t need all of it – my mirrors (binaries only) take up about 22GB – but we’d rather not be caught with too little space. I describe how to mirror the Ubuntu 7.10 repositories here in this guide but it can be applied to other versions of Ubuntu… in fact, the instructions here will work with any debian based distribution.

1) Okay, to start things off, let’s tap into the apt-mirror repository. Back up your sources.list file like so:

sudo cp /etc/apt/sources.list /etc/apt/sources.list.old

This way, if you mess things up badly you can restore from the back up and start all over.🙂

2) Edit the sources.list file like so:

gksudo gedit /etc/apt/sources.list

3) And insert the following on a new line:

deb http://apt-mirror.sourceforge.net/ apt-mirror

4) Let’s update the package index

sudo aptitude update

5) And install apt-mirror

sudo aptitude install apt-mirror

6) Apt-mirror is now nicely set up so let’s tell it what servers to mirror:

sudo cp /etc/apt/mirror.list /etc/apt/mirror.list.old
gksudo gedit /etc/apt/mirror.list

The default configuration will put your downloaded files in the /var/spool/apt-mirror folder but if you’re like me, you probably have a huge partition where you store all kinds of stuff and you’ll want to put the mirror(s) on that partition. The third line (where it says set base_path) is where you set your default download directory (mine is /media/silo/repo). Here’s how my mirror.list file looks like:

############# config ##################
#
set base_path /media/silo/repo
#
# if you change the base path you must create the directories below with write privlages
#
# set mirror_path $base_path/mirror
# set skel_path $base_path/skel
# set var_path $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch <running host architecture>
set nthreads 20
set tilde 0
#
############# end config ##############

deb http://archive.ubuntu.com/ubuntu gutsy main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu gutsy-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu gutsy-backports main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu gutsy-security main restricted universe multiverse

clean http://archive.ubuntu.com/ubuntu

I’m mirroring only binary packages but if you want source packages, you should insert the appropriate lines. They usually take the form:

deb-src http://gh.archive.ubuntu.com/ubuntu/ feisty main restricted

7) Now, we need to run apt-mirror for the first time. This is going to take quite a while so be prepared to wait for several hours… seriously, *several* hours.🙂 Don’t worry if you cannot download all the files in one go, apt-mirror can resume downloads (kill it with the Ctrl+C key combination and re-run it when you want to continue). Okay, run apt-mirror like so:

sudo apt-mirror /etc/apt/mirror.list

Right, it took a week but we finally finished the download.😀 Apt-mirror will tell you how to clean up unnecessary files when it exits… follow those instructions to reclaim some disk space. There are several ways to use your new mirror but we’ll look at two of the most common. The first involves just telling apt the physical location of the mirror on your hard disk and the other involves passing the mirror through a web server – Apache in this case. So let’s get to it.

Method One

I) Edit your sources.list file like so:

gksudo gedit /etc/apt/sources.list

II) And insert the following on a new line:

deb file:<path_to_mirror> gutsy main restricted universe multiverse

Replace <path_to_mirror> with the path to the downloaded files – /media/silo/repo/archive.ubuntu.com/ubuntu, in my case. You have to repeat this for all the other mirrors (gutsy-updates, gutsy-backports, gutsy-security, et cetera). In the end, the entries for your local mirror in /etc/apt/sources.list will resemble this:

deb file:/media/silo/repo/mirror/archive.ubuntu.com/ubuntu gutsy main restricted universe multiverse
deb file:/media/silo/repo/mirror/archive.ubuntu.com/ubuntu gutsy-updates main restricted universe multiverse
deb file:/media/silo/repo/mirror/archive.ubuntu.com/ubuntu gutsy-security main restricted universe multiverse
deb file:/media/silo/repo/mirror/archive.ubuntu.com/ubuntu gutsy-backports main restricted universe multiverse

As long as the entries here reflect the actual folder structure in your repository, nothing should break. Incidentally, you will want to comment out (place # in front of) the lines that point apt to Canonical’s online servers.

III) Update your package index like so:

sudo aptitude update

You can now start installing stuff. Simple, huh?🙂

Method Two

This method is more useful if you want access to the mirror from other machines on your network. In this example, let’s say the machine with the downloaded files is called ithilien on the middle-earth.com network. There’s another machine – gondor – on the same network. I’m going to assume Apache is installed on ithilien.middle-earth.com. Hey, if you haven’t installed Apache, you have all the packages from the Ubuntu servers on your hard disk… follow Method One and install Apache like so:

sudo aptitude install apache2

Okay let’s roll.🙂

i) We need to tell the Apache server on ithilien.middle-earth.com where to find the downloaded files:

ln -s /media/silo/repo/mirror/archive.ubuntu.com/ubuntu /var/www/ubuntu

If you’ve not tinkered with Apache’s configuration files, /var/www is usually the default virtual host.

ii) Right, now let’s make sure gondor.middle-earth.com can see our beautifully set up mirror. Edit the sources.list file on gondor.middle-earth.com like so:

gksudo gedit /etc/apt/sources.list

And insert the following on a new line:

deb http://ithilien.middle-earth.com/ubuntu gutsy main restricted universe multiverse

Alternatively, I could use ithilien’s IP address, in which case the above line would look like so:

deb http://192.168.3.144/ubuntu gutsy main restricted universe multiverse

Create similar lines for the other mirrors (gutsy-updates, gutsy-security, gutsy-backports, et cetera) and remember to comment out (place # in front of) the lines that point apt to Canonical’s online servers.

iii) Let’s update gondor’s package index:

sudo aptitude update

That’s it! Install as much as you can use.😀


Responses

  1. Nice methods.

  2. Thanks, ituner.🙂

  3. Having a strong windows software engineering background, I want to try the Linux world, specifically ubuntu on my laptop. Have you set up any ubuntu mirrors yourself? Also any tips on pitfalls to avoid whilst using ubuntu would be helpful

  4. Have you set up any ubuntu mirrors yourself? ????? WHAT ???? Ummm.. This is a tutorial for setting up a mirror. James, if you want to try the “Linux World” this is not the place to start. Start with a live cd of some sort, try Ubuntu, try Knoppix, try anything you can get your hands on. Attempt to do simple tasks one at a time. Establish a network connection, ping someone, connect to an FTP site, ssh to another linux box, (all in cli of course). Take the Ubuntu challenge. Write down every task you depend on your PC at home on. Install Ubuntu and start finding packages that do what you have listed. Google is your friend. Write those packages down. Once you have checked off everything on the windows list, dump your Ubuntu box and reinstall. Now install all the packages you wrote down and say goodby to your Windows box. you may never go back.

  5. Hi James, it’s nice to see you’re interested in linux. Regarding your initial question the tutorial above was written after I set up my first mirror so it works.🙂 I think you should check out https://help.ubuntu.com for starters… the tutorials there are generally short and you’ll avoid most of the initial hassles with stuff like codecs with their help. I know another great site – http://www.osliving.com – they maintain an excellent list of cool linux apps that you should look at.

    Happy hunting.🙂

  6. What I really wanted to know was whether you’ve set up any PUBLIC mirrors (requires a lot of bandwidth, anyway). I am running Ubuntu 7.10 (gutsy gibbon). I’m interested in AJAX development on ubuntu, and I want to install a LAMP configuration for this. It seems the apt installation relies on internet connectivity. Since I’m not always online, I would like to get the required packages for offline installation; but I been able to obtain these.

    I wonder whether there is a customized ubuntu distro that contains a certified LAMP configuration.
    If not I think that is something I might want to create later.

    NB: You might want to check out my group (by clicking on my name), so that we can discuss this off the blog.

  7. Prempeh, Ubuntu has a a DVD distribution that contains a lot more packages. However there is no link on the ubuntu website for this download. I suggest you search the site with google. You can also download and try ‘ubuntu ultimate’. All the best.

  8. Great!.. Did the mirror for Hardy, bups.. though you said it would take ages, I did not heed that. Still with my BSNL Dataone, and working only on night (2 am to 8 am), I barely completed this today moring..

    Can I pack off the mirror which I am using now to say some usb hd, and take it to my office? do I need to take some other files than that is in the mirror folder?

  9. Fantastic article mate, very clear and friendly as usual!
    What I am interested in is once you have this set up how do you update your mirror. If you just run apt-mirror does this do the whole download again or only update the changes?

  10. Baggers, running apt-mirror only grabs updates… it doesn’t do the entire download all over.

  11. Cheers man! exactly what I hoped. Will be giving this a go soon.

  12. Hi Odzangba

    Thanks for your help and quick reply.
    My question is 1) Should I go ahead and just do the method 2 or setup the mirror and then go for method 2. Bcoz all I want is to use the debs that I have in my /var/cache. I just can’t think of downloading the whole mirror, I have to pay a lot to download that.

    Awaiting your reply

  13. Hi koolneze, do the stuff in https://odzangba.wordpress.com/2006/10/13/how-to-build-local-apt-repositories/ to build a local repo using the packages currently in your /var/cache/apt/archives folder and then use method 2 to set one of the boxes up for the others to download from.

  14. Hi Odzangba
    First of all thanks for your help and quick reply.

    Could you explain the command
    ln -s /media/silo/repo/mirror/archive.ubuntu.com/ubuntu /var/www/ubuntu

    My situation is as follows.
    1)debs are in /var/cache/apt/archives
    I created the folder and copied all the debs into this.
    /home/ubuntu8/repo
    added the following line in sources.list
    deb file:/home/ubuntu8/repo/ /

    sudo dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz

    I get the following output

    ! Package passwd (filename ./passwd_1%3a4.1.1-1ubuntu1_i386.deb) is repeat;
    ignored that one and using data from ./passwd_1%3a4.1.1-1ubuntu1.2_i386.deb !
    ! Package bind9 (filename ./bind9_1%3a9.5.0.dfsg.P2-1ubuntu3_i386.deb) is repeat;
    ignored that one and using data from ./bind9_1%3a9.5.0.dfsg.P2-1ubuntu3.1_i386.deb !

    many lines like above and in the end
    Wrote 603 entries to output Packages file.
    root@ubuntu8:/home/ubuntu8/repo#

  15. Koolneze, the ln -s command creates a soft link… sort of like a shortcut (not quite but sort of). The thing is, the contents of /var/www can be accessed over your web server. So instead of copying the packages to /var/www, we create a “shortcut” to /var/www that tells the web server that the packages are in another folder… in this case, /home/ubuntu8/repo.
    So create the soft link:

    sudo ln -s /home/ubuntu8/repo /var/www/repo

    In order for the other boxes on your network to download packages from the box with the web server (and the packages) they need a line similar to:

    deb http://192.168.0.3/repo/ /

    in their sources.list file (I’m assuming the ip address of the box with the packages is 192.168.0.3). Now you can do:

    sudo aptitude update

    and start installing packages.🙂 Hope this helps.

  16. […] to make it necessary, but it would be an entertaining exercise.  Even found a basic HOW-TO or two.  However I have heard of potential issues: it can take weeks to fully mirror several […]

  17. Thanks man! I didn’t you could manually set ubuntu mirror. Great Share

  18. I have started to download ubuntu mirror, it is 35 Gb, so i have to wait a lot.
    Which command i can use to look how much is downladesd?

  19. My programm can also solve this problem)
    Screen: http://mozgoed-mgoy.narod.ru/ubuntu/screen.jpg
    Link: http://mozgoed-mgoy.narod.ru/ubuntu/ubuntu-repository.zip
    It’s portable and cross-platform.
    English version available.

  20. Thanks for your personal marvelous posting! I truly enjoyed reading it, you are a great author.

    I will make sure to bookmark your blog and will eventually
    come back in the foreseeable future. I want to encourage yourself to continue your great posts,
    have a nice holiday weekend!

  21. Hi there, this weekend is fastidious designed for me, as this occasion i am reading
    this enormous educational post here at my house.

  22. Thanks for finally writing about >Use Apt-mirror to Create Your Own Ubuntu
    Mirror | Odzangba Kafui Dake <Liked it!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: