Linux.com

Home Linux Community Community Blogs

Community Blogs



Creating an EBS backed Debian Squeeze Linux AMI for Amazon EC2

Following on from my post on how to create an S3 backed custom AMI for AWS EC2, I thought I would jot down the steps to go through if you wanted to make use of persistent disk storage by using Amazon’s Elastic Block Storage (EBS).

In my recipe I decided to use as a base AMI, the original S3 image that I rolled in my previous post:

After starting up an AWS instance using the S3 custom AMI I created, I went through the following process. – There are better ways to approach this, but this is a good starting point for anyone that is starting to experiment with the AWS platform.

Inside the AWS Dashboard create yourself an EBS volume. – This will be the one that will be used to build your future EBS machine image

Make a note of the block device id that the Dashboard assigns to your EBS volume once it has finished the provisioning process, you will need this in order to mount the EBS volume inside the S3 Instance that you have already booted up (You can find this under ‘Attachment Information’ in the Dashboard). It will be on here that you will run debootstrap,etc to build the EBS image.

Once the EBS volume is available you will see an entry like this in the dash:

Next, click the ‘Attach’ button in the AWS Dashboard as follows:

Upon clicking the Attach button you will be asked which Instance you want to attach the EBS volume to. – You need to select the Instance ID of your S3 backed AMI you just started.

Once everything is attached you will see an entry like:i-aws-instance-id:/dev/sdX. ie: i-a62bc123:/dev/sdf (attached). Notice that the ‘available‘ changes to ‘In Use‘ once you complete the attachment process. ( the ‘sdf’ is the block device id you will need to note down for the next step.)

Now on your running S3 AMI login and execute the following command as root to build an ext3 filesystem on the now attached EBS volume

 

mkfs.ext3 -F /dev/sdX

remember to set your TERM id to something valid before starting the copy_into_chroot, otherwise the LOCALE will fail

 

export TERM=vt220

You need to grab the latest compatible kernel sources from S3 as well. – Which ones you need will depend on if you are rolling a 32-bit or 64-bit AMI.

You can get them by pointing your browser to http://s3.amazonaws.com/ec2-downloads/ (or using curl from your instance) and grep’ing for the required Xenu modules,i.e:ec2-modules-2.6.21.7-2.fc8xen-ec2-v1.0-x86_64.tar, if you are building the 64-bit version.

Next run mkdir /EBSmountpoint

Next run debootstrap to pull down a netinstall of Debian Squeeze into the mountpoint we just created called /EBSmountpoint

debootstrap –arch [i386|amd64] squeeze /EBSmountpoint http://ftp.debian.org

Once this has finished, create a directory to unzip your kernel sources into:

mkdir 64bitmodules

then untar the downloaded source in that directory

cd 64bitmodules

tar xvf ec2-modules-2.6.21.7-2.fc8xen-ec2-v1.0-x86_64.tar

this should create a lib directory inside 64bitmodules, which you can then copy intolib/modules inside your /EBSmountpoint directory

cp -r 64bitmodules/* /EBSmountpoint/lib/modules

Next copy the two ec2 scripts. You’ll recognise these next few steps from my last post on Custom AMI’s, the copy_into_chroot.sh, shellscript contains additional configuration parameters that I have passed to the Debian image before it gets converted into an Amazon Machine Image.

 

cp /etc/init.d/ec2-get-credentials /EBSmountpoint/etc/init.d/

cp /etc/init.d/ec2-ssh-host-key-gen /EBSmountpoint/etc/init.d/

 

 

cp copy_into_chroot.sh /EBSmountpoint/

 

chroot /EBSmountpoint /bin/bash

mount -t proc none /proc

mount -t devpts none /dev/pts

 

 

execute copy_into_chroot.sh from inside your chroot environment.

type ‘exit‘ to leave your chroot

On your S3 Instance umount  your EBS filesystem, as your almost ready to fire it up!!

create snapshot of EBS (make note of the EBS snap id)

Next register your EBS backed image, specifying the id of the snapshot you just took (snap-1ab23c45).

ec2-register –private-key=/ec2_creds/pk-.pem –cert=/ec2_creds/cert-.pem –region=EU-WEST-1 -a x86_64 -b /dev/sda1=snap-1ab23c45:20:false -d “Matts EBS backed Squeeze test” -n “SqueezeEBS” –kernel=”aki-780d250c” –ramdisk=”ari-7f0d250b” (the 20 after the snap id is the volume size, so it must match your AMI image size).

This will send you back an ami-id to use when starting an instance, this is done in the same way as an Instance-store AMI, although you will notice that instead of ‘instance-store‘ it will now display ‘ebs‘.

What you will notice starting and stoping this Instance is that any changes that you make will be preserved and available on start, which might suit some peoples requirements.

Hope someone finds these pointers useful.

Matt Palmer  - 6th June 2011

 

 

openSUSE Weekly News 178 is out!

We are pleased to announce our new openSUSE Weekly News 178.

In this Issue:

  • First step towards 12.1 with Milestone 1
  • Andreas Jaeger: Factory Progress 2011-06-03
  • Sebastian Kügler: kollaboration in at platform 11
  • Linus Torvalds: Linux 3.0-rc1
  • and many more ...

You can download it there:

We hope you enjoy the reading :-)

If you want to help us collecting interesting articles for the openSUSE Weekly News, so you can all your stuff into our new ietherpad: http://os-news.ietherpad.com/2.

Found Bugs? Please place it in our Bugtracker: http://developer.berlios.de/bugs/?group_id=12095

Features, Ideas and Improvements can placed in our Featuretracker: http://developer.berlios.de/feature/?group_id=12095

Older content can be found there.

Flattr this

 

64 bit sytem versus Mint Linux

I built a new system  on May 6th,2011. MSI 760GM-E51 mobo, 600 watt power supply, AMD 640, quad core processor, 16 gigs DDR3 ram, and 2- 500 hundred gig HD's. I attempted to install the 64 bit version of Linux Mint 11 on it. It did not give me an option to install to sdb, only showed sda. I had a copy of Ubuntu Studio handy, so I loaded it up and tried to install it. It showed sdb, so I installed it, no problem. Not really a fan of Ubuntu, so I resized the partition and tried Linux Mint 11 again. this time it showed sdb, so I started the install. It got about halfway, on the progress bar, and everything locked up. No mouse, keyboard, nothing. Checked the md5 sum, everything checked out okay. Thought maybe I had a bad burn, so I downloaded from another mirror, burned it at 8x, slowest speed on my burner, but the same thing happened again. In total, I have tried 15 DVD's, and 2 CD's and Mint 11 just will not install on this system. There must be something in the setup, on this machine, that Mint 11 doesn't like.

    Yesterday I downloaded the live CD of the newly released Mageia 1, 32 bit. Installed it , but encountered a few problems afterwards. Got on their forum and one of the members suggested I try the 64 bit DVD. So, I downloaded the DVD, installed it without a hitch. Runs great. Had a problem with the sound, but it turned out to be an easy fix. So far it has run flawlessly, and is much faster, and better than the last Mandriva I tried.

     My first encounter with Linux was Mandrake 7.2, back in 2001. I had ran windows since 1991, and Commodores since 1981. Just got tired of all the windows errors, viruses, adware, malware, and having to reinstall about every 6 months.. Now windows is a game only OS, I don't trust it for online banking, e-mail, or general surfing, especially combined with IE. I use Firefox exclusively. I have 4 desktops, and 2 laptops currently in use, and use a variety of distros. My main system has Mint 10, Peppermint One, Kububtu, (seldom used), and XP for games. My Netbook runs PCLinuxOS, and my laptop runs PCLinuxOS, and Peppermint One. The new system has Xp,(older games) Win7, (newer games) PCLinuxOS, Peppermint One, and Mageia1. My 2 grandsons are the main player of games, when my wife isn't on it.

 

What I know I Give

I have been playing around with my systems, understanding their benefits, and in the process, broke them to a small extent. There are no howstos that I could refer to and there was no knowledge of what failed and why. All I had was a sense of exploration. 

When there is a problem, there is always a solution. All it takes is a little patience and SLOW reading. OF course, you can just reinstall the system if you like and do things the long and cheap way, if you have backups. However, the best part about living is to challenge yourself. Now, I don't usually do things via command line, but with a little exposer, the only thing in the way is time.

To start things off, one of my  ubuntu systems was in the updating process. It finished with no errors, as usual. A day later, the system would not boot. All I got was the grub2> prompt. I figured there had to be something wrong with the file system. The /var/log was viewed and from the last line read, the "init" was missing. Assuming that all I had to do  was replace it, this did not sound easy. I did a little digging around and uncovered some information about installing the kernel. The instructions unfortunately were not easy to follow.

With little choice, it was decided to just do it. Before the installation of the kernel was initiated, there was one method I did not try, running fsck.

When the update was executing something was accidently misplaced. Which means, the file system needs to be restructured. I booted my PC by using a liveCD of my distro, mounted the partition of my OS and ran #sudo fsck /dev/sdx1. 

Once that finished, the system said that all was finished with not errors. I booted my system with my hard drive and all was well. 

In conclusion, when faced with a system failure, remember the last thing you did before it happened, ALWAYS. Then read the syslog in /var/log folder to get more details on what happened. Understanding what the log says, you can take the more appropriate route into repairing your system. NOTE: If the system says it is missing something, and you did not delete anything, simply, as your first step, run fsck on your partition. 

I will return with another howto.

 

Create a custom distro: Create your first image (USB)

I've just received two messages with the same question related to my previous article Create your first ISO image: "Is there a way to create an USB stick with it ?". Short answer is: "Yes, there's  script to do it" but if you're still reading you may want more information. Here's what you might do if you want to create a bootable USB stick from your virtual slax build machine.

I'm still referring to my previous article: "Create a custom distro: Create your first ISO image", path and directories are still the same

Take a look at the source !

The creation of a bootable USB stick is not a mess, it's quite simple, if you take a closer look to "bootinst.sh" script located into the /boot directory you might understand what you can do with it but if you're lazy or you want to have more automation you may use the script listed below, I've created it for personal purposes but it's basically a wrapper of bootinst.sh. 

Let's keep the same virtual machine, same paths and locations, just place a script in the same dir where you've placed the script for creating the ISO image for CDs (/root/target/).

#!/bin/bash
# This script builds an ISO file from the image created
BASEPATH=/mnt/sda1/target
TARGET=/dev/sdb1
MOUNTPOINT=/mnt/sdb

echo "Mounting and formatting the USB Stick"
mkdir -p $MOUNTPOINT
umount $TARGET 2>/dev/null
mkfs.ext2 $TARGET -m 0
# Device not found or not mounted
if [[ $? -ne 0 ]]; then
    echo -e "TARGET USB DEVICE NOT FOUND, INSTALLATION FAILED !!"
    exit
fi
mount $TARGET $MOUNTPOINT
echo -e "Generating `du -kLs image` size"
echo "Copying contents into the USB Stick..."
cp -Rd $BASEPATH/image/* $MOUNTPOINT/
$MOUNTPOINT/boot/liloinst.sh
echo -e "Target disk size"
df $TARGET
umount $TARGET
echo -e  "Installation Completed"

This script is very simple but just to be clear here's what it does:

  1. Create the mount point for your usb stick
  2. Format the usb stick (ext2 is sufficient but you may use something else)
  3. Mount your usb drive
  4. Copy the contents with your new image in it
  5. Add lilo to make it bootable

In my virtual environment /dev/sda is my primary virtual hard disk, take a look at previous article for path usage, /dev/sdb is what VMWare uses when I connect an USB stick to the virtual machine. Before running this script make some checks in your current system, as you can see it's a batch script to format and wipe sdb internal contents, so you've been warned !

If you're inside my virtual environment you don't need to worry about it

liloinst.sh script is smart enough to understand where it resides and it applies lilo in the MBR of the mounted drive.

There's a lot of space for improvements so if you modify it or automate the script a little bit more just drop me few notes or send me your version and I'll modify and republish it for the community

As usual feel free to contribute with your comments

 

Previous step:

Create a custom distro: Create your first ISO image (CD)

Next step:

Create a bare bone machine with minimal footprint and requirements

 

Create a custom distro: Create your first ISO image

We finally have our own build machine, SLAX fully loaded, system up and running (in a virtual environment) so we can make a lot of tests with it.

Now it's time to create the first image, it's just a matter of minutes.

 

Preparation

When the login screen comes around you may enter in your system with the root user, “toor” is the default password with SLAX unless you change it, /etc/issue is reporting a lot of useful information for newbies as well (root/toor, xconf, startx, …)

 

Now in less than 5 minutes you'll be able to create your new virtual machine, let's create a simple environment:

root@slax:~# mkdir target target/image target/iso
root@slax:~# cd target/image/

Now it's time to copy the contents of your build machine into your target host space:

root@slax:~/target/image# cp /mnt/sda1/boot . -R
root@slax:~/target/image# cp /mnt/sda1/slax . -R
root@slax:~/target/image# ls -la
total 16
drwxr-xr-x 4 root root 4096 May 30 17:35 ./
drwxr-xr-x 3 root root 4096 May 30 17:33 ../
dr-xr-xr-x 6 root root 4096 May 30 17:35 boot/
dr-xr-xr-x 7 root root 4096 May 30 17:35 slax/

Where /mnt/sda1/ is my root partition, if you've followed my previous article you may have the same device name, expecially if you've installed it in a virtual host (I'm using VMWare), if you're not under /mnt/sda1 you may discover your root partition name, it ain't that difficult:

root@slax:~/target/image# df
Filesystem 1K-blocks Used Available Use% Mounted on
aufs 7115576 631668 6122448 10% /
tmpfs 255440 0 255440 0% /dev/shm
/dev/sda1 7115576 631668 6122448 10% /mnt/sda1

 

Now in your target directory you've the essential structure of your new target device

 

root@slax:~/target/image# ls -la
total 16
drwxr-xr-x 4 root root 4096 May 30 17:35 ./
drwxr-xr-x 3 root root 4096 May 30 17:33 ../
dr-xr-xr-x 6 root root 4096 May 30 17:35 boot/
dr-xr-xr-x 7 root root 4096 May 30 17:35 slax/

This is just what you need to create your new target system

 

 

 

Creating a target ISO

to create a target ISO with an exact copy of your system you only need to use a ready made script file located under target/image/slax directory, invocation is straightforward:

root@slax:~/target# ./image/slax/make_iso.sh iso/mycoolimage.iso
2.01.01a53 (i686-pc-linux-gnu)
Scanning ../.
Scanning .././boot
Scanning .././boot/dos
Scanning .././boot/isolinux
Excluded by match: .././boot/isolinux/isolinux.boot
Scanning .././boot/pxelinux.cfg
Scanning .././boot/pxelinux.cfg/web
Scanning .././boot/pxelinux.cfg/web/conf
Scanning .././boot/syslinux
Scanning .././slax
Scanning .././slax/base
Scanning .././slax/modules
Scanning .././slax/optional
Scanning .././slax/rootcopy
Scanning .././slax/tools
Scanning .././slax/tools/WIN
Writing: Initial Padblock Start Block 0
Done with: Initial Padblock Block(s) 16
Writing: Primary Volume Descriptor Start Block 16
Done with: Primary Volume Descriptor Block(s) 1
Writing: Eltorito Volume Descriptor Start Block 17
Size of boot image is 4 sectors -> No emulation
Done with: Eltorito Volume Descriptor Block(s) 1
Writing: Joliet Volume Descriptor Start Block 18
Done with: Joliet Volume Descriptor Block(s) 1
Writing: End Volume Descriptor Start Block 19
Done with: End Volume Descriptor Block(s) 1
Writing: Version block Start Block 20
Done with: Version block Block(s) 1
Writing: Path table Start Block 21
Done with: Path table Block(s) 4
Writing: Joliet path table Start Block 25
Done with: Joliet path table Block(s) 4
Writing: Directory tree Start Block 29
Done with: Directory tree Block(s) 17
Writing: Joliet directory tree Start Block 46
Done with: Joliet directory tree Block(s) 15
Writing: Directory tree cleanup Start Block 61
Done with: Directory tree cleanup Block(s) 0
Writing: Extension record Start Block 61
Done with: Extension record Block(s) 1
Writing: The File(s) Start Block 62
4.89% done, estimate finish Mon May 30 17:44:43 2011
9.77% done, estimate finish Mon May 30 17:44:43 2011
14.65% done, estimate finish Mon May 30 17:44:43 2011
19.53% done, estimate finish Mon May 30 17:44:43 2011
24.42% done, estimate finish Mon May 30 17:44:43 2011
29.29% done, estimate finish Mon May 30 17:44:43 2011
34.18% done, estimate finish Mon May 30 17:44:45 2011
39.05% done, estimate finish Mon May 30 17:44:45 2011
43.94% done, estimate finish Mon May 30 17:44:45 2011
48.82% done, estimate finish Mon May 30 17:44:45 2011
53.71% done, estimate finish Mon May 30 17:44:44 2011
58.58% done, estimate finish Mon May 30 17:44:44 2011
63.47% done, estimate finish Mon May 30 17:44:44 2011
68.34% done, estimate finish Mon May 30 17:44:44 2011
73.22% done, estimate finish Mon May 30 17:44:44 2011
78.11% done, estimate finish Mon May 30 17:44:44 2011
82.99% done, estimate finish Mon May 30 17:44:44 2011
87.86% done, estimate finish Mon May 30 17:44:44 2011
92.75% done, estimate finish Mon May 30 17:44:44 2011
97.64% done, estimate finish Mon May 30 17:44:45 2011
Total translation table size: 2048
Total rockridge attributes bytes: 9444
Total directory bytes: 32768
Path table size(bytes): 208
Done with: The File(s) Block(s) 102222
Writing: Ending Padblock Start Block 102284
Done with: Ending Padblock Block(s) 150
Max brk space used 21000
102434 extents written (200 MB)

And that's it, you now have a file called mycoolimage.iso under the iso directory

 

root@slax:~/target# ls -la iso/
total 205084
drwxr-xr-x 2 root root 4096 May 30 17:44 ./
drwxr-xr-x 4 root root 4096 May 30 17:36 ../
-rw-r--r-- 1 root root 209784832 May 30 17:44 mycoolimage.iso

you may create a little script if you want to automate the whole process:

#!/bin/bash
# This script builds an ISO file from the image created
BASEPATH=/mnt/sda1/target
REMOTEHOST=192.168.84.1

mkdir -p $BASEPATH/iso
$BASEPATH/image/slax/make_iso.sh $BASEPATH/iso/image.iso
scp $BASEPATH/iso/image.iso ben@$REMOTEHOST:/iso/image.iso

p { margin-bottom: 0.08in; }

To make my own tests I've two virtual machines, one it's called “Slax Build Machine”, one it's called “Slax Test Machine”. “Slax Build Machine” is the machine we're now using and the same machine we've built in the previous chapter. “Slax Test Machine” it's an empty VMWare virtual machine, no hard drives, just a virtual cdrom device pointing to an ISO file located on the real host (in the directory /tmp/image.iso). When I'd like to make some tests I just need to create an ISO file (in the build machine with the previous script). The script copies the .ISO image in the physical host after the creation, then I start the Test Machine when needed, I don't need to waste CDs or reinstall/configure the virtual machine each time, it's really easy

 

In this chapter we've just created an identical copy of our build machine, in the next chapter we'll customize the test machine, we'll start with a bare bone system and we'll add modules/functionalities later on

I guess this phase is really easy but it's important to understand it well, from the next phase you'll understand why customizing and building a slax machine is really easy and powerful

 

Previous Chapters

Next Chapters

 

As usual let me know your thoughts and drop a note below if you want

 

Andrea (Ben) Benini

 

 

 

 

Calibre dropped down from openSUSE Repositories

We dropping down all Calibre Packages from all openSUSE Repositories.

Since a February 2009 the openSUSE Packagers are packaging calibre for openSUSE. The first comitted Version was 0.4.134, and was comitted by Thomas Schraitle. Since that Version we have done some little things to integrate calibre better into the openSUSE Distribution. Since that time it was possible to provide packages for all actual published openSUSE Versions.

With the start of publishing 0.8.x the Developerteam switched to bleeding edge version of python. We now have that python version into our Factory Repository, but not into 11.4. In short that means, that we can't provide any calibre version for published openSUSE versions.

Yesterday i've chatted with the main developer Kovid Goyal, and explained him the trouble. But sadly he isn't interested to support us in that, through not using bleeding edge.

So we decided to drop down the calibre package from the Documentation:Tools Repository and from openSUSE:Factory, because it looks like the Project aren't interested to help us out. The removing of the packages should done this week.

I'm sorry to write that, 'cause i liked this piece of Software.

 

First blog post...

Well, I guess I will start a little about me, how I got involved in Linux and where I hope it takes me in the future!

I'm Justin, 32 years old in NJ.  I work in law enforcement and I've been dabbling in the Linux realm for the past 5 years.  My first distro was Red Hat... I bought it off ebay and I had no idea what I was doing.  I got frustrated and continued with Windows but dual booted Ubuntu and a few other distros here and there.

Over the past 2 years I've been reading and using it for data recovery jobs for my friends.  Within the past year I've really made a push to use Linux primarily and I'm on a good track right now. 

While still very green with Linux I have learned a lot and it just never stops.  The flexibility, stability and high customization is what I like the most.  The open source community is amazing, the support is amazing and it's free! 

I wiped my windows hdd in my laptop and that is purely Linux.  My hopes are to show people how Linux will benefit them and for business how much money it can save. 

 

My wife and I operate a small internet store selling vinyl wall art and I haven't changed some of the art to a different format yet so I'm sorry to say I have to keep the XP machine around a bit longer.  Perhaps after the summer as boating will occupy most of my time (hah)!

I look forward to the community and hope to contact linux users in my area soon!

 

 

 

openSUSE Weekly News 177 is out!

I'm pleased to announce our new openSUSE Weekly News Issue 177. As ever available in two different html styles and one pdf.

In this Issue:

  • openSUSE renames OBS
  • Bryen Yunashko: Ready…Set…Code!
  • Andreas Jaeger: Factory Progress
  • Matt Barringer: SUSE Gallery Desktop Client
  • and many more ...

You can download it there:

We hope you enjoy the reading :-)

If you want to help us collecting interesting articles for the openSUSE Weekly News, so you can all your stuff into our new ietherpad: http://os-news.ietherpad.com/2.

Found Bugs? Please place it in our Bugtracker: http://developer.berlios.de/bugs/?group_id=12095

Features, Ideas and Improvements can placed in our Featuretracker: http://developer.berlios.de/feature/?group_id=12095

Older content can be found there.

Flattr this

 

Build your own Linux? Where to start...Two powerful distro's you need to know..

The two distro's that will not make you headache when building a system are available from Bodhi Linux and iqunix. Both big systems with minimal  software. Iqunix is new to me and I liked it instantly and will use it for a build. Huge improvements by Bodhi. I tested Bodhi Linux about a year ago and although it had the best looking eye candy there were too many bugs to enjoy and it felt danty. This time no issues and it still looks Pimp offering various levals of distros to choose from on install including one with ( I forget maybe 3 apps at most) . Bohdi and iqunix are calling you if you want to build your own system!!! And yes they have the software to remaster your dev.

 

 

 

openSUSE Weekly News 176 is out!

We are pleased to announce our openSUSE Weekly News Issue 176.

In this Issue:

  • openSUSE Conference 2011 to be creative and open!
  • Manu Gupta: My GSoC Project – SaX3
  • Linux Tag 2011
  • Will Stephenson: Have you BURPed yet today?
  • and many more ...

You can download it there:

We hope you enjoy the reading :-)

If you want to help us collecting interesting articles for the openSUSE Weekly News, so you can all your stuff into our new ietherpad: http://os-news.ietherpad.com/2.

Found Bugs? Please place it in our Bugtracker: http://developer.berlios.de/bugs/?group_id=12095

Features, Ideas and Improvements can placed in our Featuretracker: http://developer.berlios.de/feature/?group_id=12095

Older content can be found there.

Flattr this

 
Page 7 of 122
30 Linux Kernel Developers in 30 Weeks

Who we are ?

The Linux Foundation is a non-profit consortium dedicated to the growth of Linux.

More About the foundation...

Frequent Questions

Join / Linux Training / Board