Posts Tagged ‘dev’
Tuesday, July 6th, 2010 I’ve recently broke my grub untentianally while whiping out one of my disk partitions who was prepared to run a hackintosh.
Thus yesterday while switching on my notebook I was unpleasently surprised with an error Grub Error 17 and the boot process was hanging before it would even get to grub’s OS select menu.
That was nasty and gave me a big headache, since I wasn’t even sure if my partitions are still present.
What made things even worse that I haven’t created any backups preliminary to prepare for an emergency!
Thus restoring my system was absolutely compulsory at any cost.
In recovering the my grub boot manager I have used as a basis of my recovery an article called How to install Grub from a live Ubuntu cd
Though the article is quite comprehensive, it’s written a bit foolish, probably because it targets Ubuntu novice users 🙂
Another interesting article that gave me a hand during solving my issues was HOWTO: install grub with a chroot
Anyways, My first unsuccessful attempt was following a mix of the aforementioned articles and desperately trying to chroot to my mounted unbootable partition in order to be able to rewrite the grub loader in my MBR.
The error that slap me in my face during chroot was:
chroot: cannot execute /bin/sh : exec format error
Ghh Terrible … After reasoning on the shitty error I came to the conclusion that probably the livecd I’m trying to chroot to my debian linux partition is probably using a different incompatible version of glibc , if that kind of logic is true I concluded that it’s most likely that the glibc on my Linux system is newer (I came to that assumption because I was booting from livecds (Elive, Live CentOS as well Sabayon Linux, which were burnt about two years ago).
To proof my guesses I decided to try using Debian testing Squeeze/Sid install cd . That is the time to mention that I’m running Debian testing/unstable under the code name (Squeeze / Sid).
I downloaded the Debian testing amd64 last built image from here burnt it to a cd on another pc.
And booted it to my notebook, I wasn’t completely sure if the Install CD would have all the necessary recovery tools that I would need to rebuilt my grub but eventually it happened that the debian install cd1 has everything necessary for emergency situations like this one.
After I booted from the newly burned Debian install cd I followed the following recovery route to be able to recovery my system back to normal.It took me a while until I come with the steps described here, but I won’t get into details for brevity
1. Make new dir where you intend to mount your Linux partition and mount /proc, /dev, /dev/pts filesystems and the partition itself
noah:~# mkdir /mnt/root
noah:~# mount -t ext3 /dev/sda8 /mnt/root
noah:~# mount -o bind /dev /mnt/root/dev
noah:~# mount devpts /dev/pts -t devpts
Change /dev/sda8 in the above example commands with your partition name and number.
2. chroot to the mounted partition in order to be able to use your filesystem, exactly like you normally use it when you’re using your Linux partition
noah:~# chroot /mnt/root /bin/bash
Hopefully now you should be in locked in your filesystem and use your Linux non-bootable system as usual.
Being able to access your /boot/grub directory I suggest you first check that everything inside:
/boot/grub/menu.lst is well defined and there are no problems with the paths to the Linux partitions.
Next issue the following commands which will hopefully recover your broken grub boot loader.
noah:~# grub
noah:~# find /boot/grub/stage1
The second command find /boot/grub/stage1 should provide you with your partitions range e.g. it should return something like:
root (hd0,7)
Nevertheless in my case instead of the expected root (hd0,7) , I was returned
/boot/grub/stage1 not found
Useless to say this is uncool 🙂
As a normal reaction I tried experimenting in order to fix the mess. Logically enough I tried to reinstall grub using the
noah:~# grub-install --root-directory=/boot /dev/sda
noah:~# update-grub
To check if that would fix my grub issues I restarted my notebook. Well now grub menu appeared with some error generated by splashy
Trying to boot any of the setup Linux kernels was failing with some kind of error where the root file system was trying to be loaded from /root directory instead of the normal / because of that neither /proc /dev and /sys filesystems was unable to be mounted and the boot process was interrupting in some kind of rescue mode similar to busybox, though it was a was less flexible than a normal busybox shell.
To solve that shitty issue I once again booted with the Debian Testing (Sid / Squeeze ) Install CD1 and used the commands displayed above to mount my linux partition.
Next I reinstalled the following packages:
noah:~# apt-get update
noah:~# apt-get install --reinstall linux-image-amd64 uswsusp hibernate grub grub-common initramfs-tools
Here the grub reinstall actually required me to install the new grub generation 2 (version 2)
It was also necessary to remove the splashy
noah:~# apt-get remove splashy
As well as to grep through all my /etc/ and look for a /dev/sda6 and substitute it with my changed partition name /dev/sda8
One major thing where I substituted /dev/sda6 to my actual linux partition now with a name /dev/sda8 was in:
initramfs-tools/conf.d/resumeThe kernel reinstall and consequently (update) does offered me to substitute my normal /dev/sda* content in my /etc/fstab to some UUIDS like UUID=ba6058da-37f8-4065-854b-e3d0a874fb4e
Including this UUIDs and restarting now rendered my system completely unbootable … So I booted once again from the debian install cd .. arrgh 🙂 and removed the UUID new included lines in /etc/fstab and left the good old declarations.
After rebooting the system now my system booted once again! Hooray! All my data and everything is completely intact now Thanks God! 🙂
Tags: assumption, boot manager, boot process, chroot, debian linux, dev, disk partitions, exec format error, filesystems, glibc, gnu linux, grub boot loader, grub error 17, hackintosh, incompatible version, linux partition, linux system, livecd, MBR, mkdir, mnt, noah, novice users, Recover/Restore unbootable GRUB boot loader on Debian Testing GNU/Linux using Linux LiveCD (Debian Install CD1), recovery, squeeze, unsuccessful attempt, whiping
Posted in Linux, Linux and FreeBSD Desktop, System Administration | 8 Comments »
Wednesday, April 16th, 2014 
If you login to server and after running dmesg – show kernel log command you get tons of messages like:
# dmesg
end_request: I/O error, dev sdc, sector 0
sd 0:0:0:1: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
sd 0:0:0:0: SCSI error: return code = 0x00010000
end_request: I/O error, dev sda, sector 0
sd 0:0:0:2: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdc, sector 0
sd 0:0:0:1: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
sd 0:0:0:0: SCSI error: return code = 0x00010000
end_request: I/O error, dev sda, sector 0
sd 0:0:0:2: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdc, sector 0
sd 0:0:0:1: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
sd 0:0:0:0: SCSI error: return code = 0x00010000
end_request: I/O error, dev sda, sector 0
In /var/log/messages there are also log messages present like:
# cat /var/log/messages
...
Apr 13 09:45:49 sl02785 kernel: end_request: I/O error, dev sda, sector 0
Apr 13 09:45:49 sl02785 kernel: klogd 1.4.1, ———- state change ———-
Apr 13 09:45:50 sl02785 kernel: sd 0:0:0:1: SCSI error: return code = 0x00010000
Apr 13 09:45:50 sl02785 kernel: end_request: I/O error, dev sdb, sector 0
Apr 13 09:45:55 sl02785 kernel: sd 0:0:0:2: SCSI error: return code = 0x00010000
Apr 13 09:45:55 sl02785 kernel: end_request: I/O error, dev sdc, sector 0
This is a sure sign something is wrong with SCSI hard drives or SCSI controller, to further debug the situation you can use the multipath command, i.e.:
multipath -ll | grep -i failed
_ 0:0:0:2 sdc 8:32 [failed][faulty]
_ 0:0:0:1 sdb 8:16 [failed][faulty]
_ 0:0:0:0 sda 8:0 [failed][faulty]
As you can see all 3 drives merged (sdc, sdb and sda) to show up on 1 physical drive via the remote network connected – LUN to the server is showing as faulty. This is a clear sign something is either wrong with 3 hard drive members of LUN – (Logical Unit Number) (which is less probable) or most likely there is problem with the LUN network SCSI controller. It is common error that LUN SCSI controller optics cable gets dirty and needs a physical clean up to solve it.
In case you don't know what is storage LUN? – In computer storage, a logical unit number, or LUN, is a number used to identify a logical unit, which is a device addressed by the SCSI protocol or protocols which encapsulate SCSI, such as Fibre Channel or iSCSI. A LUN may be used with any device which supports read/write operations, such as a tape drive, but is most often used to refer to a logical disk as created on a SAN. Though not technically correct, the term "LUN" is often also used to refer to the logical disk itself.
What LUN's does is very similar to Linux's Software LVM (Logical Volume Manager).
Tags: code, dev, dmesg, How to, kernel, Linux, log, Logical Unit Number, LUN, SCSI, sda, sdb, sdc, sector, sign
Posted in System Administration | No Comments »
Wednesday, September 16th, 2009 I tried using festival today just to realize it doesn’t work anymore.
For instance # echo test |festival -tts would fail with the annoying
“Linux: can’t open /dev/dsp” error message. I found the solution in ubuntuforums,
the solution is originally taken from “the Gentoo Speechd Howto“.
The solution is to create .festivalrc in your home as well as to the homes of all usersintending to use festival.
Here is how:
printf ";use ALSAn(Parameter.set 'Audio_Method 'Audio_Command)n(Parameter.set 'Audio_Command "aplay -q -c 1 -t raw -f s16 -r $SR $FILE")n" > .festivalrc
Tags: alsa, dev, doesn, dsp, echo, fix, home, Linux, message, solution, today, use, work
Posted in Linux and FreeBSD Desktop, Linux Audio & Video | No Comments »
Wednesday, February 1st, 2012 Earlier I've blogged on How to Work Around periodically occuring TrackPoint Thinkpad R61 issues on GNU / Linux . Actually I thought the fix I suggested there is working but I was wrong as the problems with the trackpoint reappeared at twice or thrice a day.
My suggested fix was the use of one script that does periodically change the trackpoint speed and sensitivity to certain numbers.
The fix script to the trackpoint hanging issue is here
Originally I wrote the script has to be set to execute through crontab on a periods like:
0,30 * * * * /usr/sbin/restart_trackpoint.sh >/dev/null 2>&1
Actually the correct values for the crontab if you use my restart_trackpoint.sh script are:
0,5,10,15,20,25,30,35,40,45,50,55,58 * * * * /usr/sbin/restart_trackpoint.sh >/dev/null 2>&3
ig it has to be set the script is issued every 5 minutes to minimize the possibility for the Thinkpad trackpoint hang up issue.
One other thing that helps if trackpoint stucks is setting in /etc/rc.local is psmouse module to load with resetafter= parameter:
echo '/sbin/rmmod psmouse; /sbin/modprobe psmouse resetafter=30' >> /etc/rc.local
Tags: amp, blogged, change, correct values, crontab, dev, fix, gnu linux, hang ups, ig, issue, issue one, Linux, modprobe, Module, mouse, mouse pointer, nbsp, null, parameter, periods, psmouse, resetafter, rmmod, sbin, sensitivity, speed, stucks, thinkpad, thrice, trackpoint, ups, use, usr
Posted in Linux, Linux and FreeBSD Desktop, Linux Audio & Video | 1 Comment »
Thursday, December 8th, 2011 The usual way that one is supposed to check a certain partition let’s say /dev/sda1 disk UUID (Universal Unique Identifier) label is through a command:
vol_id /dev/sda1
For reason however Debian does not include vol_id command. To check the UUID assigned disk labels on Debian one should use another command called blkid (part of util-linux deb package).
blkid will list all block device attributes so it doesn’t specifically, passing any partition as argument.
Here is an example output of blkid :
server:/root# blkid
/dev/sda1: UUID="cdb1836e-b7a2-4cc7-b666-8d2aa31b2da4" SEC_TYPE="ext2" TYPE="ext3"
/dev/sda5: UUID="c67d6d43-a48f-43ff-9d65-7c707a57dfe6" TYPE="swap"
/dev/sdb1: UUID="e324ec28-cf04-4e2e-8953-b6a8e6482425" TYPE="ext2"
/dev/sdb5: UUID="1DWe0F-Of9d-Sl1J-8pXW-PLpy-Wf9s-SsyZfQ" TYPE="LVM2_member"
/dev/mapper/computer-root: UUID="fbdfc19e-6ec8-4000-af8a-cde62926e395" TYPE="ext3"
/dev/mapper/computer-swap_1: UUID="e69100ab-9ef4-45df-a6aa-886a981e5f26" TYPE="swap"
/dev/mapper/computer-home: UUID="2fe446da-242d-4cca-8b2c-d23c76fa27ec" TYPE="ext3"
Tags: cca, command, computer swap, deb, deb package, debian gnu, dev, Disk, disk labels, doesn, DWe, ext, ff, gnu linux, Linux, linux deb, LVM, mapper, package, partition, partition disk, quot dev, reason, root, sda, sdb, sec type, server root, Unique, UUID, vol, way
Posted in Linux, System Administration, Various | No Comments »
Thursday, October 27th, 2011 I just installed munin to track in web the performance of few Debian servers. I’ve configured munin to open via a Virtualhosts in Apache. As its always wise to protect any statistics data about the server from the unwanted possible security violators, I decided to protect Munin with Apache .htaccess.
The munin htmldir output dir is configured to be in /var/www/munin, hence I protected my munin with password by:
1. Creating .htaccess file in /var/www/munin with following content
AuthUserFile /etc/apache2/.munin_htpasswd
AuthGroupFile /dev/null
AuthName EnterPassword
AuthType Basic
require user admin
2. Creating /etc/apache2/.munin_htpasswd with htpasswd (htaccess password generator cmd)
debian:/var/www/munin# htpasswd -c /etc/apache2/.munin_htpasswd admin
New password:
Re-type new password:
Adding password for user admin
Another important thing I had to do is set my VirtualHost file to be configured with AllowOverride All , if AllowOverride All is missing the .htaccess and .htpasswd are not red at all.
Afterwards munin is protected with password, and when my virtualdomain where munin lays e.g. http://munin.mydomain.com is accessed the .htpasswd password dialog pops up 😉
Tags: Allowoverride, apache, apache htaccess, AuthGroupFile, AuthType, AuthUserFile, Creating, data, dev, dialog, dir, EnterPassword, generator, GNU, htaccess password generator, htpasswd, Linux, null, password, performance, pops, security, servers, statistics data, var, Virtualhost, web statistics, www
Posted in Linux, System Administration, Web and CMS | No Comments »
Saturday, July 16th, 2011 
It was necessery to clean up some squid cache for some proxy users on a Debian host. Until now I’ve used to run only custom build Squid server on Slackware Linux.
Thus I was curious if Debian guys were smart enough to implement a proxy cache cleaning option as an option to be passed on to squid’s init script.
Honestly I was quite suprised squid clear cache option is not there;
squid-cache:~# /etc/init.d/squid3
Usage: /etc/init.d/squid3 {start|stop|reload|force-reload|restart}
squid-cache:/#
As it was not embedded into init script I still hoped, there might be some Debian way to do the proxy cache clearing, so I spend some 10 minutes checking online as well as checked in squid3‘s manual just to find there is no specific command or Debian accepted way to clean squid’s cache.
Since I couldn’t find any Debian specific, way I did it the old fashioned way 😉 (deleted directory/file structures in /var/spool/squid3/* and used squid’s -z option, to recreate the swap directories.
Here is how:
squid-cache:~# /etc/init.d/squid3 stop;
squid-cache:~# rm -Rf /var/spool/squid3/*;
squid-cache:~# squid3 -z; /etc/init.d/squid3 start
Finally I was quite amazed to realize, there was not even a crontab script to periodically clear and re-create proxy cache.
My previous experience with maintaning an office Squid proxy cache has prooved, that periodic cache clean ups are very helpful, especially to resolve issues with cached unreslovable DNS entries in the server.
Clearing up squid cache every week or something, guarantees that failure to resolve certain hosts at certain times would not stay unresolvable like forever 😉
In that manner of thougths, I decided to put the following crontab which will twice a month clear up proxy’s cache, to possibly solve some failed squid DNS issues.
squid-cache:~# crontab -u root -l > file;
echo '00 04 12,26 * * /etc/init.d/squid3 stop; rm -Rf /var/spool/squid3/*; squid3 -z; /etc/init.d/squid3 start >/dev/null 2>&1'
>> file; crontab file
By the way, implementing the squid clear cache in Debian and Ubuntu ‘s init scripts and putting a periodic proxy clear up cron, seems like a feature worthy to be proposed to the distro developers and hopefully be embbed in some of the upcoming distro releases 😉
Tags: amp, cache option, clear cache, clearing, command, cron, crontab script, dev, DNS, dns entries, experience, failure, feature, fi, fil, file, file structures, guarantees, Honestly, hosts, init script, Linux, necessery, online, option, proxy, proxy cache, proxy users, rf, root, something, specific, spool, squid, squid cache, Stop, thougths, Ubuntu, ups, Usage, var, way
Posted in Linux, System Administration | No Comments »
Wednesday, July 13th, 2011 One server with a broken Raid array was having troubles with it’s software raid.
I tried to scan the raid array via a rescue cd like so:
server:~# mdadm --assemble --scan /dev/md1
just to be suprised by the message:
mdadm: /dev/md1 assembled from 2 drives – not enough to start the array.
In /proc/mdstat respectively the raid was showing inactive, e.g.:
server:~# cat /proc/mdstat
Personalities : [raid10] [raid1]
md1 : inactive sda2[0] sdc2[2] sdb2[1]
12024384 blocks
Respectively trying to activate the software Linux raid array with:
server:~# mdadm -A -s
Couldn’t be completed because of the same annoying error:
/dev/md1 assembled from 2 drives – not enough to start the array.
Thanksfully finally thanks to some Russian, who posted having same issues reported to be able to active his software RAID with mdadm’s –force option.
Thus enabling the problematic RAID 5 array was possible with:
server:~# mdadm -A -s --force
This solution of course is temporary and will have to further check what’s wrong with the array, however at least now I can chroot to the server’s / directory. 😉
Tags: annoying error, cat, check, course, dev, drive, ERROR, force option, Linux, md1, mdstat, option, Personalities, proc, Raid, raid 5 array, raid array, rescue, rescue cd, sCould, sda, sdb, sdc, software, software linux, software raid, solution, Thanksfully, way
Posted in Linux, System Administration | 1 Comment »
Saturday, July 9th, 2011 I’m currently writting a script which is supposed to be adding new crontab jobs and do a bunch of other mambo jambo.
By so far I’ve been aware of only one way to add a cronjob non-interactively like so:
linux:~# echo '*/5 * * * * /root/myscript.sh' | crontab -
Though using the | crontab – would work it has one major pitfall, I did completely forgot | crontab – OVERWRITES CURRENT CRONTAB! with the crontab passed by with the echo command.
One must be extremely careful if he decides to use the above example as you might loose your crontab definitions permanently!
Thanksfully it seems there is another way to add crontabs non interactively via a script, as I couldn’t find any good blog which explained something different from the classical example with pipe to crontab –, I dropped by in the good old irc.freenode.net to consult the bash gurus there 😉
So I entered irc and asked the question how can I add a crontab via bash shell script without overwritting my old existing crontab definitions less than a minute later one guy with a nickname geirha was kind enough to explain me how to get around the annoying overwridding.
The solution to the ovewrite was expected, first you use crontab to dump current crontab lines to a file and then you append the new cron job as a new record in the file and finally you ask the crontab program to read and insert the crontab definitions from the newly created files.
So here is the exact code one could run inside a script to include new crontab jobs, next to the already present ones:
linux:~# crontab -l > file; echo '*/5 * * * * /root/myscript.sh >/dev/null 2>&1' >> file; crontab file
The above definition as you could read would make the new record of */5 * * * * /root/myscript.sh >/dev/null be added next to the existing crontab scheduled jobs.
Now I’ll continue with my scripting, in the mean time I hope this will be of use to someone out there 😉
Tags: bash scripts, bash shell script, blog, classical example, cron, cron job, cron jobs, crontab, definitions, dev, echo, echo 5, echo command, exact code, file, fileThe, gurus, irc, jambo, kind, line, Linux, mambo jambo, mean time, minute, nbsp nbsp nbsp nbsp nbsp, nickname, ovewrite, pipe, pitfall, root, scripting, Shell, solution, someone, Thanksfully, time, use, way
Posted in Linux, System Administration | 9 Comments »
Wednesday, April 20th, 2011 I’m using a backtrack Linux to recover a broken Ubuntu Linux, to fix this disastrous situation I’m using the Ubuntu Linux through chroot after mounting my /dev/sda1, where my Linux resides with:
linux-recovery:~# mkdir /mnt/test1
linux-recovery:~# mount /dev/sda1 /mnt/test1
linux-recovery:~# chroot /mnt/test1
ubuntu:~#
I consequently needed to mount up the /proc and /dev partition inside the chroot environment.
Here is how I did it:
ubuntu:~# mount /proc
ubuntu:~# mount -a
Next I switched on on a different virtual console in the backtrack and to mount /dev issued the commands:
linux-recovery:~# mount --bind /dev /mnt/test1/dev
Now once again, I can use theapt-get inside the chroot to fix up the whole mess …
Tags: aNext, backtrack, backtrack linux, Bind, chroot, chroot environment, dev, devNow, disastrous situation, environment, Linux, mess, mkdir, mnt, mount dev, partition, proc, procubuntu, sda, situation, test, test1, Ubuntu, ubuntu linux
Posted in Linux | 2 Comments »