Archive

Archive for the ‘bash’ Category

replace a string recursively in multiple files

September 4, 2015 Leave a comment

Problem
In a folder I have lots of files, and the folder also has subdirectories with further files. I want to replace a string in all of them. I want to modify the files in place.

For instance, you want to update a bunch of Python scripts from version 2 to version 3, thus you want to modify the first lines from “#!/usr/bin/env python2” to “#!/usr/bin/env python3“.

Solution
I found the solution here:

grep -rli 'old-word' * | xargs -i@ sed -i 's/old-word/new-word/g' @

WARNING! Since it modifies the files in place, be very careful! First run just the beginning of the command above:

grep -rli 'old-word' *

Refine it if necessary. If it finds exactly what you need, then proceed.

Categories: bash Tags: , , ,

check a new HDD for bad blocks

September 1, 2015 Leave a comment

Problem
You buy/get a new HDD and you want to make sure that it has no bad sectors. How to check that?

Solution
I found a solution here. In short:

sudo badblocks -wvs /dev/sdx

where /dev/sdx is the drive. WARNING! This is a destructive check, i.e. all data on the drive will be lost!

The -w option tells badblocks to write a known pattern, then read back the data to make sure it matches the pattern. It does this 4 times, using the patterns 0xaa, 0x55, 0xff, and 0x00 (alternating 0’s and 1’s, then all 1’s, then all 0’s). Note that this will overwrite all data on the drive and wipe out all the partitions, as well.” (source)

Categories: bash, linux Tags: , ,

[manjaro] compile MongoDB from source and install it manually

August 20, 2015 Leave a comment

Problem
I wanted to install MongoDB on an old 32-bit laptop of mine. After installing it from the official repository, the mongo shell died with the following error: ” Illegal instruction (core dumped)”.

Solution
The problem is that the binary file uses an instruction set that is not supported by my old 32-bit CPU:

$ cat /proc/cpuinfo | grep flags
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov clflush dts acpi mmx fxsr sse sse2 ss tm pbe nx bts est tm2

SSE3 is missing :( What to do? Uninstall the mongodb package and 1) compile it from source, 2) install it and 3) make it start at startup.

(1) Compile the source
I found an excellent blog post about it at http://kdenby.homelinux.com/?p=25 . Here I would sum up the steps.

Download the source from here: https://www.mongodb.org/downloads . It’s about 35 MB. The source uses the SCons build system, so install the “scons” package too. Note that the compilation process took about 6 hours on my old laptop and the work directory where compilation was being done grew to 18 GB! I didn’t have that much free space so I had to move this work directory to an external HDD :) However, when you install the binaries, they will only occupy 90 MB.

So, if you have an old CPU, edit the file SConstruct:

        if using_gcc() or using_clang():
                myenv.Append( CCFLAGS=['-march=pentium3', '-mtune=generic'] )
                #myenv.Append( CCFLAGS=['-march=nocona', '-mtune=generic'] )

The target architecture “nocona” was replaced by “pentium3”. Again, this tip is from Karl.

After this you can start the compilation and wait :)

$ scons --disable-warnings-as-errors --32 all

The blog post at http://kdenby.homelinux.com/?p=25 stops at this point.

(2) Install the binaries
I installed the binaries to /opt:

$ scons --prefix=/opt/mongo-3.0.5 install

Of course, replace “3.0.5” with your current version. I also put a symbolic link on this folder:

$ cd /opt
$ ln -s mongo-3.0.5 mongo

Note that the binaries are put into a “bin” folder, thus they are located in “/opt/mongo-3.0.5/bin/“.

Now put symbolic links in /usr/bin/ that point to the appropriate binaries in /opt/mongo/bin/. They look like this on my machine:

$ cd /usr/bin
$ ls -al | grep mongo
lrwxrwxrwx  1 root root         20 11.08.2015 08:30 mongo -> /opt/mongo/bin/mongo
lrwxrwxrwx  1 root root         21 11.08.2015 08:31 mongod -> /opt/mongo/bin/mongod
lrwxrwxrwx  1 root root         24 11.08.2015 08:31 mongoperf -> /opt/mongo/bin/mongoperf
lrwxrwxrwx  1 root root         21 11.08.2015 08:31 mongos -> /opt/mongo/bin/mongos
lrwxrwxrwx  1 root root         25 11.08.2015 08:31 mongosniff -> /opt/mongo/bin/mongosniff

(3) Make MongoDB start at startup
Create these two directories: /var/lib/mongodb and /var/log/mongodb . The mongo daemon shouldn’t run with root privileges, so we create a dedicated user for this purpose and set its password:

$ sudo adduser mongodb
$ sudo passwd mongodb

Now, the owner of the previously created two folders will be “mongodb“, and they will belong to the “daemon” group. Example on my machine:

$ cd /var/log
$ ls -al | grep mongodb
drwxr-xr-x  2 mongodb daemon            4096 11.08.2015 09:29 mongodb/

Use the chown and chgrp commands.

After this, you need to copy some config files as root. They are collected here.

First, create /etc/mongodb.conf . Mine looks like this:

# See http://www.mongodb.org/display/DOCS/File+Based+Configuration for format details
# Run mongod --help to see a list of options

bind_ip = 127.0.0.1
quiet = true
dbpath = /var/lib/mongodb
logpath = /var/log/mongodb/mongod.log
logappend = true

noprealloc = true
smallfiles = true

Next, create /usr/lib/systemd/system/mongodb.service . Sample:

[Unit]
Description=High-performance, schema-free document-oriented database
After=network.target

[Service]
User=mongodb
ExecStart=/usr/bin/mongod --quiet --config /etc/mongodb.conf

[Install]
WantedBy=multi-user.target

Now it’s time to enable and start the mongodb service:

$ systemctl enable mongodb.service
$ systemctl start mongodb.service

Verify if it’s running:

$ systemctl status mongodb.service

If it seems OK, try to connect to it with the mongo shell using the command “mongo“.

Troubleshooting
If the service doesn’t start with the systemctl command, then try to launch the server manually:

$ mongod --quiet --config /etc/mongodb.conf

You may want to remove the “--quiet” option. If it has some problems, it will print a more detailed error message than “systemctl start...“.

I wrote this post one week after I installed MongoDB, but I hope I didn’t forget any steps. Good luck!

Categories: bash, linux Tags: , , , ,

get the UUID of a partition

July 30, 2015 2 comments

Problem
You want to figure out the UUID of a partition. For instance, you want to mount a partition upon boot and thus you want to add it to your /etc/fstab.

Solution

$ sudo blkid

The blkid program … can determine the type of content (e.g. filesystem or swap) that a block device holds, and also the attributes (tokens, NAME=value pairs) from the content metadata (e.g. LABEL or UUID fields).” (source: man)

Update (20150803)
As msx pointed out in the comments, you can also write “lsblk -f“, which doesn’t require sudo rights at all.

U

Categories: bash Tags: , , ,

taking a screenshot about a webpage

July 11, 2015 Leave a comment

Problem

You know the URL of a webpage and you want to take a screenshot of it. For instance you want a thumbnail about the webpage.

Solution

It can be done very nicely with PhantomJS.

What is PhantomJS?
PhantomJS is a headless WebKit scriptable with a JavaScript API. It has fast and native support for various web standards: DOM handling, CSS selector, JSON, Canvas, and SVG.” (source)

How to install PhantomJS?
Follow the instructions here. Under Ubuntu I compiled it from source to get the latest version. Note that it takes a lot of time (about 30 minutes). Under Manjaro I could install it via yaourt and it took 1 minute (and got the newest version). The good news is that installation is not a problem.

How to take a screenshot?
If you download the source, you get a lot of example scripts. One of them is called rasterize.js, and this is exactly what we need.

$ phantomjs rasterize.js 
Usage: rasterize.js URL filename [paperwidth*paperheight|paperformat] [zoom]
  paper (pdf output) examples: "5in*7.5in", "10cm*20cm", "A4", "Letter"
  image (png/jpg output) examples: "1920px" entire page, window width 1920px
                                   "800px*600px" window, clipped to 800x600

Example #1:

phantomjs rasterize.js http://raphaeljs.com/polar-clock.html clock.png


Example #2:

phantomjs rasterize.js https://www.reddit.com/ red.png

It produced an image with dimension 600×3304. It’s too narrow, let’s fix that.

Example #3:

phantomjs rasterize.js https://www.reddit.com/ red.png 1024px

Its dimension is 1024×2432. Looks much better.

Example #4:
The previous image was too high. Let’s take a photo of that part that would be visible on our screen. For this we need to clip a window.

phantomjs rasterize.js https://www.reddit.com/ red.png "1024px*768px"

Great. Now scale it down to get a thumbnail.

Scaling down an image to thumbnail size

$ phantomjs rasterize.js https://www.reddit.com/ screenshot.png "1024px*768px"
$ convert -resize 250 screenshot.png thumb.jpg

The command convert comes from the ImageMagick package. Here we resize the image to width 250px. Convert will keep the image ratio, i.e. it figures out the height value.

Links

convert .wmv to .mp4

ffmpeg -i input.wmv -c:v libx264 -crf 23 -profile:v high -r 30 -c:a libfaac -q:a 100 -ar 48000 output.mp4

More info here.

Categories: bash Tags: , , , ,

open a terminal in a specified folder

Problem
From a script I want to launch a terminal that opens in a specified folder.

Solution
Thanks to #linux, I got two solutions:

1) I use konsole, which has an option for this:

konsole --workdir /tmp

It will open a terminal in the /tmp directory.

2) Change directory to the specified folder and launch the terminal:

cd /tmp; konsole

If you want to do it from a script, the idea remains the same.


If you want to execute a command automatically in the opening terminal, use this:

konsole -e mc

Find more info here.

Categories: bash, linux Tags: , ,
Follow

Get every new post delivered to your Inbox.

Join 81 other followers