How to download files in parallel?

October 10, 2017 1 comment

Problem
So far I’ve mostly used wget to download files. You can collect the URLs in a file (one URL per line), and pass it to wget: “wget -i list.txt“. However, wget fetches them one by one, which can be time consuming. Is there a way to parallelize the download?

Solution
Use Aria2 for this purpose. It’s similar to wget, but it spawns several threads and starts to download the files in parallel. The number of worker threads has a default value, but you can also change that. Its basic usage is the same:

aria2c -i list.txt
Advertisements
Categories: bash Tags: , , , ,

How to extract .7z files?

October 10, 2017 Leave a comment

Problem
You have a .7z archive and you want to extract it.

Solution

7za e archive.7z
Categories: bash Tags: ,

How to randomize the lines of a file?

October 10, 2017 Leave a comment

Problem
You have a text file and you want to mix up its lines, i.e. you want to randomize the order of lines.

Solution

sort --random-sort input.txt
Categories: bash Tags: ,

Exctract the significant parts of a web page

October 1, 2017 Leave a comment

Problem
From a web page you want to extract the significant parts: title, author, date of publication, body, etc.

Solution
Mercury Web Parser does exactly this. It’s free. After registration you get an API key. Their web service returns a structured JSON response. I tried it with my previous post:

curl -H "x-api-key: <my_api_key>" "https://mercury.postlight.com/parser?url=https://ubuntuincident.wordpress.com/2017/10/01/re-run-a-command-in-the-terminal-every-x-seconds/" | python3 -m json.tool

Output:

{
    "title": "Re-run a command in the terminal every X\u00a0seconds",
    "author": "Jabba Laci",
    "date_published": "2017-09-30T22:02:19.000Z",
    "dek": null,
    "lead_image_url": "https://secure.gravatar.com/blavatar/db6c398dc21dc8e8f82d7bc83130c0ab?s=200&ts=1506809436",
    "content": "<div class=\"content\"> <p><strong>Problem</strong><br>\nYou want re-execute a command in the terminal every X seconds. For instance, you copy a lot of big files to a partition and you want to monitor the size of the free space on that partition.</p>\n<p><strong>Solution</strong><br>\nA naive and manual approach to the problem mentioned above is to execute the commands “<code>clear; df -h</code>” regularly, say every 2 seconds.</p>\n<p>A better way is to use the command “<code>watch</code>“. Usage:</p>\n<pre> watch -n 2 df -h </pre>\n<p>That is: execute “<code>df -h</code>” every two seconds. <code>watch</code> will also clear the screen and print the result to the top. You can quit with <code>Ctrl + c</code>.</p>\n<p>Tip from <a href=\"https://askubuntu.com/questions/430382/repeat-a-command-every-x-interval-of-time-in-terminal\">here</a>.</p> </div>",
    "next_page_url": null,
    "url": "https://ubuntuincident.wordpress.com/2017/10/01/re-run-a-command-in-the-terminal-every-x-seconds/",
    "domain": "ubuntuincident.wordpress.com",
    "excerpt": "Problem You want re-execute a command in the terminal every X seconds. For instance, you copy a lot of big files to a partition and you want to monitor the size of the free space on that partition.\u2026",
    "word_count": 108,
    "direction": "ltr",
    "total_pages": 1,
    "rendered_pages": 1
}

Pretty impressive.

Categories: web Tags: , ,

Re-run a command in the terminal every X seconds

October 1, 2017 Leave a comment

Problem
You want re-execute a command in the terminal every X seconds. For instance, you copy a lot of big files to a partition and you want to monitor the size of the free space on that partition.

Solution
A naive and manual approach to the problem mentioned above is to execute the commands “clear; df -h” regularly, say every 2 seconds.

A better way is to use the command “watch“. Usage:

watch -n 2 df -h

That is: execute “df -h” every two seconds. watch will also clear the screen and print the result to the top. You can quit with Ctrl + c.

Tip from here.

Categories: bash Tags: ,

screenkey

September 30, 2017 Leave a comment

screenkey is a screencast tool to display your keys, thus your viewers can see exactly what keys you pressed.

Here is a screenshot:

And here is a short youtube video about it from gotbletu: link.

Categories: linux Tags: ,

Setting the number of open files

September 29, 2017 Leave a comment

Problem
I installed Neo4j and when I started the server, I got a warning message: “Max 1024 open files allowed, minimum of 40 000 recommended.”

How to increase the number of open files?

Solution
With “ulimit -n” you can check the current value of the number of open files. By default, it’s 1024. To increase this value, follow these steps:

Open /etc/security/limits.conf and add these lines:

root   soft    nofile  40000
root   hard    nofile  40000
jabba  soft    nofile  40000
jabba  hard    nofile  40000

Here, instead of “jabba” use your account name. I use my Linux box under the user name “jabba”.

Open /etc/pam.d/su and add this line:

session    required   pam_limits.so

Restart your machine so that these changes take effect. Re-run “ulimit -n“; now it should display 40000.

Categories: bash Tags: , ,