Archive

Archive for the ‘python’ Category

Dr. Strangelove or: How I Learned to Stop Worrying and Love Bash

July 30, 2014 1 comment

In this post, Steve shows how to include Python code snippets in Bash scripts! Do you hate Bash? Me too :) However, with this trick bash becomes much less painful.

For more info, please refer to Steve’s post. Here I write a short summary for future references.

Example 1:

#!/bin/bash

function current_datetime {
python - <<END
import datetime
print datetime.datetime.now()
END
}

# Call it
current_datetime

# Call it and capture the output
DT=$(current_datetime)
echo Current date and time: $DT

Example 2 (passing parameters to the wrapping function):


#!/bin/bash

function line {
PYTHON_ARG="$1" python - <<END
import os
line_len = int(os.environ['PYTHON_ARG'])
print '-' * line_len
END
}

# Do it one way
line 80

echo 'Handy'

# Do it another way
echo $(line 80)

You can find another example with JSON at Steve’s blog.

Categories: bash, python Tags:

Firefox: restore your lost tabs

April 30, 2014 Leave a comment

Problem
Over the last 1.5-2 years, I collected 700+ tabs in my Firefox :) Maybe this summer I will have some time to sort them out. However, today when I switched my computer on, all my tabs were gone and I got a clean Firefox instance with one tab only. Hmm… I had a similar problem once and then I installed an add-on called “Session Manager”. In this add-on I made the setting to offer the list of previous sessions upon restart but it didn’t do anything! Damn, how to get back my tab collection?

Solution
In the .mozilla directory there is a file called sessionstore.js that stores — among others — the opened tabs. However, this file was very small, my previous tabs were clearly not in it. Thank God there was a backup copy of this file next to it called sessionstore.bak. It was a big file and the timestamp of the file indicated that it was created 2 days ago when everything was OK with my tabs.

So, how to extract the old tabs from sessionstore.bak?

This is a JSON file, but it’s not pretty printed. I suggest copying this file to somewhere else where you can experiment with it. First, let’s make it readable:

$ python -m json.tool sessionstore.bak > session.json

Now you can open session.json with a text editor. You will find lines with a “url” key, but the number of these rows is huge. I had 731 tabs (that I lost) but this file contained 6500+ URLs. As I noticed, it also contains the URLs of closed tabs. How to extract the URLs of the opened tabs only?

Again, Python came to my rescue. After analyzing the structure of this JSON file, I could extract the tab URLs the following way:

$ python  # version 2.7
>>> import json
>>> f = open('session.json')  # input file
>>> g = open('tabs.txt', 'w')  # output file
>>> d = json.load(f)
>>> tabs = d["windows"][0]["tabs"]
>>> cnt = 0
>>> for t in tabs:
...     print >>g, t["entries"][0]["url"]
...     cnt += 1
>>> cnt
731    # Yeah! All of them are here!
>>> g.close()
>>> f.close()

The URLs of the lost tabs are now in the tabs.txt file.

I didn’t make a script of it but feel free to do it. From now on I will make regular backups of my opened tabs with the URL Lister add-on.

Categories: firefox, python Tags: , , , ,

ménage de printemps (spring cleaning)

March 22, 2014 Leave a comment

Problem
My Dropbox folder was at 98.5%, so it was time to do some cleanup. Which directories are the largest? Which files are the largest?

Solution

alias top10dirs='du -hsx * | sort -rh | head -10'
alias top10files='find . -type f -print0 | du -h --files0-from=- | sort -hr | head -n 10'

The first one shows the top 10 largest directories, while the second one prints the top 10 largest files. Directory and file sizes are shown in a human-readable format.

Samples

$ top10dirs 
60M     20090629-deploy
60M     20090327-deploy
56M     kgm
55M     exist-deploy-v3-20100710
55M     exist-deploy-v3-20100521
...
==============================
$ top10files 
60M     ./20090629-deploy/20090629-deploy.zip
60M     ./20090327-deploy/20090327-deploy.zip
55M     ./exist-deploy-v3-20100710/exist-deploy-v3-20100710.zip
55M     ./exist-deploy-v3-20100521/exist-deploy-v3-20100521.zip
49M     ./exist-deploy-v3-20100409/exist-deploy-v3-20100409.zip
...

Credits

  • top10dirs is from here
  • for top10files I wrote a Python script, but reddit user farsass pointed out that it can be solved easier in the shell

koremutake

February 4, 2014 Leave a comment

Koremutake is an encoding algorithm that can convert any large, unsigned number into a memorable sequence of phonetically unique syllables.

Links

Example

>>> import koremutake
>>> koremutake.encode(10610353957)
'koremutake'
>>> koremutake.decode('koremutake')
10610353957

If anyone knows why it is called “koremutake”, let me know :) I guess the answer is hidden in the number 10610353957.

Categories: golang, python Tags:

docker-redis

January 14, 2014 Leave a comment

Problem
You want to build a docker image with a Redis server in it. When you launch it, you want it to function as a redis server.

Dockerfile, build, run

Dockerfile:

# Redis
#
# VERSION       0.1

# use the ubuntu base image provided by dotCloud
FROM ubuntu

#MAINTAINER Jabba Laci, jabba.laci@gmail.com

# make sure the package repository is up to date
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
RUN apt-get update

# install redis
RUN apt-get install -y redis-server

# Launch redis-server when launching the container
ENTRYPOINT ["redis-server"]

# expose redis-server port
EXPOSE 6379

build:

$ docker build -t redis .

Here I suppose that the file Dockerfile is in the current dorectory.

run:

$ docker run -d -p 6379 redis

Run it in the background (-d) and expose its port 6379 to the host (-p).

Try it

docker ps will tell you the necessary port numbers:

$ docker ps
...PORTS...
...0.0.0.0:49163->6379/tcp...

Meaning: the port 49163 on the host is redirected to the container’s port 6379. That is, if we want to connect to the redis-server in the container, we can do it through the host’s port 49163.

Of course, in your case it will be something else instead of port number 49163…

Test code:

>>> import redis
>>> r = redis.Redis('localhost', 49163)
>>> r.info()
{'pubsub_channels': 0, 'aof_enabled': 0, 'bgrewriteaof_in_progress': 0, 'connected_slaves': 0, 'uptime_in_days': 0, 'multiplexing_api': 'epoll', 'lru_clock': 558312, 'last_save_time': 1389703273, 'redis_version': '2.2.12', 'redis_git_sha1': 0, 'loading': 0, 'connected_clients': 1, 'keyspace_misses': 0, 'used_memory': 798784, 'vm_enabled': 0, 'used_cpu_user_children': 0.0, 'role': 'master', 'total_commands_processed': 0, 'used_memory_rss': 1662976, 'total_connections_received': 1, 'pubsub_patterns': 0, 'used_cpu_sys': 0.03, 'used_memory_human': '780.06K', 'used_cpu_sys_children': 0.0, 'blocked_clients': 0, 'used_cpu_user': 0.02, 'use_tcmalloc': 0, 'client_biggest_input_buf': 0, 'arch_bits': 64, 'redis_git_dirty': 0, 'expired_keys': 0, 'hash_max_zipmap_entries': 512, 'evicted_keys': 0, 'bgsave_in_progress': 0, 'client_longest_output_list': 0, 'hash_max_zipmap_value': 64, 'process_id': 1, 'uptime_in_seconds': 167, 'changes_since_last_save': 0, 'mem_fragmentation_ratio': 2.08, 'keyspace_hits': 0}
>>> # r.info() worked, so it's OK
... 
>>> r.set('Jabba', 'the Hutt')
True
>>> r.get('Jabba')
'the Hutt'
>>>
Categories: docker, python Tags: ,

docker: installation and getting started

January 10, 2014 1 comment

Update (20140610): Docker 1.0 has come out. The installation has changed since I wrote this post. Please refer to https://docs.docker.com/installation/ubuntulinux/ for up-to-date information.


Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. (source)

Docker extends a common container format called Linux Containers (LXC) with a high-level API providing a lightweight virtualization solution that runs processes in isolation. Docker utilizes LXC, cgroups, and the Linux kernel itself. Unlike traditional virtual machines, a Docker container does not include a separate operating system, instead it relies on the operating system’s functionality provided by the underlying infrastructure. (via wikipedia)

In the following, I want to concentrate on how to install Docker and how to make it run on Ubuntu. To learn more about Docker (what is it good for, what can it do for you, etc.), you will have to do some research on your own. But I also plan to make more blog posts on Docker where I want to write about its use cases.

This entry is based on this post, written by Csaba Okrona. For more info check out his excellent post.


At the time of writing docker was not supported on 32 bit machines. Installing the prerequisites and docker on x86_64 machines:

$ sudo apt-get install linux-image-extra-$(uname -r) software-properties-common
$ sudo sh -c "wget -qO- https://get.docker.io/gpg | apt-key add -"
$ sudo sh -c "echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
$ sudo apt-get update
$ sudo apt-get install lxc-docker

Let’s try it. Pull down an ubuntu base image:

$ sudo docker pull ubuntu
Pulling repository ubuntu
...

Let’s execute a command inside docker:

$ sudo docker run ubuntu /bin/echo hello from docker
hello from docker

The echo command was executed in the docker container, it produced an output, and this output was returned to the host (your machine).

Execute docker as non-root
As you noticed, docker must be executed with sudo. If you want to execute it with your non-root account, here is what to do:

# create the 'docker' group
$ sudo groupadd docker  # probably it already exists
# add your user to the 'docker' group
$ sudo gpasswd -a USER docker  # replace USER with your user name
# restart the service
$ sudo service docker restart

You will have to log out and log back in to make the group settings active on your account. After that you can launch docker without sudo:

$ docker images
REPOSITORY          TAG                 ID                  CREATED             SIZE
ubuntu              12.04               8dbd9e392a96        4 months ago        131.5 MB (virtual 131.5 MB)
ubuntu              12.10               b750fe79269d        5 months ago        24.65 kB (virtual 180.1 MB)
ubuntu              latest              8dbd9e392a96        4 months ago        131.5 MB (virtual 131.5 MB)
ubuntu              precise             8dbd9e392a96        4 months ago        131.5 MB (virtual 131.5 MB)
ubuntu              quantal             b750fe79269d        5 months ago        24.65 kB (virtual 180.1 MB)

Connect to a container

# start a container (and save its ID in an env. variable)
$ DOCK=$(docker run -d -i -t ubuntu /bin/bash)
# it's in the list
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
fe650a3d214f        ubuntu:12.04        /bin/bash           2 minutes ago       Up 2 minutes                            sharp_archimede
# attach to the container
$ docker attach $DOCK
root@fe650a3d214f:/# 

Install some packages in the container

In order to install arbitrary packages, let’s enable the multiverse repos:

root@fe650a3d214f:/# cat > /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu precise main universe multiverse
root@fe650a3d214f:/# apt-get update
root@fe650a3d214f:/# apt-get install python-pip
root@fe650a3d214f:/# pip install virtualenv
root@fe650a3d214f:/# apt-get install libpq-dev python-dev
root@fe650a3d214f:/# apt-get install vim
...

However! Any changes you do in your container, they are not permanent! That is, if you log out and you attach to it again, all your changes will be lost and you get back the same vanilla container that you saw upon your first connect. But don’t worry! Changes can be committed, i.e. changes can be made permanent.

This behaviour is actually useful. You can do anything in the container, you can try anything inside, you can’t f* it up. Just log out and log back in, and everything is back. Imagine the possibilities: you can build a secure sandboxed environment easily where you can run any untrusted code.

Committing changes

You made some changes in your vanilla ‘ubuntu’ container and you want to commit them, i.e. you want to see these changes after a relogin. Let’s commit the changes to a new container called ‘pyenv’ for instance.

You are connected to ‘ubuntu’ in a terminal. Don’t logout, otherwise you lose your changes. Open a new terminal and execute these commands:

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
fe650a3d214f        ubuntu:12.04        /bin/bash           22 minutes ago      Up 22 minutes                           sharp_archimede
$ docker commit fe650a3d214f pyenv
f538ae2eb2562bf2ab669d7e04cf9fa49b66814d871bcbc0662deedfdcce73ab

Now you can log out of ‘ubuntu’ and you can log in to ‘pyenv’. From now on you can commit new changes to ‘pyenv’. If you need a new vanilla container, just fork ‘ubuntu’ again.

When you commit changes, docker stores just the diffs, and when you connect to a container, it puts the necessary diffs together.

Sharing a folder between the host and the container

If you want to copy files from the host to the container, the easiest way is to create a shared folder:

$ DOCK=$(docker run -d -i -t -v /tmp/shared.folder:/shared.folder ubuntu /bin/bash)
$ docker attach $DOCK

The folder /tmp/shared.folder on the host will be mounted in the container as /shared.folder . The shared folder must exist on the host.

Categories: python, ubuntu Tags: ,

The quick brown fox jumps over the lazy dog

December 20, 2013 1 comment

I’m sure you have seen the sentence “The quick brown fox jumps over the lazy dog” lots of times. I always thought it was just a stupid test sentence, but no! It’s a special sentence, a so-called pangram. A pangram is a sentence that contains all the letters of the alphabet. Here is the proof:

>>> s='The quick brown fox jumps over the lazy dog'.lower()
>>> s
'the quick brown fox jumps over the lazy dog'
>>> sorted(set(s))
[' ', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']
Categories: fun, python Tags:
Follow

Get every new post delivered to your Inbox.

Join 72 other followers