Archive

Archive for the ‘python’ Category

ménage de printemps (spring cleaning)

March 22, 2014 Leave a comment

Problem
My Dropbox folder was at 98.5%, so it was time to do some cleanup. Which directories are the largest? Which files are the largest?

Solution

alias top10dirs='du -hsx * | sort -rh | head -10'
alias top10files='find . -type f -print0 | du -h --files0-from=- | sort -hr | head -n 10'

The first one shows the top 10 largest directories, while the second one prints the top 10 largest files. Directory and file sizes are shown in a human-readable format.

Samples

$ top10dirs 
60M     20090629-deploy
60M     20090327-deploy
56M     kgm
55M     exist-deploy-v3-20100710
55M     exist-deploy-v3-20100521
...
==============================
$ top10files 
60M     ./20090629-deploy/20090629-deploy.zip
60M     ./20090327-deploy/20090327-deploy.zip
55M     ./exist-deploy-v3-20100710/exist-deploy-v3-20100710.zip
55M     ./exist-deploy-v3-20100521/exist-deploy-v3-20100521.zip
49M     ./exist-deploy-v3-20100409/exist-deploy-v3-20100409.zip
...

Credits

  • top10dirs is from here
  • for top10files I wrote a Python script, but reddit user farsass pointed out that it can be solved easier in the shell

koremutake

February 4, 2014 Leave a comment

Koremutake is an encoding algorithm that can convert any large, unsigned number into a memorable sequence of phonetically unique syllables.

Links

Example

>>> import koremutake
>>> koremutake.encode(10610353957)
'koremutake'
>>> koremutake.decode('koremutake')
10610353957

If anyone knows why it is called “koremutake”, let me know :) I guess the answer is hidden in the number 10610353957.

Categories: golang, python Tags:

docker-redis

January 14, 2014 Leave a comment

Problem
You want to build a docker image with a Redis server in it. When you launch it, you want it to function as a redis server.

Dockerfile, build, run

Dockerfile:

# Redis
#
# VERSION       0.1

# use the ubuntu base image provided by dotCloud
FROM ubuntu

#MAINTAINER Jabba Laci, jabba.laci@gmail.com

# make sure the package repository is up to date
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
RUN apt-get update

# install redis
RUN apt-get install -y redis-server

# Launch redis-server when launching the container
ENTRYPOINT ["redis-server"]

# expose redis-server port
EXPOSE 6379

build:

$ docker build -t redis .

Here I suppose that the file Dockerfile is in the current dorectory.

run:

$ docker run -d -p 6379 redis

Run it in the background (-d) and expose its port 6379 to the host (-p).

Try it

docker ps will tell you the necessary port numbers:

$ docker ps
...PORTS...
...0.0.0.0:49163->6379/tcp...

Meaning: the port 49163 on the host is redirected to the container’s port 6379. That is, if we want to connect to the redis-server in the container, we can do it through the host’s port 49163.

Of course, in your case it will be something else instead of port number 49163…

Test code:

>>> import redis
>>> r = redis.Redis('localhost', 49163)
>>> r.info()
{'pubsub_channels': 0, 'aof_enabled': 0, 'bgrewriteaof_in_progress': 0, 'connected_slaves': 0, 'uptime_in_days': 0, 'multiplexing_api': 'epoll', 'lru_clock': 558312, 'last_save_time': 1389703273, 'redis_version': '2.2.12', 'redis_git_sha1': 0, 'loading': 0, 'connected_clients': 1, 'keyspace_misses': 0, 'used_memory': 798784, 'vm_enabled': 0, 'used_cpu_user_children': 0.0, 'role': 'master', 'total_commands_processed': 0, 'used_memory_rss': 1662976, 'total_connections_received': 1, 'pubsub_patterns': 0, 'used_cpu_sys': 0.03, 'used_memory_human': '780.06K', 'used_cpu_sys_children': 0.0, 'blocked_clients': 0, 'used_cpu_user': 0.02, 'use_tcmalloc': 0, 'client_biggest_input_buf': 0, 'arch_bits': 64, 'redis_git_dirty': 0, 'expired_keys': 0, 'hash_max_zipmap_entries': 512, 'evicted_keys': 0, 'bgsave_in_progress': 0, 'client_longest_output_list': 0, 'hash_max_zipmap_value': 64, 'process_id': 1, 'uptime_in_seconds': 167, 'changes_since_last_save': 0, 'mem_fragmentation_ratio': 2.08, 'keyspace_hits': 0}
>>> # r.info() worked, so it's OK
... 
>>> r.set('Jabba', 'the Hutt')
True
>>> r.get('Jabba')
'the Hutt'
>>>
Categories: docker, python Tags: ,

docker: installation and getting started

January 10, 2014 Leave a comment

Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. (source)

Docker extends a common container format called Linux Containers (LXC) with a high-level API providing a lightweight virtualization solution that runs processes in isolation. Docker utilizes LXC, cgroups, and the Linux kernel itself. Unlike traditional virtual machines, a Docker container does not include a separate operating system, instead it relies on the operating system’s functionality provided by the underlying infrastructure. (via wikipedia)

In the following, I want to concentrate on how to install Docker and how to make it run on Ubuntu. To learn more about Docker (what is it good for, what can it do for you, etc.), you will have to do some research on your own. But I also plan to make more blog posts on Docker where I want to write about its use cases.

This entry is based on this post, written by Csaba Okrona. For more info check out his excellent post.


At the time of writing docker was not supported on 32 bit machines. Installing the prerequisites and docker on x86_64 machines:

$ sudo apt-get install linux-image-extra-$(uname -r) software-properties-common
$ sudo sh -c "wget -qO- https://get.docker.io/gpg | apt-key add -"
$ sudo sh -c "echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
$ sudo apt-get update
$ sudo apt-get install lxc-docker

Let’s try it. Pull down an ubuntu base image:

$ sudo docker pull ubuntu
Pulling repository ubuntu
...

Let’s execute a command inside docker:

$ sudo docker run ubuntu /bin/echo hello from docker
hello from docker

The echo command was executed in the docker container, it produced an output, and this output was returned to the host (your machine).

Execute docker as non-root
As you noticed, docker must be executed with sudo. If you want to execute it with your non-root account, here is what to do:

# create the 'docker' group
$ sudo groupadd docker  # probably it already exists
# add your user to the 'docker' group
$ sudo gpasswd -a USER docker  # replace USER with your user name
# restart the service
$ sudo service docker restart

You will have to log out and log back in to make the group settings active on your account. After that you can launch docker without sudo:

$ docker images
REPOSITORY          TAG                 ID                  CREATED             SIZE
ubuntu              12.04               8dbd9e392a96        4 months ago        131.5 MB (virtual 131.5 MB)
ubuntu              12.10               b750fe79269d        5 months ago        24.65 kB (virtual 180.1 MB)
ubuntu              latest              8dbd9e392a96        4 months ago        131.5 MB (virtual 131.5 MB)
ubuntu              precise             8dbd9e392a96        4 months ago        131.5 MB (virtual 131.5 MB)
ubuntu              quantal             b750fe79269d        5 months ago        24.65 kB (virtual 180.1 MB)

Connect to a container

# start a container (and save its ID in an env. variable)
$ DOCK=$(docker run -d -i -t ubuntu /bin/bash)
# it's in the list
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
fe650a3d214f        ubuntu:12.04        /bin/bash           2 minutes ago       Up 2 minutes                            sharp_archimede
# attach to the container
$ docker attach $DOCK
root@fe650a3d214f:/# 

Install some packages in the container

In order to install arbitrary packages, let’s enable the multiverse repos:

root@fe650a3d214f:/# cat > /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu precise main universe multiverse
root@fe650a3d214f:/# apt-get update
root@fe650a3d214f:/# apt-get install python-pip
root@fe650a3d214f:/# pip install virtualenv
root@fe650a3d214f:/# apt-get install libpq-dev python-dev
root@fe650a3d214f:/# apt-get install vim
...

However! Any changes you do in your container, they are not permanent! That is, if you log out and you attach to it again, all your changes will be lost and you get back the same vanilla container that you saw upon your first connect. But don’t worry! Changes can be committed, i.e. changes can be made permanent.

This behaviour is actually useful. You can do anything in the container, you can try anything inside, you can’t f* it up. Just log out and log back in, and everything is back. Imagine the possibilities: you can build a secure sandboxed environment easily where you can run any untrusted code.

Committing changes

You made some changes in your vanilla ‘ubuntu’ container and you want to commit them, i.e. you want to see these changes after a relogin. Let’s commit the changes to a new container called ‘pyenv’ for instance.

You are connected to ‘ubuntu’ in a terminal. Don’t logout, otherwise you lose your changes. Open a new terminal and execute these commands:

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
fe650a3d214f        ubuntu:12.04        /bin/bash           22 minutes ago      Up 22 minutes                           sharp_archimede
$ docker commit fe650a3d214f pyenv
f538ae2eb2562bf2ab669d7e04cf9fa49b66814d871bcbc0662deedfdcce73ab

Now you can log out of ‘ubuntu’ and you can log in to ‘pyenv’. From now on you can commit new changes to ‘pyenv’. If you need a new vanilla container, just fork ‘ubuntu’ again.

When you commit changes, docker stores just the diffs, and when you connect to a container, it puts the necessary diffs together.

Sharing a folder between the host and the container

If you want to copy files from the host to the container, the easiest way is to create a shared folder:

$ DOCK=$(docker run -d -i -t -v /tmp/shared.folder:/shared.folder ubuntu /bin/bash)
$ docker attach $DOCK

The folder /tmp/shared.folder on the host will be mounted in the container as /shared.folder . The shared folder must exist on the host.

Categories: python, ubuntu Tags: ,

The quick brown fox jumps over the lazy dog

December 20, 2013 1 comment

I’m sure you have seen the sentence “The quick brown fox jumps over the lazy dog” lots of times. I always thought it was just a stupid test sentence, but no! It’s a special sentence, a so-called pangram. A pangram is a sentence that contains all the letters of the alphabet. Here is the proof:

>>> s='The quick brown fox jumps over the lazy dog'.lower()
>>> s
'the quick brown fox jumps over the lazy dog'
>>> sorted(set(s))
[' ', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']
Categories: fun, python Tags:

take a screenshot and upload to imgur

November 5, 2013 Leave a comment

Problem
I want to grab a part of the screen by selecting a rectangle. I wanto to be able to refine the selection (resize the rectangle) like in Gimp. Then upload the selected image to imgur.

Solution
I found a great tool for this task called Nanoshot.

Read more about Nanoshot in this thread.

Download Nanoshot from here.

Note that I had to install the package python-wnck (via apt-get) to make it run.

Categories: python, ubuntu Tags: ,

HTML: add syntax-highlight to textarea

September 26, 2013 Leave a comment

Problem
You have an HTML form with a textarea where you want to accept some source code. You want to turn this simple textarea into a fancy input area that adds syntax highlighting.

Solution
I tried EditArea, and it suits my needs. See this SO page for more alternatives.

EditArea

EditArea is a free javascript editor for source code. This editor is designed to edit souce code files in a textarea. The main goal is to allow text formatting, search and replace and real-time syntax highlight (for not too heavy text).” (source)

For Python support, I had to add these lines to the HTML source:

<script language="javascript" type="text/javascript" src="../editarea/edit_area/edit_area_full.js"></script>
<script language="javascript" type="text/javascript">
editAreaLoader.init({
    id : "src_input"       // textarea id
    ,syntax: "css"          // syntax to be uses for highgliting
    ,start_highlight: true      // to display with highlight mode on start-up
    ,syntax: "python"
    ,replace_tab_by_spaces: 4
});
</script>

Related work

  • Ace (it seems a more professional solution)

alias with an argument

September 21, 2013 Leave a comment

Problem
In bash, you want to write an alias that receives an argument.

Solution
Well, you can’t do that in bash. But there is a workaround: call a bash function, and the function can treat the argument.

Let’s see a concrete example. While working with Python virtual environments, I wanted to be able to activate a virtual env. easily with the command “workon <DIR>“, where “workon” would be an alias and “<DIR>” is the name of a directory. (Detailed post is here.)

Add the following lines to your ~/.bashrc:

func_workon()
{
if [[ -z "$1" ]]
then
    echo "Usage: workon <venv>"
else
    . $1/bin/activate
fi
}
alias workon=func_workon
Categories: bash, python Tags: ,

redis: getting started

August 12, 2013 Leave a comment

Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.” (source)

Also, redis commands are atomic operations.

Installation

# Check out the home page for the latest version!
$ wget http://redis.googlecode.com/files/redis-2.6.14.tar.gz
$ tar xvzf redis-2.6.14.tar.gz
$ cd redis-2.6.14
$ make
$ sudo make install

Then start the redis server:

$ redis-server

The server, by default, will listen on localhost:6379.

Graphical interface
Do you like phpMyAdmin? Well, there is something similar to redis too called Redis Commander. Check out the project’s home page for installation instructions.

Redis Commander is written in node.js. For installing node.js, refer to this post: Installing Node.js and NPM on Ubuntu/Debian.

Python binding
At this point you might ask the natural question: “OK, but how can I use it from Python?”. You need the following client:

sudo pip install redis -U

Let’s test it:

>>> import redis
>>> r = redis.Redis()    # default: localhost, port 6379
>>> r.set("name", "jabba")
True
>>> r.get("name")
'jabba'

Let’s see another example with a list:

>>> import redis
>>> r = redis.Redis()
# the list is called "test"
# rpush: right push, i.e. put an element on its right side (tail)
>>> r.rpush("test", 24)
1L
>>> r.rpush("test", 67)
2L
>>> r.rpush("test", 9)
3L
# list all the elements (-1 is the index of the last element)
>>> r.lrange("test", 0, -1)
['24', '67', '9']
# number of elements 
>>> r.llen("test")
3
# delete the list if you don't need it anymore
>>> r.delete("test")
1

Save the database
It is possible to save the database to disk. When you restart the server, it can restore the database from disk.

The database will be saved to the directory where you started redis! For instance, you started redis from your HOME directory. In a Python script you called “r.save()“, which creates a snapshot of the database and dumps it to “~/dump.rdb“. When you start the server, it checks for this dump file in the current directory. If “./dump.rdb” is found, it will be loaded.

If you run redis-server in a terminal, check out its log. It will indicate if it found and loaded a database upon startup.

Links

Categories: python Tags: ,

PrimCom: manage and access your personal knowledge base easily

August 4, 2013 Leave a comment

I am proud to announce my newest project called PrimCom, which is a personal knowledge base manager, primarily made for programmers. I made a detailed description of the project here, I don’t want to repeat the whole text here.

A screenshot:

Links

Update (20130808)
PrimCom was included in Python Weekly #99. Great!

Related work

  • cheat (It was designed to help remind *nix system administrators of options for commands that they use frequently, but not frequently enough to remember.)
Categories: python Tags: ,
Follow

Get every new post delivered to your Inbox.

Join 61 other followers