Install a Greasemonkey script from a local file

Problem
You have downloaded a Greasemonkey script but… how to install it?

Solution
Here I suppose you have the Greasemonkey add-on installed (Hey, what is GM?). Well, I didn’t find anywhere the option “install from local file“. A GM script is called *.user.js and if you open the URL (http://...) of such a script, GM will recognize it and offer to install it (for this GM must be enabled). However! If you open your script locally (file://...), nothing happens. WTF?

Calm down. You know Python, right? The Swiss army knife of programmers. Just navigate to the directory where the GM script is located and start a web server:

$ python -m SimpleHTTPServer

Now open the URL http://localhost:8000 , click on the user script and GM will offer to install it. Python, what would I do without you?

User scripts are installed in this directory: “~/.mozilla/firefox/xxxxxxxx.default/gm_scripts“. Here you will find a config.xml file too that is created by GM.

Example
I installed this way this script: Greasemonkey: focus first input field. This script puts the focus on the first input field. I use it with Amazon and IMDB and works like a charm.

Automate tasks with AutoKey

“AutoKey is a desktop automation utility for Linux and X11. It allows you to create scripts and assign hotkeys to these scripts, allowing you to execute them on demand in whatever program you are using.” (source)

I’ve already done similar automatizations with autopy, which is not difficult, but life is much easier with AutoKey. AutoKey is a framework for these kinds of scripts and the great news is you can program it with Python!

Links

• AutoKey HQ
• Autokey: Make Your Own Keyboard Shortcuts In Linux
• Using Autokey Scripts to Automate Your Linux Desktop
• AutoKey : Linux Utility for Text Substitution , Hotkeys and Desktop Automation (excellent tutorial)
• AutoKey API reference

Use Case
Now let’s see a concrete use case. Printing at my workplace is a bit painful. If I want to send jobs to a powerful printer here, I need to authenticate myself each time (see the figure; click for a larger image). Or, I can use another (shitty) printer…

I must provide my username / password, then click on the OK button. If I print a large document, it’s OK, but if I want to print several small files, it becomes a pain in the ass.

And here is where AutoKey comes to the rescue. I made a simple script that automates this task (click to enlarge):

There is a hotkey associated to this script: CTRL+F1. There is also a window filter, thus it cannot be activated by accident in a wrong environment.

The script is here:

with open(".../print.txt") as f:
    username = f.readline().rstrip("\n")
    password = f.readline().rstrip("\n")

keyboard.send_keys("<shift>+<tab>")
keyboard.send_keys(username)
keyboard.send_keys("<tab>")
keyboard.send_keys(password)
for _ in range(3):
    keyboard.send_keys("<tab>")
keyboard.send_keys("<enter>")
# END

KeePassX + TrueCrypt + Dropbox: a secure and portable password management solution

Problem
I’ve arrived at the point that I’m fed up with the f* passwords. I can’t memorize them all so I usually write them in an exercise book that I keep at home. But what if I need something from it at my workplace? On the other hand, this booklet is already full (with other pieces of info too), so when I need a password from it, I need to search it for minutes… Damn. It would be so nice if I had all this information in a file on my machine but in a secure way.

Solution
The ideal solution is a password manager. But which one to choose? There are a lot. Since I also use Windows from time to time, I needed a cross-platform solution. First I thought of using a command line manager but finally I decided to use a graphical one; after all it looks nicer and easier to use (and I didn’t want to learn new command line options that I forget if I don’t use it for a few weeks…). This is how I got to KeePassX, which perfectly fulfills my needs. It’s also in the Ubuntu repos.

As I use several machines, the password database should be available everywhere. So let’s store it on Dropbox. But how safe is it? Well, it’s rather safe; your KeePassX database has a master password, which uses an AES-256 encryption but still… the devil never sleeps. Could we add an extra layer of security?

Yes, we could. With TrueCrypt you can create an encrypted file that can be mounted as a new volume (as if you had attached a USB stick for instance). I put the KeePassX database on this volume. Thus, in order to use the database, first I must mount the container file as a TrueCrypt volume, and then I can open the database file, but it also asks for the master password. Now I dare put the TrueCrypt container file on Dropbox :)

So, here is my setup (summary):

• Create a KeePassX database and provide a master password. You can change this password later under the File menu. It uses AES-256 encryption.
• Create a container file with TrueCrypt. The KeePassX database is very small so I set the container’s size to 1 MB. Encryption algorithm: AES-Twofish-Serpent cascading encryption with the XTS method. Hash algorithm: Whirlpool (tip from here). Of course, use a different password for this container file than for the KeePassX database. The TrueCrypt password should be long (20 to 30+ characters).
• Mount the container file and move the KeePassX database on the mounted volume.

OK. So far so good. But how to use the database painlessly? I made a simple script that mounts the container file and then opens the database. Just customize the constants in the header part. Launch it and simply type in the passwords. Instead of one password (for the database), you will have to provide two extra ones (for the TrueCrypt volume and your root password for being able to mount a new volume). I think this sacrifice is worth considering the additional security you gain. It may be a bit paranoid but on the Internet be paranoid. You know: Trust is a weakness :)

#!/usr/bin/env python

"""
Start KeePassX.
Mount the truecrypt container if necessary.

by Jabba Laci 2013 (jabba.laci@gmail.com)

http://ubuntuincident.wordpress.com/2013/04/14/keepassx-truecrypt-dropbox/

"""

import os

TRUECRYPT = '/usr/bin/truecrypt'
KEEPASSX = '/usr/bin/keepassx'
#
CONTAINER_FILE = "{home}/Dropbox/keepassx/container.dat".format(
    home=os.path.expanduser('~')
)
MOUNT_POINT = '/media/truecrypt9' 
KDB = '/media/truecrypt9/JabbaDB.kdb'

def mount_truecrypt_file():
    """
    Open the truecrypt container file that 
    includes the keepassx database.
    """
    if not os.path.isfile(KDB):
        cmd = 'sudo {tc} {container} {mount}'.format(
            tc=TRUECRYPT, container=CONTAINER_FILE, mount=MOUNT_POINT
        )
        print '#', cmd
        os.system(cmd)
    else:
        print '# container already mounted to', MOUNT_POINT

def open_kdb():
    """
    Open the keepassx database file on the previously mounted volume.
    """
    if not os.path.isfile(KDB):
        print "Error: the container file was not mounted."
    else:
        cmd = "{kpx} {f} &".format(kpx=KEEPASSX, f=KDB)
        print '#', cmd
        os.system(cmd)

def main():
    mount_truecrypt_file()
    open_kdb()

###################################################################

if __name__ == "__main__":
    main()

_{[ comments @reddit ]}

Update (20130501)
After two weeks of usage, I think adding truecrypt is an overkill. The problem is the following: I want to use this keepassx database on several machines, that’s why I put it on dropbox. That’s fine. But each time I need to mount the truecrypt volume that I often forget to dismount. At my workplace my machine is always on, so sometimes (often) I leave the volume mounted when I go home. If I want to add a new password to the database at home, dropbox creates a conflicted copy of the truecrypt file when I save the keepassx database. So I end up with two different databases that I will have to merge manually. It’s already happened to me 2 or 3 times…

So I removed truecrypt from the chain. Now I have a keepassx database (with a long password) stored on dropbox. I only have to pay attention to close keepassx when I leave my workplace but it’s feasible: when I copy a password from it, I close it immediately.

Get a TrashMail.net disposable email address from the command line

Problem
You want access to a website but it asks you to register. Damn. You don’t want to provide your real email address; you need a disposable (temporary) address.

Solution
I love simple solutions. All I want is a command that creates a disposable email address for me and prints this new address on the standard output.

There are lots of services where you can create temporary email addresses. I tried a few but several of them didn’t work or I didn’t like them. TrashMail.net is quite nice but I don’t want to use their web interface each time. I wanted a command line solution.

On trashmail.net I found a bash script (trashmail_sdk_1_1.tar.gz) that does the job. However, some time ago I promised I would never touch bash scripts again. So I rewrote it in Python. The project can be found here: https://github.com/jabbalaci/TrashMail.net-disposable-email-address.

Usage:

./trashmail.py

Output:

# copied to the clipboard
prcb107f@trashmail.net

_{[ comments @reddit ]}

Nikola: a static site and blog generator

“Nikola is a static website and blog generator. The very short explanation is that it takes some texts you wrote, and uses them to create a folder full of HTML files. If you upload that folder to a server, you will have a rather full-featured website, done with little effort.“

Links:

• Nikola home page
• handbook
• source @github

Automate a Windows application from Linux using Python

Problem
For my everyday work I use Linux. I also have a Windows 7 in VirtualBox. Under Windows I’ve found a very nice movie catalogue called Movie Collector. It’s not free though; unfortunately I didn’t find any good open-source alternative for this task.

So, I had this software running in VirtualBox and I wanted to input a list of movies. I collected the list in a file. Movie Collector has a GUI interface where you can insert the title of a movie and it’s put in a queue (see the figure below). Then you can verify for each item if it found the correct movie.

The question is: how to insert the movie titles via the GUI interface in an automated fashion? I won’t type 100 titles by hand…

Solution
It’s a perfect job for Python. There is a great module called autopy that includes functions for controlling the keyboard and mouse. I already had it under Linux. Fortunately, you can launch an autopy script under Linux and it can interact with a Windows GUI that runs in VirtualBox.

The idea is simple: read the input file, where there is a movie title in each line. Put the movie title on the clipboard (we are lucky again, if you put something on the clipboard under Linux, it’s available under Windows too in VirtualBox). Click in the Title text field, select everything with Ctrl+A, press Del to delete the content, paste from the clipboard and click on the Add to Queue button. Then repeat: take the next movie title, etc.

The following script does exactly this:

#!/usr/bin/env python

from autopy import key
from time import sleep
from jabbapylib.mouse import mouse
from jabbapylib.clipboard.clipboard import text_to_clipboards

TITLE = (68, 124)    # input text field's position
ADD = (575, 123)     # button's position

def delete():
    mouse.click_to(TITLE)
    key.tap('a', key.MOD_CONTROL)    # Ctrl+A
    key.tap(key.K_DELETE)            # Del
    sleep(.1)

def paste():
    mouse.click_to(TITLE)
    key.tap('v', key.MOD_CONTROL)    # Ctrl+V
    sleep(.1)

def add():
    mouse.click_to(ADD)
    sleep(.1)

def main():
    cnt = 0
    with open('movies.txt') as f:
        for title in f:
            title = title.rstrip('\n')
            text_to_clipboards(title)

            delete()
            paste()
            add()
            cnt += 1

    print cnt

#############################################################################

if __name__ == "__main__":
    print 'Switch to VirtualBox...'
    sleep(3)
    main()

The imported modules are here:

• mouse.py
• clipboard.py

To figure out the position of GUI elements (to know where to click to), you can use this little helper.

Extracting Colors

At http://99designs.com/tech-blog/blog/2012/05/11/color-analysis/ I found an interesting project called “colorific”. “colorific determines what the most important colors used in your image are, and if one of them is a background color.” So it won’t list all the colors, just the most important ones.

Links

• blog post
• @github

Installation:

sudo pip install colorific

Let’s take the following image: ares.png.

Usage #1:

$ colorific ares.png 
ares.png        #0065b9,#bbd6ec,#ff0000,#2d4456,#77adda

Usage #2:

$ colorific -o ares.png 
ares.png        #0065b9,#bbd6ec,#ff0000,#2d4456,#77adda

This one, in addition to usage #1, creates a file called “ares_palette.png”, which shows the extracted colors:

Scraping AJAX web pages (Part 4)

Don’t forget to check out the rest of the series too!

I managed to solve a problem that bugged me for a long time. Namely, (1) I want to download the generated source of an AJAX-powered webpage; (2) I want a headless solution, i.e. I want no browser window; and (3) I want to wait until the AJAX-content is fully loaded.

During the past 1.5 years I got quite close :) I could solve everything except issue #3. Now I’m proud to present a complete solution that satisfies all the criteria above.

#!/usr/bin/env python

import os
import sys

from PySide.QtCore import *
from PySide.QtGui import *
from PySide.QtWebKit import QWebPage

SEC = 1000 # 1 sec. is 1000 msec.
USER_AGENT = 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/20100101 Firefox/17.0'

class JabbaWebkit(QWebPage):
    # 'html' is a class variable
    def __init__(self, url, wait, app, parent=None):
        super(JabbaWebkit, self).__init__(parent)
        JabbaWebkit.html = ''

        if wait:
            QTimer.singleShot(wait * SEC, app.quit)
        else:
            self.loadFinished.connect(app.quit)

        self.mainFrame().load(QUrl(url))

    def save(self):
        JabbaWebkit.html = self.mainFrame().toHtml()

    def userAgentForUrl(self, url):
        return USER_AGENT

def get_page(url, wait=None):
    # here is the trick how to call it several times
    app = QApplication.instance() # checks if QApplication already exists
    if not app: # create QApplication if it doesnt exist
        app = QApplication(sys.argv)
    #
    form = JabbaWebkit(url, wait, app)
    app.aboutToQuit.connect(form.save)
    app.exec_()
    return JabbaWebkit.html

#############################################################################

if __name__ == "__main__":
    url = 'http://simile.mit.edu/crowbar/test.html'
    print get_html(url)

It’s also on GitHub. The GitHub version contains more documentation and more examples.

_{[ reddit comments ]}

Update (20121228)
Jabba-Webkit got included in Pycoder’s Weekly #46. Awesome.

Image Gallery from a list of URLs

Problem
I have several scrapers that extract images. How to visualize them? One way is to open each one in a new browser tab but it’s slow and who wants to have several hundreds of tabs? Is there a way to browse these images in one single tab?

Solution
A primitive solution would be to create an HTML page that lists all the images one below the other. But again, what if you have lots of images?

A better way is to organize the images in a gallery. There are tons of image gallery generators out there but most of them work with local images. I want to browse remote images when only their URLs are available. So I made my own image gallery generator that works with URLs. Available on github.

There is also a live demo, check it out.

The software is written in Python. See the README file for usage examples.

[Python] Open a URL in Firefox in the *current* tab

Python is f*ing awesome. I realize it every single day. Here is the issue that I faced today:

Problem
From a Python script, I wanted to open a URL in the current tab of Firefox. How to do that?

Difficulty
In the standard library there is a module called webbrowser. However, it only has these functions: open, open_new, and open_new_tab, i.e. you can open a URL in a new window or in a new tab but not in the current tab!

Solution
After doing some research, I found a Firefox add-on (MozRepl) that allows you to program Firefox and other Mozilla-based applications from the inside. Here are some links:

• https://github.com/bard/mozrepl/wiki
• video demonstration
• https://addons.mozilla.org/en-US/firefox/addon/mozrepl/

Install the add-on, then go to Tools -> MozRepl -> Start. Here check the option “Activate on startup” too. Now fire up a terminal and try these commands:

$ telnet localhost 4242
Trying 127.0.0.1...
Connected to localhost.
...
repl> content.location.href = 'http://www.python.org'
repl> repl.quit()
Connection closed by foreign host.
$

Check the browser. You have http://www.python.org opened in your current tab. Great! But how to automate these steps from a script? How to automate a telnet session?

Automating telnet
Needless to say, there is a telnet module in the standard library :) Using it, I came up with the following solution:

#!/usr/bin/env python

# firefox.py

import telnetlib

HOST = 'localhost'
PORT = 4242

def open_curr_tab(url):
    tn = telnetlib.Telnet(HOST, PORT)
    cmd = "content.location.href = '{url}'".format(url=url)

    tn.read_until("repl> ")
    tn.write(cmd + "\n")

    tn.write("repl.quit()\n")

#############################################

if __name__ == "__main__":
    open_curr_tab('http://www.python.org')

If you put it in a module called firefox, then you can call it like this: firefox.open_curr_tab(url). Pretty awesome, I think.

Related
For closing the current tab, use gBrowser.removeCurrentTab() .

Links
I also looked at these links but I didn’t use them at the end:

• pexpect (wiki)
• pexpect (pypi)

Update (20121211)
I’ve made a Python module out of it. Available here as part of my jabbapylib library. Currently implemented methods:

• open_curr_tab(url) : open URL in current tab
• open_new_empty_tab() : open a new empty tab and put the focus on it
• close_curr_tab() :close the current tab
• is_mozrepl_installed() : test if MozRepl is installed