Archive

Archive for the ‘python’ Category

purge a reddit account

August 2, 2016 Leave a comment

Problem
You have a reddit account that you want to empty, i.e. delete all the posts and comments you have made.

Solution
Use Shreddit. It deletes a limited number of posts/comments in a session, so you may have to re-run it several times. When it cannot remove anything, then it’s done.

Categories: python Tags: ,

Jinja2-like template for PHP

August 1, 2016 Leave a comment

Problem
My primary language is Python. When I need to do a simple webpage or a REST API, I use Flask with its built-in Jinja2 template engine.

However, I started to work on a project with some friends and our UI developer chose PHP for the frontend. As I also want to contribute to the UI, I looked around the PHP template engines if there is someting similar to Jinja2.

Solution
It turned out that Jinja2 was ported to PHP! It’s called Twig and it’s almost the same. So if you use Flask, Twig is a natural choice for PHP.

There are also several MVC frameworks for PHP but I don’t use any (yet?). I have a PHP file (the controller), and a corresponding HTML file (the view, i.e. the template). Let’s see a simple example:

index.php:

<?php
require_once 'vendor/twig/lib/Twig/Autoloader.php';
Twig_Autoloader::register();

$loader = new Twig_Loader_Filesystem('templates');
$twig = new Twig_Environment($loader, array(
    // 'cache' => 'compilation_cache',
));

$context = array(
    'name' => 'Twig',
);

echo $twig->render('index.html', $context);
?>

index.html:

Hello {{ name }}!

It will print the text “Hello Twig!” to the screen.

What happens? The index.php file is the controller. Here you collect all the data that you want to print in the resulting HTML output. These data are put in a hash table (dictionary), and it’s passed to the template file index.html.

You can enable the cache in the index.php file. In this case the view will be “compiled” to a PHP file, making it faster. However, during the development you’d better switch it off. As I noticed, when I change the source code, the cache is not always updated automatically. So if you enable the cache and change the source, don’t forget to purge the cache.

Project layout
My project structure looks like this:

.
├── compilation_cache
├── index.php
├── templates
│   └── index.html
└── vendor
    └── twig
        └── lib
            └── Twig
                └── Autoloader.php
                └── ... (other files of the Twig template engine)

For security reasons, I think it’s a good idea to move the “vendor” folder somewhere else that is not accessible via the http protocol. That is, if your project is served from your “~/public_html” folder, move the “vendor” folder outside of “~/public_html“. I’m not sure but it may be true for the “compilation_cache” folder too.

Categories: php, python Tags: , ,

Scraping AJAX web pages (Part 5.5)

July 13, 2016 Leave a comment

Don’t forget to check out the rest of the series too!

This post is very similar to the previous one (Part 5), which scraped a webpage using PhantomJS from the command line and sent the output to the stdout.

This time we use PhantomJS again, but we do it from a Python script and wrap Selenium around PhantomJS. The generated HTML source will be available in a variable. Here is the source:

#!/usr/bin/env python3
# encoding: utf-8

"""
required packages:
* selenium
optional packages:
* bs4
* lxml
"""

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
# from bs4 import BeautifulSoup

url = "http://simile.mit.edu/crowbar/test.html"

dcap = dict(DesiredCapabilities.PHANTOMJS)
dcap["phantomjs.page.settings.userAgent"] = (
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/53 "
    "(KHTML, like Gecko) Chrome/15.0.87"
)
driver = webdriver.PhantomJS(desired_capabilities=dcap)
driver.get(url)
html = driver.page_source
print(html)
# soup = BeautifulSoup(driver.page_source, "lxml") #page_source fetches page after rendering is complete
# driver.save_screenshot('screen.png') # save a screenshot to disk

driver.quit()

The script sets the user agent (optional but recommended). The source is captured in a variable. The last two lines are in comments but they would work. You could feed the source to BeautifulSoup and then you could extract part of the HTML source. If you uncomment the last line, then you can create a screenshot of the webpage.

email notification from a script

June 15, 2016 Leave a comment

Problem
I have a Digital Ocean VPS box where several scripts are running. Some of them run for a day. I would like to get an email notification when a particular script starts / ends, or when something happens.

In short: how to send an email from the command line?

Solution
First, do the necessary configuration to be able to send emails from the command line (more details here).

Sending email without a body:

mailx -s "subject" < /dev/null "to@email.com" 2>/dev/null

Sending email with a body:

echo 'this is the body of the email' | mailx -s "subject" "to@email.com" 2>/dev/null

I also made a Python wrapper for it that you can find here.

Categories: bash, python Tags: ,

[vim] run current file with Python

Problem
You use (neo)vim for editing your Python code and you want to execute the source code in your editor. The output of the script should appear in the editor.

Solution
I came up with a dynamic solution, i.e. the interpreter is taken from the first line of the code. If you specified “#!/usr/bin/env python2“, then python2 is used; if you have “#!/usr/bin/env python3“, then python3 is used.

But what if you use Anaconda and you have for instance “#!/opt/anaconda3/bin/python3” in the first line? Then simply this interpreter is used.

Here is the snippet from my config file:

" run python script {{{
    function! RunWithPython()
        let first = getline(1)
        let first = substitute(first, "^#!", "", "")
        let first = substitute(first, "\n", "", "")
        let exe = ""    " the Python binary to call

        if first =~ "/usr/bin/env "
            let exe = split(first)[-1]
        elseif first == "/opt/anaconda3/bin/python3"
            let exe = first
        endif
        if exe == ""
            echo "Error: unknown Python interpreter in the first line."
            return
        endif
        " echo exe
        echo system(exe . " " . expand('%'))
    endfunction

    au FileType python nnoremap <buffer> <F9> :call RunWithPython()<cr>
" }}}

If you want to use Anaconda, then simply customize line 10.

Categories: python, vim Tags:

setting the volume from the command line

Problem
I have a laptop where the default volume is weak. On the system tray the volume is on 100% but it’s still weak. So far I started the program “pavucontrol”, which is a GUI application, and there I set the volume to 150% (that’s the maximum). However, if I watch a youtube video and pause it, in pavucontrol the volume falls back to 100%, so I need to adjust it after each pause.

Solution
I found a command line program that allows one to set the volume. It’s independent of pavucontrol. So I made a script that runs automatically when the graphical interface comes up:

#!/usr/bin/env bash

#
# from http://askubuntu.com/questions/44680
# listing current volume:
#
#     pacmd list-sinks | grep volume
#

cmd="pacmd set-sink-volume 0 100000"
echo "#" $cmd
$cmd

First list your sinks with “pacmd list-sinks”. I only had one, hence the id 0. The value 100000 is the volume (on my system it’s equivalent to 153%). Value 65535 is volume 100%.

Tip from here.

Update (20160604)
I made a wrapper script around pacmd; you can find it here on GitHub. Its usage is very simple. Do you want to increase the volume? Just call “volume.py 140%” and you are done.

drawing trees / graphs easily

Problem
You want to draw a binary tree / graph quickly. You’ve already used graphviz but you forgot its DOT language and you don’t want to read its documentation again. What do you do?

Solution
I found a nice Python project that does exactly this: pygraph.

Command to execute:

$ pygraph -u -e neato circle ab bc cd de ea

Output (circle.png):
circle

See the project’s page for more examples and figures.

I sent a request to the author and he was kind to implement the --dot option that prints the .dot source in a file. This way you can easily tweak your graph. Don’t write the .dot file from scratch: do it quickly with pygraph, get a basic source, and refine it manually if necessary.

Categories: python Tags: , , ,
Follow

Get every new post delivered to your Inbox.

Join 91 other followers