Archive

Archive for the ‘python’ Category

JSON Path

May 20, 2017 1 comment

I wrote a command-line program that outputs the full path of every key / value in a JSON file.

Example

$ ./json_path.py sample.json
root.a => 1
root.b.c => 2
root.b.friends[0].best => Alice
root.b.friends[1].second => Bob
root.b.friends[2][0] => 5
root.b.friends[2][1] => 6
root.b.friends[2][2] => 7
root.b.friends[3][0].one => 1
root.b.friends[3][1].two => 2

More information at the project’s github page.

Categories: bash, python Tags: ,

sending e-mails from your app.

If you want to send e-mails from your application, check out https://www.mailgun.com/. You can register for free and with your free account you can also send a good number of e-mails. And if you pay for it, you can send thousands of mails if you want. Their service can be used from Bash, Python, PHP, etc.

On my VPS I used a command-line solution (see here) that sends e-mails with sendmail. Unfortunately, Gmail treats these mails as spam, so it’s not appropriate for sending messages to other people. Mailgun is a better solution.

Categories: api, bash, python Tags: ,

Java profiling

March 18, 2017 Leave a comment

YourKit is a great Java profiling tool. It is a commercial software but you can get a free 15-day evaluation license key for a fully functional version of the profiler.

The past week I was working on a Java project and the software was very slow. Using YourKit I could easily find the bottleneck. It turned out that 94% of the time was spent in a function that I implemented in a naive way. Choosing a better algorithm the software got much faster. A profiler is really useful…

For Java I use Eclipse. YourKit integrates in Eclipse perfectly through an Eclipse plugin.

Python profiling

Here is an excellent post about Python profiling.

Categories: java, python Tags: , ,

grab a Twitch video in mp3

January 17, 2017 Leave a comment

Problem
You want to grab a Twitch video in mp3. For instance, you want to listen to it offline.

Solution
You need two programs for it: youtube-dl and ffmpeg. Let’s take a concrete example:

$ youtube-dl -g "https://www.twitch.tv/wearethevr/v/115335579"
https://vod067-ttvnw.akamaized.net/v1/AUTH_system/vods_c631/wearethevr_24261824064_585034506/chunked/index-dvr.m3u8
$ ffmpeg -i "https://vod067-ttvnw.akamaized.net/v1/AUTH_system/vods_c631/wearethevr_24261824064_585034506/chunked/index-dvr.m3u8" -f mp3 out.mp3

Where https://www.twitch.tv/wearethevr/v/115335579 is the URL of this particular Twitch video.

I wrote a script for it to automate the whole process: twitch2mp3.

Categories: bash, python Tags: , ,

Detailed Twitter info in JSON: an undocumented feature

October 24, 2016 Leave a comment

Problem
Using a script, I wanted to figure out the number of my followers on Twitter. Here is my (mostly abandoned) Twitter page: https://twitter.com/szathmar . I didn’t want to use any API since I didn’t want to register for an API key so I went on the easy way: let’s scrape the necessary data out :) Digging in the HTML code I found the number of followers, but I also found a hidden treasure!

Solution
And the hidden treasure is a long json string that contains all kinds of information about a twitter user:

hidden_json2

Here on the screenshot you can see just an extract, the json string is much longer. Fine, let’s get it!

#!/usr/bin/env python3
# coding: utf-8

import json
import readline
import sys
from pprint import pprint

import requests
from bs4 import BeautifulSoup

def main():
    url = input("Full twitter URL: ")
    html = requests.get(url).text
    soup = BeautifulSoup(html, "lxml")

    tag = soup.find('input', {'class': 'json-data'})
    j = tag['value']
    d = json.loads(j)
    json_out = json.dumps(d, indent=4)
    print(json_out)

    # followers = d['profile_user']['followers_count']
    # print(followers)

##############################################################################

if __name__ == "__main__":
    main()

If you want the number of followers for instance, then uncomment the last two lines.

Thank you Twitter! It’s really nice of you to provide all these data in JSON!

Sample
The JSON that I could extract from my page is 743 lines long! Here is an extract of it:

...
"profile_image_url": "http://pbs.twimg.com/profile_images/459783802395430912/vcMT0CGX_normal.png",
"business_profile_state": "none",
"url": null,
"profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme6/bg.gif",
"screen_name": "szathmar",
"is_translator": false,
"friends_count": 123,
"followers_count": 70,
"profile_text_color": "333333",
"profile_link_color": "FF3300",
"translator_type": "none",
"profile_background_color": "709397",
...
Categories: python Tags: , , ,

[wordpress] using the old-style editor

October 16, 2016 Leave a comment

Problem
Wordpress.com introduced a while ago a new-style editor for writing posts. However, I really hate it, it’s unusable. How to get back to the old-style editor?

Solution
Transforming the URL you can get back to the old-style editor. For instance:

new style: https://wordpress.com/post/ubuntuincident.wordpress.com/5865
old style: https://ubuntuincident.wordpress.com/wp-admin/post.php?post=5865&action=edit

Let’s automate the task with Python:

#!/usr/bin/env python3
# coding: utf-8

import readline
import webbrowser

def main():
    url = input("New style URL: ")
    parts = url.split("/")
    new = "{0}//{1}/wp-admin/post.php?post={2}&action=edit".format(
        parts[0], parts[4], parts[-1]
    )
    print("New style:", new)
    webbrowser.open_new_tab(new)

##############################################################################

if __name__ == "__main__":
    main()

Screenshots

New-style shit.

New-style shit.

Old-style goodie.

Old-style goodie.

purge a reddit account

August 2, 2016 Leave a comment

Problem
You have a reddit account that you want to empty, i.e. delete all the posts and comments you have made.

Solution
Use Shreddit. It deletes a limited number of posts/comments in a session, so you may have to re-run it several times. When it cannot remove anything, then it’s done.

Categories: python Tags: ,