Home > Uncategorized > Download files with wget from sites that verify your user-agent

Download files with wget from sites that verify your user-agent

Problem
You want to download a file from a given site with your favourite wget utility but you get a “403 Forbidden” error in your face. Of course, everything works from your browser. What to do?

Solution
If it works from the browser but it fails with wget, then the site must check your user-agent. If it sees “User-Agent: Wget/1.12 (linux-gnu)” (version may vary), then it simply blocks you.

But don’t fear for a second. Simply fake a different user agent with wget and continue downloading.

Solution 1:

wget --user-agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0" http://host/file.jpg

Solution 2:
If you don’t want to provide a user agent each time, put the following your ~/.wgetrc file:

# custom .wgetrc file
user_agent = Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0

Then:

wget http://host/file.jpg
Advertisements
  1. September 2, 2015 at 19:02

    For python wget, you impersonate a different User-Agent like this:

    USER_AGENT_FIREFOX_11 = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0"
    urllib.URLopener.version = USER_AGENT_FIREFOX_11
    

    Thanks to http://stackoverflow.com/a/32155139/673991

  2. February 11, 2016 at 20:12

    Thanks, I was needing this do download the Webmin key.

  1. No trackbacks yet.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: