WikiLeaks CableGate (.txt format)

Secret US Embassy Cables -- FILE DESCRIPTION : -- First 219 Diplomatic Cables from http://cablegate.wikileaks.org The HTML pages from the torrent have been parsed and the *.txt files only content the cables themselves (yet sorted by date, classification, source ... into different directories). -- ABOUT THE CABLES : -- Wikileaks began on Sunday November 28th publishing 251,287 leaked United States embassy cables, the largest set of confidential documents ever to be released into the public domain. The documents will give people around the world an unprecedented insight into US Government foreign activities. The cables, which date from 1966 up until the end of February this year, contain confidential communications between 274 embassies in countries throughout the world and the State Department in Washington DC. 15,652 of the cables are classified Secret. The embassy cables will be released in stages over the next few months. The subject matter of these cables is of such importance, and the geographical spread so broad, that to do otherwise would not do this material justice. The cables show the extent of US spying on its allies and the UN; turning a blind eye to corruption and human rights abuse in "client states"; backroom deals with supposedly neutral countries; lobbying for US corporations; and the measures US diplomats take to advance those who have access to them. This document release reveals the contradictions between the US's public persona and what it says behind closed doors - and shows that if citizens in a democracy want their governments to reflect their wishes, they should ask to see what's going on behind the scenes. Every American schoolchild is taught that George Washington - the country's first President - could not tell a lie. If the administrations of his successors lived up to the same principle, today's document flood would be a mere embarrassment. Instead, the US Government has been warning governments - even the most corrupt - around the world about the coming leaks and is bracing itself for the exposures. The full set consists of 251,287 documents, comprising 261,276,536 words (seven times the size of "The Iraq War Logs", the world's previously largest classified information release). The cables cover from 28th December 1966 to 28th February 2010 and originate from 274 embassies, consulates and diplomatic missions. -- [From http://cablegate.wikileaks.org/index.html] -- Enjoy and share! . . . | | | |,---. ,-.-.,---.|__/ ,---. ,---.,---.,---.,---. | | ||---' | | |,---|| : |---' | || || | | `-'-'`---' ` ' '`---^` ` `---' |---'`---'` ` ' | Tetalab

udp://tracker.openbittorrent.com:80
udp://open.demonii.com:1337
udp://tracker.coppersurfer.tk:6969
udp://exodus.desync.com:6969

SB555 (2010-12-02)

If I may ask, how did you get these files to a .txt format?

Skinkie (2010-12-03)

Don't know how he did it but this is what I used:
import os
import sys
import re
from BeautifulSoup import BeautifulSoup
from BeautifulSoup import BeautifulStoneSoup
import re, htmlentitydefs
from os.path import join
def unescape(text):
def fixup(m):
text = m.group(0)
if text[:2] == ""&#":
# character reference
try:
if text[:3] == ""&#x":
return unichr(int(text[3:-1], 16))
else:
return unichr(int(text[2:-1]))
except ValueError:
pass
else:
# named entity
try:
text =
except KeyError:
pass
return text # leave as is
return re.sub(""&#?\w+;", fixup, text)
for root, dirs, files in os.walk('cable'):
for name in files:
if name.endswith('.html') and not os.path.exists(os.path.join(root, name.replace('.html', '.txt'))):
f = open(os.path.join(root, name), 'r')
soup = BeautifulSoup(''.join(f.read()),
f.close()
pre = soup.findAll("pre")
f = open(os.path.join(root, name.replace('.html', '.txt')), 'w')
for i in pre:
for j in i.contents:
try:
f.write(unescape(j))
except:
pass
f.close()