Live Music Archive
Librivox Free Audio
Metropolitan Museum
Cleveland Museum of Art
Internet Arcade
Console Living Room
Open Library
American Libraries
TV News
Understanding 9/11Stack Exchange Data Dump
Item Preview
logo-stackoverflow.png
Share or Embed This Item
This is an anonymized dump of all user-contributed content on the Stack Exchange network. Each site is formatted as a separate archive consisting of XML files zipped via 7-zip using bzip2 compression. Each site archive includes Posts, Users, Votes, Comments, Badges, Tags, PostHistory, and PostLinks. For complete schema information, see this meta post.
All user content contributed to the Stack Exchange network is cc-by-sa 4.0 licensed, intended to be shared and remixed. We even provide all our data as a convenient data dump.
License: https://creativecommons.org/licenses/by-sa/4.0/
But our cc-by-sa 4.0 licensing, while intentionally permissive, does require attribution:
Attribution — You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).
Specifically the attribution requirements are as follows:
License: https://creativecommons.org/licenses/by-sa/4.0/
But our cc-by-sa 4.0 licensing, while intentionally permissive, does require attribution:
Attribution — You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).
Specifically the attribution requirements are as follows:
- Visually display or otherwise indicate the source of the content as coming from the Stack Exchange Network. This requirement is satisfied with a discreet text blurb, or some other unobtrusive but clear visual indication.
- Ensure that any Internet use of the content includes a hyperlink directly to the original question on the source site on the Network (e.g., http://stackoverflow.com/questions/12345)
- Visually display or otherwise clearly indicate the author names for every question and answer used
- Ensure that any Internet use of the content includes a hyperlink for each author name directly back to his or her user profile page on the source site on the Network (e.g., http://stackoverflow.com/users/12345/username), directly to the Stack Exchange domain, in standard HTML (i.e. not through a Tinyurl or other such indirect hyperlink, form of obfuscation or redirection), without any “nofollow” command or any other such means of avoiding detection by search engines, and visible even with JavaScript disabled.
- Addeddate
- 2014-01-21 18:54:32
- Identifier
- stackexchange
comment
Reviews
(48)
Reviewer:
SakuraKawaii
-
favorite -
August 1, 2024
Subject: SE's archive.org data dump has been cancelled
Subject: SE's archive.org data dump has been cancelled
They're moving it onto their own infrastructure, as per
https://meta.stackexchange.com/q/401324
As an added bonus, the next data dump has been delayed ... until mid-august, whenever they get around to actually releasing the new system. To add insult to injury, the complete data dump can't be downloaded either anymore, so it has to be downloaded in 365 pieces manually.
Please go there and express your dissatisfaction with this decision. Unless someone else beats me to it, I'll be uploading the next data dump manually to archive.org. They're not killing the data dump this easily.
https://meta.stackexchange.com/q/401324
As an added bonus, the next data dump has been delayed ... until mid-august, whenever they get around to actually releasing the new system. To add insult to injury, the complete data dump can't be downloaded either anymore, so it has to be downloaded in 365 pieces manually.
Please go there and express your dissatisfaction with this decision. Unless someone else beats me to it, I'll be uploading the next data dump manually to archive.org. They're not killing the data dump this easily.
Reviewer:
zeyu Cui
-
favoritefavoritefavoritefavoritefavorite -
July 15, 2024
Subject: when will the latest version update
Subject: when will the latest version update
it seems that the Data Dump should release at June 30, 2024， however， there are still the version from April.
Is there any changes in schedule. ... https://meta.stackexchange.com/questions/396597/data-dumps-releases-timeline-updates-and-clarification .
Is there any changes in schedule. ... https://meta.stackexchange.com/questions/396597/data-dumps-releases-timeline-updates-and-clarification .
Reviewer:
Daniel Wagner
-
favoritefavoritefavoritefavoritefavorite -
May 5, 2024
Subject: Discovery Channel
Subject: Discovery Channel
Thank you.
Reviewer:
Stack Exchange
-
favoritefavoritefavoritefavoritefavorite -
March 3, 2024
Subject: Mapping site names to database/file names
Subject: Mapping site names to database/file names
For vonver, these are just cases where the database name, site name, and URL don't all match, for various historical reasons - mainly because changing
...
a URL (or adding an alias) is much easier than changing the name of an established database. The data dump is based on the database name. If you follow the below mapping, you'll see all 7 files you mentioned are here:
- writing -> writers.stackexchange.com
- video -> avp.stackexchange.com
- psychology -> cogsci.stackexchange.com
- alcohol -> beer.stackexchange.com
- communitybuilding -> moderators.stackexchange.com
- medicalsciences -> health.stackexchange.com
- mattermodeling -> materials.stackexchange.com
For a much more thorough listing of all sites, mapping between site name, URL, and database name, which apply to both SEDE and the data dump, see:
https://meta.stackexchange.com/q/359794/165455
You can also piece this together from Sites.xml:
https://ia904700.us.archive.org/6/items/stackexchange/Sites.xml
- writing -> writers.stackexchange.com
- video -> avp.stackexchange.com
- psychology -> cogsci.stackexchange.com
- alcohol -> beer.stackexchange.com
- communitybuilding -> moderators.stackexchange.com
- medicalsciences -> health.stackexchange.com
- mattermodeling -> materials.stackexchange.com
For a much more thorough listing of all sites, mapping between site name, URL, and database name, which apply to both SEDE and the data dump, see:
https://meta.stackexchange.com/q/359794/165455
You can also piece this together from Sites.xml:
https://ia904700.us.archive.org/6/items/stackexchange/Sites.xml
Reviewer:
vonver
-
favoritefavoritefavoritefavoritefavorite -
February 2, 2024
Subject: Great, some sites are missing
Subject: Great, some sites are missing
Some stackexchange sites are missing
Great source of valuable data. But as of 2024-02-02 comparing the list from (https://stackexchange.com/sites) with ... the list here the missing ones are
- writing.stackexchange.com
- video.stackexchange.com
- psychology.stackexchange.com
- alcohol.stackexchange.com
- communitybuilding.stackexchange.com
- medicalsciences.stackexchange.com
- mattermodeling.stackexchange.com
Great source of valuable data. But as of 2024-02-02 comparing the list from (https://stackexchange.com/sites) with ... the list here the missing ones are
- writing.stackexchange.com
- video.stackexchange.com
- psychology.stackexchange.com
- alcohol.stackexchange.com
- communitybuilding.stackexchange.com
- medicalsciences.stackexchange.com
- mattermodeling.stackexchange.com
Reviewer:
Louis Hurris
-
favoritefavorite -
September 4, 2023
Subject: Why the download velocity so slow?
Subject: Why the download velocity so slow?
I have tried many ways, including change numerous VPNs but still slow! Hovever just 1 months ago the speed is normal.
Reviewer:
Vahid887
-
favoritefavorite -
March 7, 2023
Subject: Not updated yet
Subject: Not updated yet
The lastest data is still for December.
Reviewer:
jesan2021
-
favoritefavoritefavoritefavorite -
February 21, 2023
Subject: stackoverflow.com-Posts.7z file is missing
Subject: stackoverflow.com-Posts.7z file is missing
I cannot find the stackoverflow.com-Posts.7z file. Am I missing something?
Reviewer:
Aleksandar Jeftić
-
October 10, 2022
Subject: When I try to unarchive it, I get error "Is not archive"
Subject: When I try to unarchive it, I get error "Is not archive"
Then I tried solutions from below and not working, not sure if I am missing something, or files are bad?
Reviewer:
mikeeus
-
favoritefavoritefavoritefavoritefavorite -
October 7, 2022
Subject: Clubmed you're a life saver
Subject: Clubmed you're a life saver
If anyone is struggling with opening the 7z files look at the review below. This beautiful angel saved me a massive headache trying to get these archives
...
unzipped.
Reviewer:
clubmed
-
October 6, 2022
Subject: Corrupt archive/data error issues
Subject: Corrupt archive/data error issues
I was running into the same corrupt archive issues that others have mentioned, regardless of the way in which I downloaded the files (tried browser, the
...
IA CLI utility, wget and the torrent file). Not sure what the issue was, but there were three lines being added to the beginning of the archives before the 7z archive header proper starts, that include the "Content-Disposition" header. I was able to fix the archives by opening the files in VIM in binary mode and deleting the first three lines, but it can also be done with a single line of bash:
FILE=FILENAME; head -3 $FILE | wc -c | xargs -I{} dd if=$FILE bs={} skip=1 conv=notrunc of=fixed-$FILE
Just replace FILENAME with the file you're working with and you should be good to go.
FILE=FILENAME; head -3 $FILE | wc -c | xargs -I{} dd if=$FILE bs={} skip=1 conv=notrunc of=fixed-$FILE
Just replace FILENAME with the file you're working with and you should be good to go.
Reviewer:
SkobelevIgor
-
favoritefavoritefavoritefavorite -
December 19, 2020
Subject: Variant of possible schema (pg & mysql )
Subject: Variant of possible schema (pg & mysql )
Here is a variant of possible schema: https://github.com/SkobelevIgor/stackexchange-xml-converter/tree/main/schema_example Hope that helps!
Reviewer:
jean_1mm
-
favoritefavoritefavoritefavoritefavorite -
July 22, 2020
Subject: Can someone tell me the table structure of Posts、Votes?
Subject: Can someone tell me the table structure of Posts、Votes?
Can someone tell me the table structure of Posts、Votes、Badegs、Comments、Users、Tags? thansks
Reviewer:
andreymal
-
October 1, 2019
Subject: License
Subject: License
Don't trust the description. The license is actually CC BY-SA 3.0 because Stack Exchange changed it without explicit permission from original authors.
...
Read more: https://meta.stackexchange.com/questions/333089
Reviewer:
Tim_Nagy
-
favoritefavoritefavoritefavoritefavorite -
September 25, 2019
Subject: Old dump
Subject: Old dump
I downloaded the Stack Overflow posts dump ('stackoverflow.com-Posts.7z') on 13 Sept 2019 which is lastly modified on 4 Sept 2019 according to https://archive.org.
...
However, I found the latest post in the dump with date 11-03-2018 which does not make sense to me. I am not sure what the modified date of the dump means?
I am looking forward to your replies.
I am looking forward to your replies.
Reviewer:
Habit Lin
-
favoritefavoritefavoritefavoritefavorite -
September 5, 2019
Subject: Meaning of some links in stackoverflow.com-PostLinks.7z.
Subject: Meaning of some links in stackoverflow.com-PostLinks.7z.
Can anyone tell me the meaning of those links whose type is 'Linked'?
For instance, if there is a 'Linked' type edge between question A and question B, ... what information can we get from this relationship? Does it mean that question A references question B? Or it means something else?
Thanks.
For instance, if there is a 'Linked' type edge between question A and question B, ... what information can we get from this relationship? Does it mean that question A references question B? Or it means something else?
Thanks.
Reviewer:
abhishekverma3007
-
favoritefavoritefavoritefavorite -
May 20, 2019
Subject: Spark project to review all stackoverflow data
Subject: Spark project to review all stackoverflow data
Reviewer:
hearrywang
-
favoritefavoritefavoritefavorite -
October 17, 2018
Subject: information about data
Subject: information about data
I want to know every type of attribute in stackoverflow.com-Comments.7z. I did not find the link of this specific information.
Reviewer:
jickerson
-
favoritefavoritefavorite -
July 29, 2018
Subject: No files available
Subject: No files available
Similar to the previous comment, I also am unable to download the files despite being logged in. Is the archive no longer available?
Reviewer:
RamtinYazdanian
-
February 20, 2018
Subject: Access restricted
Subject: Access restricted
Despite my having signed in, access to virtually all files (except maybe 5 of them) is restricted and I have no idea why.
Reviewer:
nederlandsmeisje
-
favoritefavoritefavoritefavoritefavorite -
February 20, 2018
Subject: XML file into Stata
Subject: XML file into Stata
Did some of you guys manage to get the XML files into Stata? I really want to know how I can tackle this.
Reviewer:
optimus9p
-
favorite -
February 19, 2018
Subject: Not available for download
Subject: Not available for download
The dataset cannot be downloaded even though I am logged into archive.org.
Reviewer:
artoog2
-
favorite -
November 5, 2017
Subject: Data Error
Subject: Data Error
I tried downloading this 4 times today and each time I'm getting a "Data Error" while extracting the Posts.xml file through 7zip. Can someone post the
...
checksum for the Posts archive?
Reviewer:
Muldones
-
favorite -
October 20, 2017
Subject: stackoverflow.com-PostLinks.7z corrupted
Subject: stackoverflow.com-PostLinks.7z corrupted
The file stackoverflow.com-PostLinks.7z seems to be corrupted. I can not unzip it...
Reviewer:
aNeutrino
-
favoritefavoritefavoritefavorite -
July 14, 2017
Subject: File size limit
Subject: File size limit
Hi :)
I can not use torrent (I am in Cuba)
The only option for us is to download file with wget.
However, when we try to download full stack exchange data ... dump we have a message:
"total size of requested files (44 GB) is too large for zip-on-the-fly"
Can I ask please to remove this limitation?
I can not use torrent (I am in Cuba)
The only option for us is to download file with wget.
However, when we try to download full stack exchange data ... dump we have a message:
"total size of requested files (44 GB) is too large for zip-on-the-fly"
Can I ask please to remove this limitation?
Reviewer:
Venkatesh Prasad
-
favoritefavoritefavoritefavoritefavorite -
April 21, 2017 (edited)
Subject: Great data set
Subject: Great data set
My first time dabbling with this data set and I am having fun. Thanks for using a simple line-by-line XML format :)
Reviewer:
JasonC3
-
favoritefavoritefavoritefavoritefavorite -
April 1, 2017
Subject: Documentation
Subject: Documentation
Being discussed at https://meta.stackexchange.com/questions/293127. A list of sites and their corresponding database filenames can be found there as well.
...
Also note: Info about sites and their corresponding file names can be indirectly obtained via the /sites API query (http://api.stackexchange.com/docs/sites): For every site, the file name will match either the primary URL of that site *or* the URL of one of that site's aliases. There are no exceptions beyond SO (whose dump is uniquely split among multiple files).
Sites that are very new relative to the archive date are not included in the dump.
Sites that are very new relative to the archive date are not included in the dump.
Reviewer:
Mooash
-
June 28, 2016
Subject: Torrent out of date
Subject: Torrent out of date
Looks like the torrent is out of date again, I'm stuck at 800MB whilst it continually tries to download files.
Reviewer:
vishal14
-
March 5, 2016
Subject: How to open posts.xml file?
Subject: How to open posts.xml file?
I have downloaded and extracted posts.xml file of stackoverflow. Size of the file is around 40 GB and I am not able to open it in xml editors. Can someone
...
please suggest how to open or parse this huge file?
Reviewer:
Greg Lindahl
-
favoritefavoritefavoritefavoritefavorite -
February 21, 2016 (edited)
Subject: .torrent is fixed
Subject: .torrent is fixed
The former limitation of 25 gigabytes for a torrent have been relaxed for this item, and the torrent is again working.
If you want to parse these large ... XML files, you need to use a streaming parser. I'm not surprised that an XML editor would have problems with a 40 gigabyte XML file! Most XML parsing software libraries have a streaming option.
If you want to parse these large ... XML files, you need to use a streaming parser. I'm not surprised that an XML editor would have problems with a 40 gigabyte XML file! Most XML parsing software libraries have a streaming option.
Reviewer:
shankar321
-
favorite -
January 11, 2016
Subject: What sense do we make of the files?
Subject: What sense do we make of the files?
I see over 300 files, whereas there are only 6 XML files (large ones) as far as I remember.
I cannot download the torrent version either - can someone ... help make sense of the files?
Shankar
I cannot download the torrent version either - can someone ... help make sense of the files?
Shankar
Reviewer:
jlewi
-
December 30, 2015
Subject: torrent out of date
Subject: torrent out of date
The bit torrent link:
https://archive.org/download/stackexchange/stackexchange_archive.torrent
appears to link to an older version of the data dump. For ... stackoverflow the latest post I saw was from 2014.
However, if the .zip file
https://archive.org/compress/stackexchange/formats=7Z&file=/stackexchange.zip
appears to have data upto august 2015.
https://archive.org/download/stackexchange/stackexchange_archive.torrent
appears to link to an older version of the data dump. For ... stackoverflow the latest post I saw was from 2014.
However, if the .zip file
https://archive.org/compress/stackexchange/formats=7Z&file=/stackexchange.zip
appears to have data upto august 2015.
Reviewer:
Jesse_W
-
November 6, 2015
Subject: This hit the shuffle.php bug
Subject: This hit the shuffle.php bug
And that broke the webseeds (see https://archive.org/post/1047899 ). This review should hopefully fix the issue.
Reviewer:
D1Doris
-
favoritefavoritefavoritefavorite -
September 4, 2015
Subject: stackexchange_meta.sqlite
Subject: stackexchange_meta.sqlite
StevenLJohnson, I have the file and can open it without any problems. What exactly do you mean by "unable to access"? If you mean that you don't have the
...
file, I can send it to you.
Reviewer:
StevenLJohnson
-
favoritefavoritefavorite -
August 31, 2015
Subject: Missing File
Subject: Missing File
I am unable to access the file stackexchange_meta.sqlite
Does anyone know of a source for this file?
Does anyone know of a source for this file?
Reviewer:
xcombelle
-
July 21, 2015
Subject: for those which can't make bittorent working
Subject: for those which can't make bittorent working
Reviewer:
timiblossom
-
favoritefavorite -
July 2, 2015
Subject: Bittorrent download broken
Subject: Bittorrent download broken
It stopped at around 70% for a couple of days and could never move forward.
Reviewer:
sathvik
-
favoritefavoritefavoritefavoritefavorite -
May 22, 2015 (edited)
Subject: Thanks for sharing
Subject: Thanks for sharing
Thanks for sharing the community data. It will greatly benefit research groups.
Reviewer:
dmpetrov
-
favoritefavoritefavoritefavoritefavorite -
May 3, 2015
Subject: April data
Subject: April data
Great data set. Thank you for sharing.
I see only March data. How can I get April data?
What about January and February?
Thanks,
Dmitry
I see only March data. How can I get April data?
What about January and February?
Thanks,
Dmitry
Reviewer:
alisa1
-
April 10, 2015
Subject: Resolved!
Subject: Resolved!
I also tried couple of times. It was failed at the same point.
But then I tried when I logged in, and I was able to download the whole file :-)
But then I tried when I logged in, and I was able to download the whole file :-)
Reviewer:
big_t_dub
-
favorite -
April 2, 2015 (edited)
Subject: 70%
Subject: 70%
stuck at 70.7% download complete via utorrent- arg!
this should be made avail via ftp!!!
this should be made avail via ftp!!!
Reviewer:
Ihor Bobak
-
March 27, 2015
Subject: File is broken
Subject: File is broken
At the top right corner of this page there is a link to zip archive. I've downloaded it twice (on different machines, in different countries). The file
...
was always broken.
Torrent stucks on 70.8%.
Can anyone help to get this file?
Torrent stucks on 70.8%.
Can anyone help to get this file?
Reviewer:
gnijuohz
-
favoritefavoritefavorite -
March 25, 2015
Subject: No seed?
Subject: No seed?
It stopped at around 70%.
Reviewer:
klitzkrieg
-
favoritefavoritefavorite -
March 24, 2015
Subject: Seeds for 3/16/15 version?
Subject: Seeds for 3/16/15 version?
Everybody's stuck at 70.7%
Reviewer:
Nemo_bis
-
favoritefavoritefavoritefavoritefavorite -
February 7, 2015
Subject: Thanks and tests
Subject: Thanks and tests
Thanks for the September update, eager to see the next one. Did someone try importing this data into a StackExchange instance?
Fun to see how small the ... whole SE network is after all, only few GB compressed. Wikimedia projects dumps compress very well too, but they're still much bigger (while fitting a common hard disk anyway!).
Fun to see how small the ... whole SE network is after all, only few GB compressed. Wikimedia projects dumps compress very well too, but they're still much bigger (while fitting a common hard disk anyway!).
Reviewer:
shamsazad
-
favoritefavoritefavoritefavorite -
August 19, 2014
Subject: Latest Dump.
Subject: Latest Dump.
When will be latest dump from stackoverflow will be posted over here.
Reviewer:
Jenson555
-
favoritefavoritefavoritefavoritefavorite -
July 26, 2014
Subject: Really Cool
Subject: Really Cool
This is an Awesome Stuff..Cheers..:)
There are 48 reviews for this item. .
392,340 Views
74 Favorites
DOWNLOAD OPTIONS
7Z
Uplevel BACK
1,022.0M
askubuntu.com.7z download
486.8M
mathoverflow.net.7z download
1,011.7M
ru.stackoverflow.com.7z download
819.8M
serverfault.com.7z download
IN COLLECTIONS
Unsorted Datasets
The Dataset Collection 
Uploaded by Stack Exchange on
