{"title":"stosb","link":[{"@attributes":{"href":"https:\/\/stosb.com\/","rel":"alternate"}},{"@attributes":{"href":"https:\/\/stosb.com\/feeds\/all.atom.xml","rel":"self"}}],"id":"https:\/\/stosb.com\/","updated":"2018-02-07T12:00:00+00:00","entry":[{"title":"Recovering Data From A Corrupt tar Archive","link":{"@attributes":{"href":"https:\/\/stosb.com\/blog\/recovering-data-from-a-corrupt-tar-archive\/","rel":"alternate"}},"published":"2018-02-07T12:00:00+00:00","updated":"2018-02-07T12:00:00+00:00","author":{"name":"Tom Hacohen"},"id":"tag:stosb.com,2018-02-07:\/blog\/recovering-data-from-a-corrupt-tar-archive\/","summary":"

A few days ago, a bug<\/a> in the task list app I use on my phone, Orgzly<\/a>, wiped all of my tasks. This was terribly annoying, but I wasn't too concerned because I knew I had backups. Well, at least I thought I had.<\/p>\n

I intend to write a post \u2026<\/p>","content":"

A few days ago, a bug<\/a> in the task list app I use on my phone, Orgzly<\/a>, wiped all of my tasks. This was terribly annoying, but I wasn't too concerned because I knew I had backups. Well, at least I thought I had.<\/p>\n

I intend to write a post about my phone's backup setup in the future, but in short, it's something like this: I turn on \"backup to file\" in apps that support it, use syncthing<\/a> to copy the files to my home server, and then use duplicity<\/a> to securely back everything up to a remote location. I have yet to find a simpler, secure, and privacy respecting backup solution for Android. If you have any suggestions, please let me know.<\/p>\n

After realising my task app wiped everything, I quickly checked the local file backup on my phone. Unfortunately, this had already synced and also got wiped.<\/p>\n

My syncthing is configured to only sync when charging, so I tried checking my home server, though unfortunately I was too late, and it was already synced and thus was empty as well.<\/p>\n

At least, I thought to myself, I have my duplicity backups, I can just restore the backup from there. I opened duplicity only to find out that it was not configured to backup the syncthing directory, meaning I had no backups there too!\nI occasionally verify my backups actually work (can restore from them), though I haven't noticed that this important directory was not included in the backup.<\/p>\n

At this point I was starting to get worried, thinking I'd lose all of my (many!) tasks, though then I realised that I upgraded my phone's firmware a few days before, and had a full NANDroid backup<\/a>. However, the files I was looking for were not showing up, and tar<\/code> was reporting an error. This is where our story begins.<\/p>\n

The Issue<\/h2>\n

As I said above, the files I was looking for weren't showing up (though some other files were), and tar<\/code> was reporting the following error:<\/p>\n

\ntar: Malformed extended header: missing equal sign\n<\/samp><\/p>\n

I tried searching the internet for this error message, but found no solutions, only what seems to be the same issue<\/a> reported by another user. After reading a bit more, it looks like TWRP<\/a> (the Android recovery I made the backup with), had a bug with creating backups in some cases. In my case it wasn't even affecting all of the files, just some, though the ones I cared about were among those.<\/p>\n

I then tried searching the web for corrupted tar recovery<\/q> and such similar terms. I found some suggestions and ideas on how to solve it, but nothing worked. I tried opening it with gnu tar<\/code>, bsdtar<\/code>, busybox<\/code> and cpio<\/code> to no avail. At this point I got a bit desperate and decided it's time to check if there's even data there. To do that, I used grep on my 2GB backup file (data.tar<\/samp>).<\/p>\n

<\/span>$ <\/span>grep \/data\/data\/com.orgzly data.tar\nBinary file data.tar matches<\/span>\n<\/code><\/pre><\/div>\n\n
I was excited to see that the files seem to be there, and to verify they actually contained my data, I opened the file with less<\/code>, searched the above string and looked around. My data was there!<\/p>\n
The Solution<\/h2>\n
I remembered that the tar archive format was quite simple, so I was initially planning on writing a small python program that would parse the tar archive and will help me extract my file. However, I then realised that it's actually not necessary, and I can just achieve everything I want from the shell.<\/p>\n
Let's start by finding my data in the file. As you remember, earlier I used less<\/code> to check if my data was there. When I searched there I noticed my file (\/data\/data\/com.orgzly\/backups\/orgzly.db<\/samp>) was the second match, and that the next match was just after what seemed to be the end of my file. Based on these assumptions, I ran grep to find the offsets, and got the following:<\/p>\n
<\/span>$ <\/span>grep --byte-offset --only-matching --text \/data\/data\/com.orgzly\/databases\/orgzly.db data.tar\n2095003651:\/data\/data\/com.orgzly\/databases\/orgzly.db<\/span>\n2095004675:\/data\/data\/com.orgzly\/databases\/orgzly.db<\/span>\n2095349251:\/data\/data\/com.orgzly\/databases\/orgzly.db<\/span>\n2095350275:\/data\/data\/com.orgzly\/databases\/orgzly.db<\/span>\n... snip ...<\/span>\n<\/code><\/pre><\/div>\n\n
Based on this, I concluded that the relevant tar entry starts at 2095004675<\/samp> and ends at 2095349251<\/samp>. I then used dd<\/code> to extract this chunk to make the file easier to work with.<\/p>\n
<\/span>$ <\/span>dd skip<\/span>=<\/span>2095004675<\/span> count<\/span>=<\/span>344576<\/span> bs<\/span>=<\/span>1<\/span> if<\/span>=<\/span>data.tar of<\/span>=<\/span>orgzly.db\n344576+0 records in<\/span>\n344576+0 records out<\/span>\n344576 bytes (345 kB, 336 KiB) copied, 0.393915 s, 875 kB\/s<\/span>\n<\/code><\/pre><\/div>\n\n
<\/span>$ <\/span>grep --byte-offset --only-matching --text SQLite orgzly.db\n509:SQLite<\/span>\n<\/code><\/pre><\/div>\n\n
I then trimmed the file accordingly, and tried opening it with sqlite:<\/p>\n
<\/span>$ <\/span>dd skip<\/span>=<\/span>509<\/span> bs<\/span>=<\/span>1<\/span> if<\/span>=<\/span>orgzly.db of<\/span>=<\/span>orgzly.fixed.db\n344067+0 records in<\/span>\n344067+0 records out<\/span>\n344067 bytes (344 kB, 336 KiB) copied, 0.397661 s, 865 kB\/s<\/span>\n\n$ <\/span>sqlite3 orgzly.fixed.db\nSQLite version 3.22.0 2018-01-22 18:45:57<\/span>\nEnter ".help" for usage hints.<\/span>\nsqlite> .tables<\/span>\nandroid_metadata         note_properties          repos<\/span>\nbook_links               notes                    rook_urls<\/span>\nbook_syncs               notes_view               rooks<\/span>\nbooks                    org_ranges               searches<\/span>\nbooks_view               org_timestamps           times_view<\/span>\ncurrent_versioned_rooks  properties               versioned_rooks<\/span>\ndb_repos                 property_names<\/span>\nnote_ancestors           property_values<\/span>\n<\/code><\/pre><\/div>\n\n
And it worked!<\/p>\n
Now that I managed to recover the database, I copied it to my phone, fixed the permissions and SELinux attributes and started orgzly. Everything worked, and my tasks were saved!<\/p>","category":[{"@attributes":{"term":"misc"}},{"@attributes":{"term":"backup"}},{"@attributes":{"term":"linux"}}]},{"title":"Signed Web Pages","link":{"@attributes":{"href":"https:\/\/stosb.com\/blog\/signed-web-pages\/","rel":"alternate"}},"published":"2017-12-28T14:00:00+00:00","updated":"2017-12-28T14:00:00+00:00","author":{"name":"Tom Hacohen"},"id":"tag:stosb.com,2017-12-28:\/blog\/signed-web-pages\/","summary":"
There is one caveat though. The above describes the code in \u2026<\/p>","content":"
As some of you already know, I've recently released the EteSync Web App<\/a>. It's a fully blown EteSync client that lets you access all of your data while maintaining the End-to-End encryption assurances you've come to expect from EteSync.<\/p>\n
There is one caveat though. The above describes the code in the EteSync repos, not necessarily the code on the server or the code being served by the server.<\/p>\n
One way to solve it, is to clone the EteSync Web App's repo, verify it locally using the signed tags, and then run a local version. This will give you the same assurances as a native app. The other, more convenient solution, is using the new web extension I just released: Signed Pages<\/a>.<\/p>\n
The Problem<\/h2>\n
Unlike with native applications, where you can install a version you can verify (or even build yourself), with web apps, you are served the app every time you enter the website.\nThis means, that the server could all of a sudden serve malicious code to everyone, a small group of users or a specific targeted user. Even worse, a state-sponsored attacker could issue a certificate for a certain domain, man-in-the-middle the connection to the server and serve malicious code to the user. This is slightly mitigated by certificate transparency<\/a>, but the issue still remains.<\/p>\n
The Solution<\/h2>\n
One solution to this problem is signing web pages using PGP and using trusted code (a browser extension) to verify it.<\/p>\n
As a user, all you need to do is install Signed Pages<\/a> and add the websites you want. At the moment, you need to manually add the pattern and the expected public key of the signer as provided by the developer, though I plan on making it more user friendly in the future.<\/p>\n
How it Works<\/h2>\n
Developers sign their pages using a detached PGP signature, and put the signature as the first comment in the page after the doctype. That's it.\nThere's even a script that does that for you, which you can get from here<\/a>.<\/p>\n
Users with the browser extension configured will then see a shield in their address bar (or tool bar, based on browser) based on the validity of the signature. If a signature is valid, a green shield with a black tick will be shown, and if it's invalid or missing (but expected for a page) a red shield with a big black X<\/em> will be shown.<\/p>\n
\"Valid<\/p>\n
We however only solved part of the problem. We now verify that the page we are accessing is indeed what we expect, but what about its external scripts and CSS? Luckily, this is already a solved problem. All you need to do is enable subresource integrity<\/a> for all of external resources in the page (event if served from the same server!), and the browser will verify them automatically.<\/p>\n
For example, to include jquery:<\/p>\n
<\/span><script<\/span> src=<\/span>"\/assets\/jquery-2.1.4.min.js"<\/span>\n    integrity=<\/span>"sha384-R4\/ztc4ZlRqWjqIuvf6RX5yb\/v90qNGx6fS48N0tRxiGkqveZETq72KgDVJCp2TC"<\/span>\n    crossorigin=<\/span>"anonymous"<\/span>><\/span>\n<\/script><\/span>\n<\/code><\/pre><\/div>\n\n
You can also include jquery from a CDN and you'll have the same integrity assurances.<\/p>\n
Quick Setup Before We Begin<\/h2>\n
In case you don't already have these set, you should set your name and email. I set\nthe globally to all repos, but if you omit the --global--<\/code> directive, you can set them locally per repo.<\/p>\n
<\/span>$ <\/span>git config --global user.name "Tom Hacohen"<\/span>\n$ <\/span>git config --global user.email "tom.git@stosb.com"<\/span>\n<\/code><\/pre><\/div>\n\n
You should also enable colours, set your default editor (what git will use for\nediting commit messages, for example) and an external tool that can be used to view diffs\nwhen running git difftool<\/code><\/p>\n
<\/span>$ <\/span>git config --global color.ui true<\/span> # Default since git 1.8.4<\/span>\n$ <\/span>export<\/span> EDITOR<\/span>=<\/span>"vim"<\/span> # Optional<\/span>\n$ <\/span>git config --global diff.tool "vimdiff"<\/span> # Extra optional<\/span>\n<\/code><\/pre><\/div>\n\n
You should also enable git command autocompletion (system dependant).<\/p>\n
At this point it's also worth mentioning that git has built-in support for aliases.\nSome of the commands presented here will be rather long, so it may be useful to alias some of the more commonly used ones.<\/p>\n
<\/span>$ <\/span>git config --global alias.unstage 'reset HEAD --'<\/span> # Set it up<\/span>\n$ <\/span>git unstage # Use it<\/span>\n<\/code><\/pre><\/div>\n\n
A Few Useful Properties of Git<\/h2>\n
Branches and Tags are References<\/h3>\n
\"Commit<\/p>\n
Branches and tags are just references to commits, and master is just a normal branch, so a reference too.\nThis means you can make branches and tags point to any commit in your history, or use them like you would any commit hash.<\/p>\n
Nearly Every Operation is Local<\/h3>\n
Because of git's decentralised nature, all the operations other than the ones that sync with a remote (either push or pull\/fetch) are local. This means that:<\/p>\n
More Information<\/h2>\n
For more information please refer to the README<\/a> and the source code, or let me know if you have any questions.<\/p>","category":[{"@attributes":{"term":"misc"}},{"@attributes":{"term":"pgp"}},{"@attributes":{"term":"encryption"}},{"@attributes":{"term":"web"}},{"@attributes":{"term":"security"}}]},{"title":"Advanced Git Commands You Will Actually Use","link":{"@attributes":{"href":"https:\/\/stosb.com\/blog\/advanced-git-commands-you-will-actually-use\/","rel":"alternate"}},"published":"2017-12-25T13:00:00+00:00","updated":"2017-12-25T13:00:00+00:00","author":{"name":"Tom Hacohen"},"id":"tag:stosb.com,2017-12-25:\/blog\/advanced-git-commands-you-will-actually-use\/","summary":"
This is the blog version of the talk I gave<\/a> at Samsung's UK office. People have been asking for a blog version, so here it is. While it covers mostly the same topic as the talk, it's not exactly the same. If you are only going to look at one \u2026<\/p>","content":"
This is the blog version of the talk I gave<\/a> at Samsung's UK office. People have been asking for a blog version, so here it is. While it covers mostly the same topic as the talk, it's not exactly the same. If you are only going to look at one of them, it's better to read this post.<\/p>\n
I'm intentionally brief throughout this post. The goal of this post is to expose you to new commands you may have not known. However, it should only serve as a starting point, and it's up to you to further explore those you found useful.<\/p>\n
This post assumes some basic working knowledge of git, though still covers some basics.\nIf you feel like you're already familiar with git, you can skip past the introduction<\/a><\/p>\n
As some of you already know, I've recently released the EteSync Web App<\/a>. It's a fully blown EteSync client that lets you access all of your data while maintaining the End-to-End encryption assurances you've come to expect from EteSync.<\/p>\n
I now needed to remove the tar headers. One way of doing it is to search for the tar header format, parse it, extract the file's length, and use that. However, I had a better idea: I figured out I could just find the beginning of the file using its magic number<\/a>.<\/p>\n