Implement import of databases in CSV (Comma Separated Values) format#146
Conversation
|
Hi, I think this is a long overdue feature that is much needed by pretty much anyone migrating to keepassx from another password manager (1password, etc). |
|
We will merge this pull request into 2.2.0 after we reviewed it in-depth. |
src/core/CsvParser.cpp
Outdated
| m_isFileLoaded = false; | ||
| } | ||
| else { | ||
| device->close(); |
src/core/CsvParser.cpp
Outdated
| skipLine(); | ||
| return; | ||
| } | ||
| else { |
There was a problem hiding this comment.
@seatedscribe no need to enclose in an else block here, because of the early return just above.
There was a problem hiding this comment.
Return is expected only when if (isComment()) is true. I think you made a typo 👀
There was a problem hiding this comment.
@seatedscribe at the end of the if (isComment()), you return from the function. So, no need to enclose the rest of the function in a else block. This is mostly a matter of style though.
There was a problem hiding this comment.
@louib Ah, now I get it. Yes, it's a matter of style. My point is that with the else 'label' you stick with the usual code flow and the eye can catch the link/separation of the two logical branches immediately.
There was a problem hiding this comment.
Early return guards are nicer and IMHO also easier to read. Unnecessary indented else blocks add more visual complexity that can be avoided.
src/core/CsvParser.h
Outdated
| #include <QTextStream> | ||
|
|
||
| typedef QStringList csvrow; | ||
| typedef QList<csvrow> csvtable; |
There was a problem hiding this comment.
@seatedscribe I think these need to be PascalCase, since they define types. e.g. typedef QStringList CsvRow
| #include "gui/MessageBox.h" | ||
|
|
||
| #include <QFile> | ||
| #include <QFileInfo> |
There was a problem hiding this comment.
@seatedscribe I think the general guideline for imports in the project is:
- _ui and associated .h file
- QT imports
- imports from the project.
These three sections separated by a blank line.
Look at SearchWidget.cpp's import section, for example.
There was a problem hiding this comment.
We don't really have a guideline for the order of imports. Most of the code uses what louib describes, so maybe it's best to stick with it for now. However, I much dislike it. I would much more prefer
- ui and associated .h file
- project includes
- Qt imports
- system imports
src/gui/csvImport/CsvImportWidget.h
Outdated
| Database *m_db; | ||
|
|
||
| KeePass2Writer m_writer; | ||
| static const QStringList m_columnheader; |
| QFont font = m_ui->labelHeadline->font(); | ||
| font.setBold(true); | ||
| font.setPointSize(font.pointSize() + 2); | ||
| m_ui->labelHeadline->setFont(font); |
| int x=i%combo_rows; | ||
| int y= 2*(i/combo_rows); | ||
| m_ui->gridLayout_combos->addWidget(label, x, y); | ||
| m_ui->gridLayout_combos->addWidget(combo, x, y+1); |
There was a problem hiding this comment.
@seatedscribe missing spaces around operators in the last couple of lines.
| m_ui->tableViewFields->resizeRowsToContents(); | ||
| m_ui->tableViewFields->resizeColumnsToContents(); | ||
|
|
||
| for (int c=0; c<m_ui->tableViewFields->horizontalHeader()->count(); ++c) { |
There was a problem hiding this comment.
🆗 I also made the same correction here and there in my files
There was a problem hiding this comment.
When you're done with your changes, can you rebase your branch on develop, so we can re-review it, fix any remaining issues and then finally merge this baby?
| } | ||
| else { | ||
| //SHOULD NEVER GET HERE | ||
| m_db->rootGroup()->setName("ROOT_FALLBACK"); |
There was a problem hiding this comment.
@seatedscribe but we do get here, if we try importing an empty csv file...
| } | ||
|
|
||
|
|
||
| void CsvImportWidget::checkGroupNames() { |
There was a problem hiding this comment.
@seatedscribe since this function sets the root group of the db, why not name it setRootGroup?
|
@seatedscribe thanks for the PR! Some comments: When trying to load the following file (created by exporting from KeePassXC The program seg faults if skip first rows is set to 1 before trying to import the csv file. About that field, is it really a use case to skip anything else than the first (header) line? Why not just a checkbox When cancelling the CSV import, the user gets asked 2 times to discard changes instead of one. |
phoerious
left a comment
There was a problem hiding this comment.
I marked a few more things that need to be changed. Other than those, I'm really excited to merge. Good work!
src/core/CsvParser.cpp
Outdated
| bool CsvParser::parseFile() { | ||
| parseRecord(); | ||
| while (!m_isEof) | ||
| { |
There was a problem hiding this comment.
Please put opening braces of loops and conditions in the same line as the loop header (issue also found in other places of the code).
src/core/CsvParser.cpp
Outdated
| if (isQualifier(m_ch)) { | ||
| parseQuoted(field); | ||
| } | ||
| else { |
There was a problem hiding this comment.
For new code, please write if/else as
if (condition) {
// ....
} else {
// ...
}
There was a problem hiding this comment.
🆗 also for old code now
There was a problem hiding this comment.
Well, as far as KeePassXC is concerned, everything in this PR is new code. 😉
src/core/CsvParser.cpp
Outdated
| void CsvParser::fillColumns() { | ||
| //fill the rows with lesser columns with empty fields | ||
|
|
||
| for (int i=0; i<m_table.size(); ++i) { |
There was a problem hiding this comment.
Please add spaces around operators (also found in many other parts of the code)
There was a problem hiding this comment.
done already
src/core/CsvParser.h
Outdated
| void setFieldSeparator(const QChar c); | ||
| void setTextQualifier(const QChar c); | ||
| void setBackslashSyntax(bool set); | ||
| int getFileSize() const; |
There was a problem hiding this comment.
Uff, 🆗 and you dudes, don't over-analyze 😄
src/core/CsvParser.h
Outdated
| bool isFileLoaded(); | ||
| //reparse the same buffer (device is not opened again) | ||
| bool reparse(); | ||
| void setCodec(const QString s); |
There was a problem hiding this comment.
Use const references for parameters of non-basic types, not const copy
| } | ||
|
|
||
| void CsvImportWidget::showReport() { | ||
| MessageBox::warning(this, tr("Syntax error"), tr("While parsing file...\n") |
There was a problem hiding this comment.
Maybe use the new MessageWidget instead of a MessageBox? (here and for the other two or three times you are showing error messages)
There was a problem hiding this comment.
@phoerious It's a pretty good widget and for short messages it certainly is the right tool; in my use cases, however, I think it could bring more problems than it solves, especially with the "Show parser warnings" where it exposes a possibly long list which is handled incorrectly.

I didn't dig to understand if it can handle a sidebar or can be adapted for this kind of use, but I would keep the (already large) PR codebase as it is (WRT this topic).
There was a problem hiding this comment.
Well, you shouldn't show a gazillion error messages at once anyway. A message box that extends beyond the edge of the screen and therefore has no usable OK button isn't any better.
You should only show the first few message and then add " more messages" if there are any errors left.
The MessageWidget should also be at the top of the CSV widget. Right now it appears to be below the "Import CSV fields" heading and also doesn't expand horizontally. Compare its use in the DatabaseOpenWidget.
There was a problem hiding this comment.
@seatedscribe what does it look like with the same error message with a MessageBox? Maybe the MessageWidget could display a generic Invalid CSV format message when the parsing failed, and clicking Show parser warnings would show the MessageBox with all the details?
If we are to display 1 error line per line in the csv file, It might be hard to display elegantly whatever the chosen method is, so I wouldn't mind a MessageBox in this particular case.
| QString groupLabel; | ||
| QStringList groupList; | ||
| bool is_root = false | ||
| , is_empty = false |
There was a problem hiding this comment.
Please don't batch initialize variables
There was a problem hiding this comment.
Hm? How would you put those lines?
There was a problem hiding this comment.
Each one with a separate type identifier.
src/gui/csvImport/CsvParserModel.cpp
Outdated
| m_skipped = skipped; | ||
| QModelIndex topLeft = createIndex(skipped,0); | ||
| QModelIndex bottomRight = createIndex(m_skipped+rowCount(), columnCount()); | ||
| Q_EMIT dataChanged(topLeft, bottomRight); |
There was a problem hiding this comment.
We decided on using MOC keywords now instead of Q_* macros, so use emit instead of Q_EMIT
src/gui/csvImport/CsvParserModel.cpp
Outdated
| } | ||
|
|
||
| QVariant CsvParserModel::data(const QModelIndex &index, int role) const { | ||
| if ( (index.column() >= m_columnHeader.size()) |
There was a problem hiding this comment.
Weird formatting, I think this fits into one line
src/gui/csvImport/CsvParserModel.h
Outdated
| void setHeaderLabels(QStringList l); | ||
| void mapColumns(int csvColumn, int dbColumn); | ||
|
|
||
| virtual int rowCount(const QModelIndex &parent = QModelIndex()) const Q_DECL_OVERRIDE; |
There was a problem hiding this comment.
Use C++11 keyword override, not Q_DECL_OVERRIDE
There was a problem hiding this comment.
Oh, and the virtual keyword is not needed when the overridden method is already virtual, of course.
This is surprise since I tested by hand (ok, more than a year ago on the original KeepassX project, but still...) Nonetheless you are right; I'll check.
Because I don't want to make assumptions on users' habits...
🆗 |
|
@phoerious Ehm, I may have made a mess with the merge? I wanted to rebase but it seems I am still learning by errors 😄 |
|
Your call. It's your branch, nobody else has committed to. So feel free to do whatever you want. |
| KeePass2Writer writer; | ||
| writer.writeDatabase(&buffer, m_db); | ||
| if (writer.hasError()) { | ||
| MessageBox::warning(this, tr("Error"), tr("CSV import: writer has errors:\n") |
There was a problem hiding this comment.
@phoerious Would not use messageWidget here because subsequent emit editFinished prevents the dialog to show up
| QApplication::restoreOverrideCursor(); | ||
|
|
||
| if (!result) { | ||
| MessageBox::critical(this, tr("Error"), tr("Unable to calculate master key")); |
There was a problem hiding this comment.
@phoerious Would not use messageWidget here because subsequent emit importFinished prevents the dialog to show up
8a71168 to
8e8dd51
Compare
|
@phoerious @louib This is definitive commit for you re-review. |
|
I only have two (three) more things I need you to fix: You should rebase this branch on the current develop. It's quite behind the current HEAD revision and we can't merge before it's up2date with develop. It also generates two warnings in your own code which should be fixed: and then there are a few quirks in the user interface: The message widget has very large padding. You should not wrap the message in newlines (maybe show no success message at all?). Also the input fields in the "Encoding" section look pretty ugly. The dropdowns have inhomogeneous heights and are lacking padding and the spin box has no border. You can easily fix that by removing the min-height of the group box and setting its vertical size policy to fixed. You should do the same to the group "Column layout" group box. While you do that, please right-align all the labels. |
| QApplication::restoreOverrideCursor(); | ||
| if (good) | ||
| //four newline are provided to avoid resizing at every Positive/Warning switch | ||
| m_ui->messageWidget->showMessage(QString("\n\n").append(tr("CSV syntax seems in good shape !")) |
There was a problem hiding this comment.
Remove the newlines here (see comment above).
There was a problem hiding this comment.
messageWidget height changes with the text, and all the rest of the window content will follow, sliding up and down. This happens every single time you change any of the Encoding parameters, so every format which is not "standard" (i.e. every format which doesn't come from a password manager) will probably require some tweaking triggering this behaviour.
Use the widget with naked messages - no newline padding - and within a couple of swirls your eyes will implore harakiri. If you can play around with this db you will understand what I mean (note the second row with a field equal to "2\"":
"Account","Login Name","Password","Web Site","Comments"
"test1","2\"","pwd","URL","some notes..."
"test","€èéç","hahaha","",""
work/group1;acc3;log3;pass3;ws3;com3;field3;lco3
work/group2/subgroup3;acc4;log4;pass4;ws4;com4;field4;lco4
fun/basket;acc5;log5;pass5;ws5;com5;field5;lco5
fun/basket;acc6;log5;pass5;ws5;com5;field5;lco5
fun/basket;acc7;log5;pass5;ws5;com5;field5;lco5
abbaz\as;accZ;log5;pass5;ws5;com5;field5;lco5
Do you think a large, reassuring green pad is so bad? Or that is it really worst than an elevator-style GUI? 😄
There was a problem hiding this comment.
-
I actually rebased on develop before pushing, only commits of yesterday are not included (they will be with next rollout)
-
I cannot reproduce the warnings (though I get a lot of other warnings around qhttp, Z-order assignment in EditEntryWidgetAutoType.ui and couple of 'unused result' in Application.cpp). Any hint on how can I set up to have them displayed?
-
will fix the GUI as per your comments
There was a problem hiding this comment.
If it's as bad as you say, then remove the message completely. I don't really see the need for a success message. Show an error if one occurs, but we don't need an explicit success message. One can see that everything went well when the CSV contents show up correctly in the table below.
To reproduce the warnings you have to clear your built and force re-compilation of everything, I suppose. The qHttp warnings have been fixed in the release/2.1.3 branch. I think we haven't merged everything back into develop yet. Will do asap.
There was a problem hiding this comment.
Updated develop, please rebase.
There was a problem hiding this comment.
I don't really see the need for a success message.
Me neither. But we need to handle the fact that an error message CAN happen and will dislocate the entire GUI back and forth. Is this acceptable for you? I hope not.
There was a problem hiding this comment.
I don't have an issue with an error message shifting down the GUI, since it shouldn't happen more than once. In the best case, the user will not see any message at all.
There was a problem hiding this comment.
I agree.
So I will cut out the green message and limit the warning to the first two issues. 👍
|
@phoerious Rebuild is not giving me those warnings, moreover I have no apparent issues on combo paddings... Wondering what could be misbehaving. What PC are you running on? |
2cb3bec to
39057a6
Compare
|
@seatedscribe the crash happens because of an assertion. Can you try with the assertions enabled? |
| else | ||
| for (int i = 0; i < 2 - items; i++) | ||
| text.append(QString("\n")); | ||
| return text; |
There was a problem hiding this comment.
I'm still getting the following warning, which is generated by this code section:
/.../keepassxc/src/gui/csvImport/CsvImportWidget.cpp: In member function ‘QString CsvImportWidget::formatStatusText() const’:
/.../keepassxc/src/gui/csvImport/CsvImportWidget.cpp:191:5: warning: this ‘else’ clause does not guard... [-Wmisleading-indentation]
else
^~~~
/.../keepassxc/src/gui/csvImport/CsvImportWidget.cpp:194:9: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘else’
return text;
^~~~~~
The only reason it works as intended is because it's the last statement in the function and the if block already contains a return. I would change it like this:
if (items > 2) {
return text.section('\n', 0, 1)
.append("\n[").append(QString::number(items - 2))
.append(tr(" more messages skipped]"));
}
for (int i = 0; i < 2 - items; ++i) {
text.append(QString("\n"));
}
return text;
You should also replace i++ with ++i as I've done it here.
There was a problem hiding this comment.
On a different note, can items == 0 ever happen? If not, you should replace the for loop with an if condition, as it will only run when items == 1. Also do we still need this here at all? This is also only used to add newlines to error messages which isn't exactly pretty.
There was a problem hiding this comment.
I don't know if it's reproducible there also, but if I skip this part altogether I get a weird layout of the messageWidget when first text showed is shorter than subsequent messages. I think the widget is expected to be sort of "one shot" and then disappear, and not supposed to update its content while not hidden:
Note the lower corners are not round:

So I would keep it, updated with your changes. 🆗?
There was a problem hiding this comment.
Yeah, ok. But still fix the first issue.
|
@louib Can you verify that the crash is fixed? |
|
@phoerious yup, no more seg fault when importing with asserts enabled! |
|
How this behave if there are more than 6 Column (e.g. Columns for Advanced attributes ?) |
|
If you have more than five columns in the csv file you can select from the
GUI which are the five you want your new db composed of.
If you are asking about the PR capabilities instead, currently those five
columns only are supported, and it would not be a lot of work to add
another field.
I'm however wondering why this PR is not still good to your eyes, or I'm
just misleaded by lack of communication.
I am eager to add/modify capabilities as needed when users will propose for
it and a discussion would arise. Now it seems to me a bit premature.
First thing, I would like this feature be available to a wide audience that
also could make a transition to this PM from other brands. This is still
not possible therefore I ask to retain any other request, such this, for
the near future development, after this PR will be merged.
I "officially" give my support as a maintainer for this part of the code,
and looking forward for this phase to be closed soon.
…On 15 Mar 2017 13:32, "TheZ3ro" ***@***.***> wrote:
How this behave if there are more than 6 Column (e.g. Columns for Advanced
attributes ?)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#146 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AMqpZFYt3arp28r_in9cxBTEePycve3Lks5rl9pCgaJpZM4Ld4C_>
.
|
|
I think this PR is generally ready for merging. Only need to have a last look through everything. |
|
Ok let me know when you are finished with your review so I can rebase on
the last one
…On 15 Mar 2017 14:38, "Janek Bevendorff" ***@***.***> wrote:
I think this PR is generally ready for merging. Only need to have a last
look through everything.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#146 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AMqpZAMd0eReIAppw9z0xheZECeeGmvSks5rl-nVgaJpZM4Ld4C_>
.
|
|
I just wanted to know how this perform with custom attributes, a future improvement can be the option to load custom attributes from CSV and using the column name as attributes name. |
src/gui/csvImport/CsvImportWidget.h
Outdated
| ~CsvImportWidget(); | ||
| void load(const QString& filename, Database* const db); | ||
|
|
||
| Q_SIGNALS: |
There was a problem hiding this comment.
I think this can be merged as is. But please run a search and replace for Q_EMIT -> emit, Q_SIGNALS -> signals and Q_SLOTS -> slots over the source code before rebasing it. We migrated to MOC keywords recently.
There was a problem hiding this comment.
Done presently.
Modified some other files which had Q_XYZ keywords:
src/keys/drivers/YubiKey.h
tests/TestYkChallengeResponseKey.h
| <item> | ||
| <widget class="QTabWidget" name="tabWidget"> | ||
| <property name="currentIndex"> | ||
| <number>0</number> |
There was a problem hiding this comment.
This needs to be reverted to 0. You must have accidentally changed that.
There was a problem hiding this comment.
Hmmm, don't know how it happened 😕
There was a problem hiding this comment.
Happens when you open the UI file in the designer, click on a tab and save.
- Added YubiKey 2FA integration for unlocking databases [#127] - Added TOTP support [#519] - Added CSV import tool [#146, #490] - Added KeePassXC CLI tool [#254] - Added diceware password generator [#373] - Added support for entry references [#370, #378] - Added support for Twofish encryption [#167] - Enabled DEP and ASLR for in-memory protection [#371] - Enabled single instance mode [#510] - Enabled portable mode [#645] - Enabled database lock on screensaver and session lock [#545] - Redesigned welcome screen with common features and recent databases [#292] - Multiple updates to search behavior [#168, #213, #374, #471, #603, #654] - Added auto-type fields {CLEARFIELD}, {SPACE}, {{}, {}} [#267, #427, #480] - Fixed auto-type errors on Linux [#550] - Prompt user prior to executing a cmd:// URL [#235] - Entry attributes can be protected (hidden) [#220] - Added extended ascii to password generator [#538] - Added new database icon to toolbar [#289] - Added context menu entry to empty recycle bin in databases [#520] - Added "apply" button to entry and group edit windows [#624] - Added macOS tray icon and enabled minimize on close [#583] - Fixed issues with unclean shutdowns [#170, #580] - Changed keyboard shortcut to create new database to CTRL+SHIFT+N [#515] - Compare window title to entry URLs [#556] - Implemented inline error messages [#162] - Ignore group expansion and other minor changes when making database "dirty" [#464] - Updated license and copyright information on souce files [#632] - Added contributors list to about dialog [#629]






Description
IMPORT CSV FORMAT
It is now possible for KeepassXC to import databases from CSV text files.
PARSER
I developed the parser with a BNF approach for syntax validation; just for reference, this is the RFC grammar as provided by the IETF (RFC 4180):
file = [header CRLF] record *(CRLF record) [EOF] #[CRLF]
header = name *(COMMA name)
record = field *(COMMA field)
name = field
field = (escaped | non-escaped)
escaped = DQUOTE *(TEXTDATA | COMMA | CR | LF | 2DQUOTE) DQUOTE
non-escaped = *TEXTDATA
COMMA = %x2C
CR = %x0D
DQUOTE = %x22
LF = %x0A
CRLF = CR LF
TEXTDATA = %x20-21 | %x23-2B | %x2D-7E
I needed to validate the two most used versions - the "doubling quotes" and
the "escape character" - into the same engine, so I slightly modified
the grammar to better reflect actual code implementation. I also extended it
to validate the case of "escaped characters". This is the result:
file = record *(CRLF record) EOF
record = field *(separator field) | comment --e.g. ,,A,B has 2 empty fields and is valid
field = simple | quoted
simple = TEXT -- e.g. f""ie"l"""d is OK
quoted = qualifier escaped qualifier
escaped = ESC_TEXT *(escape_mark ESC_TEXT)
escape_mark = (2qualifier | BACKSLASH qualifier)
qualifier = one of {DQUOTE, COLON, BACKSLASH}
separator = one of {COMMA, COLON, SEMICOLON}
comment = *(SPACE | TAB) COMMENT
COMMA = %x2C --i.e. ','
COLON = %x58 --i.e. ':'
SEMICOLON = %x59 --i.e. ';'
COMMENT = %x23 --i.e. '#'
DQUOTE = %x22 --i.e. '"'
BACKSLASH = %x5C --i.e. ''
CR = %x0D
LF = %x0A
SPACE = %x20
CRLF = CR LF | LF
TEXT = *(any char except CRLF, separator, EOF) (also TEXT="" shall be valid)
ESC_TEXT = +(any char except qualifier, EOF)
GUI
I enjoyed putting some effort in GUI design to make the user experience enjoyable and at the same time to be future-proof and avoid reinventing the wheel; for further details look at the comment around
CsvImportWidget::m_columnHeader.TEST
Last but not least, in particular when we talk about security, TDD is a must. I think I put for the parser testing all the conceivable cases together and get to successfully pass them (at the end).
Hope will help and be useful!
Regards,
Enrico
Motivation and Context
This is the original PR made for KeepassX rebased on keepassxc/develop branch.
How Has This Been Tested?
Test cases are covered in the appropriate file under tests/ directory.
Mainly manual user tests for the GUI.
Types of changes
Checklist: