refactor writeto db to smaller function#399
Conversation
yokofly
changed the title
|
Thanks, started to test locally but dealing with another issue. Also, there are a few instances of the use of For example, instead of |
|
hahaha, let's enjoy the trip first! |
locally done some simple test(including halt fresh insert)
|
I locally performed some simple tests with clickhouse to clickhouse |
|
Tested, all good 👍 . |
| if cnt != tCnt { | ||
| err = g.Error(err, "inserted in temp table but table count (%d) != stream count (%d). Records missing/mismatch. Aborting", tCnt, cnt) | ||
| return 0, err | ||
| } else if tCnt == 0 && len(sampleData.Rows) > 0 { | ||
| err = g.Error(err, "Loaded 0 records while sample data has %d records. Exiting.", len(sampleData.Rows)) | ||
| return 0, err | ||
| } |
There was a problem hiding this comment.
this code is buggy, sorry about that, I find the err will pass to the internal, and got something like
WRN NewError called with nil error:
old code
tCnt, _ := tgtConn.GetCount(tableTmp.FullName())
if cnt != tCnt {
err = g.Error("inserted in temp table but table count (%d) != stream count (%d). Records missing/mismatch. Aborting", tCnt, cnt)
return
} else if tCnt == 0 && len(sampleData.Rows) > 0 {
err = g.Error("Loaded 0 records while sample data has %d records. Exiting.", len(sampleData.Rows))
return
}
…tetoDB refactor writeto db to smaller function
2 participants
I intend to split as 2 prs for #35
the first part is to refactor the writetodb function, I have run some tests, probably the CI will run a more compatible suite.
I conduct a more direct test to verify the temp table: insert a big table, stop with ctrl+c during the insert, and collect the log
this is the whole log difference(between release 1.2.20 and commit ac8802a), from the log seems all good
https://www.diffchecker.com/f4YCxD9W/