Skip to content

[qm-elk] Add support for QM data enrichment#902

Closed
vchrombie wants to merge 2 commits intochaoss:masterfrom
vchrombie:gitlabqm+gitqm
Closed

[qm-elk] Add support for QM data enrichment#902
vchrombie wants to merge 2 commits intochaoss:masterfrom
vchrombie:gitlabqm+gitqm

Conversation

@vchrombie
Copy link
Copy Markdown
Member

@vchrombie vchrombie commented Jun 23, 2020 •
edited
Loading

This PR contains gitlabqm and gitqm enrichers.

metrics discussion for gitqm: vchrombie/gsoc#9
output example for gitqm: gist-file-gitqm

micro-mordred configurations

project.json

{
    "potioncraft" : {
        "gitqm": [
            "https://github.com/vchrombie/gsoc.git"
        ],
        "gitlabqm:issue": [
            "https://gitlab.com/amfoss/cms-mobile"
        ],
        "gitlabqm:merge": [
            "https://gitlab.com/amfoss/cms-mobile"
        ]
    }
}

setup.cfg

[gitqm]
raw_index = gitqm_raw
enriched_index = gitqm
category = commit

[gitlabqm:issue]
api-token = xxxx
raw_index = gitlabqm-issues_raw
enriched_index = gitlabqm
sleep-for-rate = true
category = issue
no-archive = true

[gitlabqm:merge]
api-token = xxxx
raw_index = gitlabqm-merge_raw
enriched_index = gitlabqm
sleep-for-rate = true
category = merge_request
no-archive = true

@vchrombie vchrombie mentioned this pull request Jun 23, 2020
Closed
4 tasks
@vchrombie vchrombie marked this pull request as ready for review June 24, 2020 12:59
@vchrombie
[gitlabqm] Add gitlabqm enricher
8b709bf 
This commit adds the gitlabqm enricher. The base file, qmenrich.py
is added. The connectors for the enricher are also updated.

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
vchrombie
vchrombie commented Jun 24, 2020
View reviewed changes
Comment thread grimoire_elk/enriched/gitqm.py
vchrombie
vchrombie commented Jun 24, 2020
View reviewed changes
Copy link
Copy Markdown
Member Author

@vchrombie vchrombie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @valeriocos
I have completed working on the enricher but I had one doubt in the current implementation regarding to files_changed.

I think it is ready for review, please have a look when you are free. 😄

@vchrombie
[gitqm] Add gitqm enricher
5e620dc 
This commit adds the gitlabqm enricher. The connectors
for the enricher are also updated.

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
@coveralls
Copy link
Copy Markdown

coveralls commented Jun 24, 2020 •
edited
Loading

Pull Request Test Coverage Report for Build 2314

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 36 unchanged lines in 1 file lost coverage.
  • Overall coverage decreased (-1.5%) to 80.663%
Files with Coverage Reduction New Missed Lines %
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/utils.py 36 66.42%
Totals Coverage Status
Change from base Build 2313: -1.5%
Covered Lines: 8368
Relevant Lines: 10374
💛 - Coveralls

@vchrombie vchrombie mentioned this pull request Jun 24, 2020
Closed
valeriocos
valeriocos reviewed Jul 20, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@valeriocos valeriocos left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @vchrombie , the PR looks great, good job! I tested the PR on https://gitlab.com/Bitergia/devops/indexwarrior. I paid special attention to git, and compare the output of the data retrieved with git log --pretty=format:"%h%x09%an%x09%cd%x09%s" --all with the one available on the enriched indexes.

I left some comments mostly around the timezone. Please check them when you have time.

It would be convenient to add the schema of the quality models data at https://github.com/chaoss/grimoirelab-elk/tree/master/schema and the tests.

Thanks

EDIT: I forgot to add one of the things we discussed in the last GSoC meeting. The QM enrichers should go to a specific module, as similarly done for the SCMS ones. On the other hand, the tests for the QM enrichers don't need to be separated from the rest.

Comment thread grimoire_elk/enriched/gitqm.py
Comment thread grimoire_elk/enriched/gitqm.py
Comment thread grimoire_elk/enriched/gitqm.py
Comment thread grimoire_elk/enriched/gitlabqm.py
@vchrombie
Copy link
Copy Markdown
Member Author

vchrombie commented Jul 26, 2020

Hi @valeriocos
Thanks for the review. I have checked the comments. I have faced such issues while implementing the pipermailqm enricher. I fixed it. Updating the normalizing date function will fix most of the things.

EDIT: I forgot to add one of the things we discussed in the last GSoC meeting. The QM enrichers should go to a specific module, as similarly done for the SCMS ones. On the other hand, the tests for the QM enrichers don't need to be separated from the rest.

Sure, I will implement all these things and produce the next PR as an extension of this PR soon.

vchrombie added a commit to vchrombie/grimoirelab-elk that referenced this pull request Jul 30, 2020
@vchrombie
[gitqm] Address the issues raised in chaoss#902
7b46377 
This commit is in reference to the issues raised
- fix the date normalizing errors and also problems
related to the timezone
- made the '.git' string optional

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
vchrombie added a commit to vchrombie/grimoirelab-elk that referenced this pull request Jul 30, 2020
@vchrombie
[gitqm] Address the issues raised in chaoss#902
8f03208 
This commit is in reference to the issues raised
- fix the date normalizing errors and also problems
related to the timezone
- made the '.git' string optional
- fix the value error

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
vchrombie added a commit to vchrombie/grimoirelab-elk that referenced this pull request Jul 30, 2020
@vchrombie
[gitqm] Address the issues raised in chaoss#902
b7db4c9 
This commit is in reference to the issues raised
- fix the date normalizing errors and also problems
related to the timezone
- made the '.git' string optional
- fix the value error

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
vchrombie added a commit to vchrombie/grimoirelab-elk that referenced this pull request Jul 30, 2020
@vchrombie
[gitqm] Address the issues raised in chaoss#902
24d92b4 
This commit is in reference to the issues raised
- fix the date normalizing errors and also problems
related to the timezone
- made the '.git' string optional
- fix the value error

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
@vchrombie
Copy link
Copy Markdown
Member Author

vchrombie commented Jul 30, 2020

closing this PR, in reference to #926

@vchrombie vchrombie closed this Jul 30, 2020
vchrombie added a commit to vchrombie/grimoirelab-elk that referenced this pull request Aug 10, 2020
@vchrombie
[gitqm] Address the issues raised in chaoss#902
0a1244b 
This commit is in reference to the issues raised
- fix the date normalizing errors and also problems
related to the timezone
- made the '.git' string optional
- fix the value error

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
vchrombie added a commit to glab-prosoul/grimoirelab-elk that referenced this pull request Aug 21, 2021
@vchrombie
[gitqm] Address the issues raised in chaoss#902
7ade8d3 
This commit is in reference to the issues raised
- fix the date normalizing errors and also problems
related to the timezone
- made the '.git' string optional
- fix the value error

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
vchrombie added a commit to glab-prosoul/grimoirelab-elk that referenced this pull request Aug 23, 2021
@vchrombie
[gitqm] Address the issues raised in chaoss#902
68f3c28 
This commit is in reference to the issues raised
- fix the date normalizing errors and also problems
related to the timezone
- made the '.git' string optional
- fix the value error

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@valeriocos valeriocos valeriocos left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@vchrombie @coveralls @valeriocos