[IMPROVE] Slack "corporate-export"-import (groups, direct messages and multi-user-messages) by niklasdahlheimer · Pull Request #13942 · RocketChat/Rocket.Chat

niklasdahlheimer · 2019-03-28T21:43:01Z

An attempt to include abilities to import all of the data from slack-corporate-exports.
Slack corporate-exports are including private chats, direct messages and multi-user-direct-messages.

The structure of the corporate-export zip-file

├── channels.json
├── subfolder_channel_1 (i.e. "General", "public Information")
│   ├── 2019-12-20.json            
│   ├── 2019-12-21.json
│   └── ... 
├── ... (more subfolders)
├── groups.json
├── subfolder_group_1 (i.e. "secret Group", "Management")
│   ├── 2019-12-20.json            
│   ├── 2019-12-21.json
│   └── ... 
├── ... (more subfolders)
├── mpims.json
├── subfolder_multi_person_chat_1 (i.e. "mpdm-joe-lisa-anna-john", "mpdm-lisa-john-peter")
│   ├── 2019-12-20.json            
│   ├── 2019-12-21.json
│   └── ... 
├── ... (more subfolders)
├── dms.json
├── subfolder_direct_messages_1 (i.e. "D38RUJBCP", "D39AJJG7L")
│   ├── 2019-12-20.json            
│   ├── 2019-12-21.json
│   └── ... 
├── ... (more subfolders)

groups.json and mpims.json looks just like the channels.json.

The structure of the dms.json

[
    {
        "id": "D38RUJBCP",
        "created": 1529308826,
        "members": [
            "U63MAB7U5",
            "U1D4CHJLV"
        ]
    },
    {
        "id": "D39AJJG7L",
        "created": 1527578772,
        "members": [
            "U63MAB7U5",
            "U1GKUUV7H"
        ]
    },...
]

I added parsing of groups.json, which is the equivalent to private channels in Rocket-Chat.

Next steps

process mpims.json, which contains the multi-user-direct-messages.
multi-user-direct-messages are like private channels without a name. Because Rocket-Chat has no feature like this, it needs to be handled as private channels. But the channel name "user1-user2-user3-.." would look silly. Any suggestions on this?
direct messages should be parsed also. Two cases:
- rocketchat_room-document for the private-chat already exists -> add messages to this room id
- rocketchat_room-document for the private-chat does not exist -> the importer needs to create the rocketchat_room-document for this private chat. How does Rocket-Chat add the rocketchat_room-document for private chats? I only found lib/server/methods/createPrivateGroup.js and lib/server/methods/createChannel.js. Is there an equivalent for private chats?

is now processing groups.json which is the slack-export-file for slack-private-channels. Also done some preperations to process direct chats later on.

CLAassistant · 2019-03-28T21:43:09Z

All committers have signed the CLA.

+added a few log-infos

+minor fixes, reformating

This reverts commit 96668db.

niklasdahlheimer · 2019-04-01T13:32:06Z

I added support for direct message import.

should multi-user-chats be handled the same way as private channels?
Should the user at the import-selection have the possibility to check/uncheck which direct-messages are being imported? Right now the importer imports all direct messages here both users are available.

@geekgonecrazy and @Hudell, any comments about this?
Some feedback on the code would be appreciated due to my lack of experience with javascript.

test

niklasdahlheimer · 2019-04-11T10:18:28Z

I managed to import the data including direct chats and private channels from the slack-corporate-import-zip-file, backuped and restored the mongo-data and use it in the docker production environment of our company right now.
There is still a bug which leads to a fail of the first import, the second import works flawlessly, it can be traced back to a undefined room object in line 381, where I'm not really sure why this happens.

However, some comments of the project maintainers would be appreciated because I still need to know:

how we should handle multi-user-chats
is there any objection against appending the private channel data to the channel collection (see line 98)
what should be the way of improving the feedback for the user? Right now uploading and parsing of the zip file, even if everything is running on localhost, takes really long (~5min for a 10mb-file) and the only feedback the user gets is a connection timeout and a page reload after uploading and preparing has finished. But that not leads back to the import selection, the user has manually navigate to the slack importer again to see the import selection.

Hudell · 2019-05-22T20:42:13Z

Multi user messages can be skipped for now. We are going to implement a similar feature on Rocket.Chat soon, but until then it's ok to skip those conversations.

geekgonecrazy · 2019-05-31T06:00:44Z

Are we planning to get this in a version near? I know there are several people wanting these changes to begin using rocket.chat

sampaiodiego · 2019-07-02T14:45:44Z

can someone please fix the conflicts? 🤗

niklasdahlheimer · 2019-07-10T17:38:30Z

I think, beside fixing the conflicts, additional testing is needed. I managed to use this version to import the corporate data into Rocket.Chat but had to delete a few files from the import-data-zip-file.
It worked for our company so I didn't spent more time on that.
Unfortunately I can not provide any test-data, because we deleted the uncrypted corporate-import-data due to privacy reasons.

If someone could provide test-data I could bring this feature to a more stable state, so it could be merged into a release.

pigmonkey · 2019-07-11T00:50:56Z

I have corporate exports and am waiting on this feature before migrating to rocket chat. I cannot share the data, but I can try to spin up a development environment to test this branch on my exports and report the results.

On the subject of multi-party direct messages, I would be happy for them to be imported as user1-user2-user3... private channels -- unless rocket chat's version of that feature is imminent. I would rather not skip that data.

AitorCM · 2019-11-26T11:12:07Z

So, any news on when this feature is going to be added to RocketChat?

Thanks!

geekgonecrazy · 2020-01-29T23:17:41Z

#16279 - this major refactor might be related

niklasdahlheimer added 7 commits March 28, 2019 13:55

added support for private channel import

df077ab

is now processing groups.json which is the slack-export-file for slack-private-channels. Also done some preperations to process direct chats later on.

reformat

6c36651

reformat

386b703

reformat

a631a86

reformat

989c3ad

reformat

dc66417

reformat

9b66fa3

niklasdahlheimer and others added 7 commits March 28, 2019 15:30

Merge branch 'develop' into slack-coporate-import

7c098b6

added direct message channel support

6721119

added direct message import

3f336de

+added a few log-infos

added progress steps

34c5e1f

+minor fixes, reformating

Auto stash before rebase of "refs/heads/slack-coporate-import"

96668db

Revert "Auto stash before rebase of "refs/heads/slack-coporate-import""

d3f1657

This reverts commit 96668db.

improvements

a4bcb4d

niklasdahlheimer force-pushed the slack-coporate-import branch from 3f6884b to a4bcb4d Compare April 1, 2019 13:26

niklasdahlheimer and others added 4 commits April 1, 2019 15:41

reformat

0342005

remformat

453e743

reformat

48f091b

Merge branch 'develop' into slack-coporate-import

4672a82

niklasdahlheimer changed the title ~~[NEW] Slack "corporate-export"-import (groups, direct messages and multi-user-messages)~~ [IMPROVE] Slack "corporate-export"-import (groups, direct messages and multi-user-messages) Apr 2, 2019

niklasdahlheimer and others added 2 commits April 4, 2019 21:25

Merge pull request #1 from RocketChat/develop

713f02a

test

Merge remote-tracking branch 'origin/develop' into slack-coporate-import

b613780

engelgabriel requested a review from Hudell April 6, 2019 07:20

engelgabriel assigned Hudell Apr 6, 2019

engelgabriel added this to the 1.1.0 milestone Apr 6, 2019

niklasdahlheimer and others added 3 commits April 6, 2019 23:21

fixed private channel import

8845861

reformat code

756233b

reformat

1accd89

reformat

c1083d4

rodrigok modified the milestones: 1.1.0, 1.2.0 May 15, 2019

Hudell approved these changes May 22, 2019

View reviewed changes

engelgabriel modified the milestones: 1.2.0, 1.3.0 Jul 10, 2019

sampaiodiego modified the milestones: 1.3.0, 1.4.0 Jul 25, 2019

sampaiodiego modified the milestones: 2.0.0, 2.1.0 Sep 6, 2019

rodrigok assigned pierre-lehnen-rc Sep 19, 2019

engelgabriel modified the milestones: 2.1.0, 2.2.0 Oct 13, 2019

rodrigok modified the milestones: 2.2.0, 2.3.0 Oct 19, 2019

rodrigok added the stat: conflict label Oct 19, 2019

pierre-lehnen-rc mentioned this pull request Jan 30, 2020

[IMPROVE] Major overhaul on data importers #16279

Merged

12 tasks

sampaiodiego closed this in #16279 Feb 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IMPROVE] Slack "corporate-export"-import (groups, direct messages and multi-user-messages)#13942

[IMPROVE] Slack "corporate-export"-import (groups, direct messages and multi-user-messages)#13942
niklasdahlheimer wants to merge 24 commits intoRocketChat:developfrom
niklasdahlheimer:slack-coporate-import

niklasdahlheimer commented Mar 28, 2019 •

edited

Loading

Uh oh!

CLAassistant commented Mar 28, 2019 •

edited

Loading

Uh oh!

niklasdahlheimer commented Apr 1, 2019 •

edited

Loading

Uh oh!

niklasdahlheimer commented Apr 11, 2019

Uh oh!

Hudell commented May 22, 2019

Uh oh!

geekgonecrazy commented May 31, 2019

Uh oh!

sampaiodiego commented Jul 2, 2019

Uh oh!

niklasdahlheimer commented Jul 10, 2019

Uh oh!

pigmonkey commented Jul 11, 2019

Uh oh!

AitorCM commented Nov 26, 2019

Uh oh!

geekgonecrazy commented Jan 29, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Conversation

niklasdahlheimer commented Mar 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The structure of the corporate-export zip-file

The structure of the dms.json

Next steps

Uh oh!

CLAassistant commented Mar 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

niklasdahlheimer commented Apr 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

niklasdahlheimer commented Apr 11, 2019

Uh oh!

Hudell commented May 22, 2019

Uh oh!

geekgonecrazy commented May 31, 2019

Uh oh!

sampaiodiego commented Jul 2, 2019

Uh oh!

niklasdahlheimer commented Jul 10, 2019

Uh oh!

pigmonkey commented Jul 11, 2019

Uh oh!

AitorCM commented Nov 26, 2019

Uh oh!

geekgonecrazy commented Jan 29, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

niklasdahlheimer commented Mar 28, 2019 •

edited

Loading

CLAassistant commented Mar 28, 2019 •

edited

Loading

niklasdahlheimer commented Apr 1, 2019 •

edited

Loading