@olihey any ideas how we could make this work?

Changing the resource url in the rclone.conf will not work as the access token is only valid for ONE resource url.

But we use the discovery endpoint to get all available services the user has access to and then automatically choose the OneDrive drive. Maybe we could present the list to the user during config and let him choose?!?! Maybe pre-select the OneDrive?

+1

I have created a pull request here #1780
This will ask the user for the correct resourceURL if the discovery service returns more than one resource URL.
BUT I couldn't test it as I only have a single OneDrive Business account.

Find this fix in the latest beta - please can you have a test and report back.

I tested but it does not work.
It is the same behavior than before. Output only the result_url https://orgname-my.sharepoint.com/.

Can you please try to setup the endpoint and use the "-v" when starting rclone?
You should see some debug messages and after "Got code" it should list all services Microsoft's discovery services returns for your user.
Mine looks like this:

Got code
2017/11/07 08:12:30 INFO  : Skipping API v1.0 endpoint https://XXXXXXX-my.sharepoint.com/_api/v1.0/me
2017/11/07 08:12:30 INFO  : Skipping API v2.0 endpoint https://XXXXXXX-my.sharepoint.com/_api/v2.0/me
--------------------

Here's what I got:

Waiting for code...
Got code
2017/11/07 08:35:02 INFO  : Skipping API v1.0 endpoint https://XXXXX-my.sharepoint.com/_api/v1.0/me
2017/11/07 08:35:02 INFO  : Skipping API v2.0 endpoint https://XXXXX-my.sharepoint.com/_api/v2.0/me

After that it dumped the config output for the remote site. There were no prompts for the correct resourceURL.

As far as I can tell, this remote is now pointing to my personal space in the business one drive.

Thanks for the feedback, unfortunately I only have a OneDrive Business account and NO additional Team sites so I can't test it.

I have created a commit in my repo which let a user type in a resource URL before getting the final access token. You can build that commit and see if typing in works for you.

Yes, that is the same output that I get when I discover.
Got code
2017/11/07 09:16:36 INFO : Skipping API v1.0 endpoint https://xxxxx-my.sharepoint.com/_api/v1.0/me
2017/11/07 09:16:36 INFO : Skipping API v2.0 endpoint https://xxxxx-my.sharepoint.com/_api/v2.0/me

The sites from the organization that we are trying to access are not displayed there.
My organization does not have something like this:
https://orgname.sharepoint.com/
https://orgname.sharepoint.com/Shared Documents

All sites are team sites like this:
https://orgname.sharepoint.com/site/site-name

Hi, could anyone please confirm if this was resolved. I downloaded the latest beta yesterday. I tried the webdav option to my sharepoint/teams files and also used the business one drive option and manually changed the resource_url and attempted to validate the token again.

I couldn't get either option to work. Happy to help test further if needed.

Hej, the latest beta only uses the discovery service for getting resource urls. Unfortunately the discovery service does not "see" the sharepoint sites/team sites.
As I said before, I only have one Business account that isn't connected to any sharepoint sites, so I can't test it. Changing the resource url will NOT work, as the refresh token in the rclone.conf is only valid for the resource_url specified in the config file. Changing the URL invalidates the refresh token.

I made a change in my forked repo where you get the option to input an arbitrary resource_url during setup of the remote.

Can you try to get this branch compiled and run?

I used the following command to build your repo:-
go get -u -v github.com/olihey/rclone

It is not prompting me to enter a resource _url when creating a onedrive business remote though.

Is there something different I need to do to build via go?

I have subsequently tried cloning your git fork and switch to the onedrive_resoure_select branch. I then used go build to check that branch and it created a new rclone file. But I must still be doing something wrong as I am pretty sure it is not using onedrive.go file from that branch.

I no very little about go projects, so not able to troubleshoot further

What OS are you on? Maybe I can create a build for you to test.

Debian stretch 64 bit.

Is there something I can read up to build properly via go. Or is it the git part I might be getting wrong?

OKay, I build an executable on my Ubuntu 16.04 AMD64, you can get it here: https://drive.google.com/open?id=1IsNSE34QK_UO-qzgSv2wuedxwsiMqlCZ

Please be aware that this test build currently can't do "headless" authorization, you have to use a machine which can show a web browser

Thanks. I have requested access to the file, as not a public share from Google Drive

Sorry, not that skilled with Google Drive. Try this: https://drive.google.com/file/d/1IsNSE34QK_UO-qzgSv2wuedxwsiMqlCZ/view?usp=sharing

Did not prompt me output shown below

tim@somepc:~$ ./rcloneodb -V
rclone v1.38-DEV

• os/arch: linux/amd64
• go version: go1.9.2

tim@somepc:~$ ./rcloneodb -vv config
2017/12/08 11:26:07 DEBUG : Using config file from "/home/tim/.config/rclone/rclone.conf"
2017/12/08 11:26:07 DEBUG : rclone: Version "v1.38-DEV" starting with parameters ["./rcloneodb" "-vv" "config"]
Current remotes:

Name Type
==== ====
gDrive drive
oneTest onedrive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> oneAsk
Type of storage to configure.
Choose a number from below, or type in your own value
1 / Amazon Drive
\ "amazon cloud drive"
2 / Amazon S3 (also Dreamhost, Ceph, Minio)
\ "s3"
3 / Backblaze B2
\ "b2"
4 / Box
\ "box"
5 / Cache a remote
\ "cache"
6 / Dropbox
\ "dropbox"
7 / Encrypt/Decrypt a remote
\ "crypt"
8 / FTP Connection
\ "ftp"
9 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
10 / Google Drive
\ "drive"
11 / Hubic
\ "hubic"
12 / Local Disk
\ "local"
13 / Microsoft Azure Blob Storage
\ "azureblob"
14 / Microsoft OneDrive
\ "onedrive"
15 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
16 / Pcloud
\ "pcloud"
17 / QingClound Object Storage
\ "qingstor"
18 / SSH/SFTP Connection
\ "sftp"
19 / Webdav
\ "webdav"
20 / Yandex Disk
\ "yandex"
21 / http Connection
\ "http"
Storage> 14
Microsoft App Client Id - leave blank normally.
client_id>
Microsoft App Client Secret - leave blank normally.
client_secret>
Remote config
Choose OneDrive account type?

• Say b for a OneDrive business account
• Say p for a personal OneDrive account
  b) Business
  p) Personal
  b/p> b
  Use auto config?
• Say Y if not sure
• Say N if you are working on a remote or headless machine
  y) Yes
  n) No
  y/n> y
  2017/12/08 11:27:55 DEBUG : Starting auth server on 127.0.0.1:53682
  If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
  Log in and authorize rclone for access
  Waiting for code...
  2017/12/08 11:28:11 DEBUG : Received request on auth server
  2017/12/08 11:28:11 DEBUG : Successfully got code
  Got code
  2017/12/08 11:28:12 DEBUG : oneAsk: Added new token to config, still needs to be saved
  2017/12/08 11:28:12 DEBUG : Closing auth server
  2017/12/08 11:28:12 DEBUG : Closed auth server with error: http: Server closed
  2017/12/08 11:28:15 INFO : Skipping API v1.0 endpoint https://xyxyxy-my.sharepoint.com/_api/v1.0/me
  2017/12/08 11:28:15 INFO : Skipping API v2.0 endpoint https://xyxyxy-my.sharepoint.com/_api/v2.0/me
  2017/12/08 11:28:15 DEBUG : oneAsk: Added new token to config, still needs to be saved

[oneAsk]
client_id =
client_secret =
token = {"access_token":"my secret token","expiry":"2017-12-08T12:28:11.494924Z"}
resource_url = https://xyxyxy-my.sharepoint.com/

y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name Type
==== ====
gDrive drive
oneAsk onedrive
oneTest onedrive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config

Hej, sorry for the long "pause". Problem is, I only have a Office 365 account for testing. Meaning this is very hard for me to fix without having anyway to try it. Sorry.
Maybe someone with such an account can help?

Can someone please go to this page https://developer.microsoft.com/en-us/graph/graph-explorer, authenticate your account and run the URL: https://graph.microsoft.com/v1.0/me/drives left to the "Run Query" button (you have to click the button after entering the URL).

Do you see your sharepoint / team sites in the "Response Preview" section?

I only see my personal one drive account. The team site is not returned in the response.

Looking at that graph explorer, there are several examples that deal with sharepoint. So far I was able to find the shared site by using the request

https://graph.microsoft.com/v1.0/sites?search=my_keyword

Where my_keyword is part of the name for the shared site.

I'm still looking to see if there are ways to list shared sites (it wasn't in https://graph.microsoft.com/v1.0/sites/root/drives). However, maybe you could use the search and prompt the user for the keyword?

Thanks!

I'm sure you've seen this:

https://developer.microsoft.com/en-us/graph/docs/api-reference/v1.0/api/site_get

But I couldn't find anyway to list the sharepoint sites I have access to. Maybe GET /groups/{group-id}/sites/root would work but I don't have permission to lists the groups I am a member of (/me/memberOf).

So it looks search may be the only way to go.

I haven't found a way to properly query for available sites per user. I can see that I can access them by "id". Do you have any chance to see the ID for the drives you want to access?
One on mine looks like this: b!9Z8pPhaBr06tcI6zsoXXXXXXXXXXXXXXXXD2yjX1ayLDuZSTKTwRAieJlnL

Because then I could add an option during setting up the remote to type in the ID by yourself.

Yes there is an id returned for the site in the response to

https://graph.microsoft.com/v1.0/sites?search=my_keyword

However, the id is in the form:

"id": "mycompany.sharepoint.com,########-####-####-####-############,########-####-####-####-############","

Where # is a hex digit. I see the id in the form you list above from the "me/drives" query. It appears sharepoint uses a different form. Don't know if that means it's incompatible with the access by id method you mentioned.

Hello,

I've had a hard time setting up a remote to my university sharepoint site. The default config procedure only offered my university-provided personal OneDrive, upecnumerique-my.sharepoint.com/personal/[mail address], but not the Sharepoint site, upecnumerique.sharepoint.com/sites/[name].
The Graph Explorer gave no result with keyword "upecnumerique".

I opened the Sharepoint site HTML source code and looked for IDs. Given the above comment #1770 (comment) and some trial-and-error, I found the ID is upecnumerique.sharepoint.com,[siteId],[webId] (I tested this by inserting the ID in request https://graph.microsoft.com/v1.0/sites/[ID]/drive/root/children as advised above).

The sharepoint site "contains" many folders. I thought rclone config would add the site as a remote, but instead each folder is seen as a drive and I have to choose only one among the 57 of them :-/ Fortunately I only need 4 of them, so I can cope with 4 separate remotes.

Thank you for the great software.

@llevrel I'm a little lost in your words. Are you looking for some way to add the whole site as a drive? I doubt if it's possible because if you look at the response of https://graph.microsoft.com/v1.0/sites/[ID]/drives, you'll see those "folders" (if I get your point) are really different drives.

@Cnly I think you got my point. I'm new to this so "sites", "drives", etc. are vague concepts for me. Anyway I understand my wish cannot be fulfilled. Never mind, as I need only 4 of them it's no big deal. Thank you for answering!

@llevrel In my research of this problem, it seems like this is more of a shortcoming of the graph API and its permissions model rather than rclone. Unfortunately, I don't see any feasible way to support this.

Every resource that rclone would require in order to "search" for team sites or sub-sites within sharepoint requires some form of admin privilege, even for you to check your own drives. You can probably thank Microsoft's absurd permissions model which doesn't distinguish an individual's resources from a team's resources. In order to list sites/drives for team sites and sub-sites that you have been granted access to, you need the Sites.ReadAll and User.ReadAll permissions according to the documentation, which is admin-only. To add content, you would need to elevate to the ReadWrite permission respectively.

https://docs.microsoft.com/en-us/graph/api/user-list-joinedteams?view=graph-rest-1.0

Rclone would either need to be granted permission to an organization's entire root site's resources as an application (a massive security hole), or you would need to have your user added as an administrator to grant yourself permission to your own team site, which would also grant rclone permission to ALL team sites under your credentials (impractical, and yet another security hole).

This is the sad story that is the Microsoft Graph API, which isn't much better than Windows permissions 😞

One can lookup site ID manually by using https://developer.microsoft.com/graph/graph-explorer/ and login with default permissions. After that she will be able to lookup site by name with request like https://graph.microsoft.com/v1.0/sites/mysite.sharepoint.com:/sites/my-team and received ID can be used by rclone.

Don't see any particular reason why rclone can't lookup site my-team by it's name in the same way.

But anyway thanks for the current implementation. Please let me know if I can help you with test(s).

@mkoegel wrote:

Version 1.41 should support SharePoint via webdav, but the docu is not updated yet. Did someone get this to run?

You can find the docs here: https://rclone.org/webdav/#sharepoint

@heerohawwah
Those "DEBUG : Couldn't decode error response: EOF" messages mean that the remote end returned an HTTP error rather than an XML Webdav error most likely. You can ignore them - they aren't important.

I wonder if the "Failed to copy: object not found" errors are caused by the rclone uploading the file, then going to check it but finding it isn't there yet due to some eventual consistency stuff.

If you retry the sync, does it clean up any it missed OK?

You saved me with the webdav option (that I obviously wan't aware of)!
Spent s*****ad of time trying to configure my Onedrive for Business personal account in vain!

One can lookup site ID manually by using https://developer.microsoft.com/graph/graph-explorer/ and login with default permissions. After that she will be able to lookup site by name with request like https://graph.microsoft.com/v1.0/sites/mysite.sharepoint.com:/sites/my-team and received ID can be used by rclone.

To provide a bit more detail:

• Go to the Microsoft Graph Explorer and sign in with your account (the pane on the left says something about "you're currently using a sample account")
• Query https://graph.microsoft.com/v1.0/sites/mysite.sharepoint.com:/sites/my-team, substituting in your own site URL (for mysite.sharepoint.com) and site name (for my-team). Your site name is apparent in the URL when you're simply browsing your team SharePoint site
• Copy and paste the "id" returned in the response as the "siteid" in the rclone config

The above did not work for me for some reason. The search however worked great!

As above go to the MS Graph Explorer and authenticate.

Then get the siteID by searching part of the site name with the below url:
https://graph.microsoft.com/v1.0/sites?search={your query}
Finally use option 4 to add the site and paste the id there.

There is a bug in the search in rclone:

Choose a number from below, or type in an existing value
 1 / OneDrive Personal or Business
   \ "onedrive"
 2 / Root Sharepoint site
   \ "sharepoint"
 3 / Type in driveID
   \ "driveid"
 4 / Type in SiteID
   \ "siteid"
 5 / Search a Sharepoint site
   \ "search"
Your choice> 5
What to search for> xxx000
2020/01/08 16:10:30 Failed to query available sites: HTTP error 403 (403 Forbidden) returned body: "{\r\n  \"error\": {\r\n    \"code\": \"accessDenied\",\r\n    \"message\": \"The caller does not have permission to perform the action.\",\r\n    \"innerError\": {\r\n      \"request-id\": \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\",\r\n      \"date\": \"2020-01-08T16:10:30\"\r\n    }\r\n  }\r\n}"

If you do it manually via the Graph API it works and returns the siteID in the id field.

2020/01/08 16:10:30 Failed to query available sites: HTTP error 403 (403 Forbidden) returned body: "{\r\n "error": {\r\n "code": "accessDenied",\r\n "message": "The caller does not have permission to perform the action.",\r\n "innerError": {\r\n "request-id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",\r\n "date": "2020-01-08T16:10:30"\r\n }\r\n }\r\n}"

That says access denied... Is it a user permissions problem?

Nope. It's a missing Rclone permission maybe? The graph api explorer works for the same account and same query.

It looks like the app is not requesting the Sites.Read.All permission.

Sites.Read.All should be sufficient for search (https://docs.microsoft.com/en-us/graph/api/site-search?view=graph-rest-1.0&tabs=http) and does not require admin consent.

Confirmed. Search works if I add "Sites.Read.All" to the backend and to the azure app.

@telepath wrote:

Confirmed. Search works if I add "Sites.Read.All" to the backend and to the azure app.

Ah ha!

@Cnly what do you think about adding this to the requested scopes?

@ncw Although I myself haven't seen this problem before, this seems a plausible fix. Does @telepath want to open a PR for it? :)

I've created a pull request, though I'm not sure if any documentation or tests are missing.

@telepath I think we'd better add the permission under the docs here: https://github.com/rclone/rclone/blob/master/docs/content/onedrive.md#getting-your-own-client-id-and-key. And can you confirm whether the new scope takes effect immediately with the code change, or only after the user is shown the consent page again?

New consent ist required.

I've merged @telepath 's fix for this to master now which means it will be in the latest beta in 15-30 mins and released in v1.51

When using rclone 1.51.0 and doing a "Search a Shaepoint site" I got:

2020/05/07 11:55:52 Failed to query available sites: HTTP error 503 (503 Service Unavailable) returned body: "{\r\n  \"error\": {\r\n    \"code\": \"serviceNotAvailable\",\r\n    \"message\": \"Service unavailable\",\r\n    \"innerError\": {\r\n      \"request-id\": \"d3090cd6-acec-4573-b5bd-5dc088e6269d\",\r\n      \"date\": \"2020-05-07T09:55:52\"\r\n    }\r\n  }\r\n}"

When using "Type in SiteID", it works ok.

I also get the same error as @jrd when trying to use the "Search a SharePoint Site" feature
Did the API/namespace change a little bit perhaps?

Maybe this issue should get re-opened?

@jrd @mikey32230 The HTTP 503 error is usually temporary and should go away if you retry some time later; I didn't find any changes in the API. Alternatively if you download the latest beta, you can now also choose the newly added Sharepoint site name or URL (e.g. mysite or https://contoso.sharepoint.com/sites/mysite) option so you don't need to search.

The request for Sites.Read.All permission should be explicitly mentioned in the document. I've run into permission issues on a domain with an excessively strict authorization policy (my university). If the domain disallow users to consent permissions on their own and the Sites.Read.All permission is undocumented thus not configured, the extra Sites.Read.All scope when requesting an access token will trigger "Application needs permission to access resources in your organization that only an admin can grant. Please ask an admin to grant permission to this app before you can use it." message.

I've created a version of rclone with an advanced config entry to decide whether to contain the Sites.Read.All scope: https://github.com/cqjjjzr/rclone/tree/fix-onedrive-sites