Wshadow statistics:
1 coinsbyscript.cpp:180:2200: warning: declaration shadows a local variable [-Wshadow]

@paveljanik - Thanks! Fixed that up. Looks like that's the only shadow warning.

Thanks for reviving this. IMO this is important.

Concept ACK, are there any statistics on how much extra burden this places on an ordinary node with the index off?

needs rebase

@instagibbs - Rebased. Thanks for the heads up. (Side note: Is there any way to have GH tell you when a conflict occurs? That would be really handy.)

@sidhujag - Good question. I don't know offhand. I'm happy to do some benchmarking. I may need some help with that. If anybody would like to make some suggestions, I'm all ears.

All - Will get more movement on this. Life intervened for awhile and has finally slowed down enough to where I can dedicate more time to this.

Thanks droark great feature btw.. this makes walletless spending very easy

Am placing a to-do list here to remind myself of what I need to do, and solicit feedback on anything people might think is missing.

• Remove all Boost code from the PR.
• Address feedback from txoutsbyaddress index (take 2) #8660 (primarily @ryanofsky). (Mostly done but I want to double check a couple of things before proceeding.)
• Add some C++ tests for CCoinsView classes.
• Probably switch names of this & that per the suggestion of @gmaxwell. ("utxoindex" is my current choice but I'm not attached to it.)
• Add a default constructor value for CTxMemPool. (A recently merged PR, combined with this PR, inadvertently nukes the default constructor.)
• Check in once a day to see if a rebase is needed. (Ongoing task.)

needs rebase.

@btcdrak - Thanks. Rebased.

Running this on a .bitcoin/ folder that was last used with 0.13 and was run pruned to 2000. It seems 1GB of ram is too low for this enabled as I got an out of memory error. Now on restarting I get the following after/during the "Rescanning..." step:

~/.bitcoin# bitcoind: coinsbyscript.cpp:53: CCoinsByScript& CCoinsViewByScript::GetCoinsByScript(const CScript&, bool): Assertion `it != cacheCoinsByScript.end()' failed.

Thanks for preparing this @droark. Is this PR still alive?

@dexX7 - Thanks for checking in. Some family-related issues came up recently that I had to deal with for awhile. I'm almost done rebasing the PR and will catch up with the remaining feedback ASAP, along with some test harness changes that probably need to be folded in. The path forward looks pretty clear to me, IMO. I just need to wrap up the work.

I still need to read more, but...

Regarding searching by address , by scriptPubKey, or by COutPoint (tx_id, output_pos), I'm not sure whether I want them all or a subset of them.
The later doesn't require any further index.
Searching by scriptPubKey only requires an additional scriptPubKey -> COutPoint index but not much logic.
Searching by address on the other hand...there's many types of addresses and people still propose new better ones...I wonder if perhaps the translation address -> scriptPubKey (which shouldn't require an extra index on top of the scriptPubKey one) belongs to an upper layer like the wallet and/or ui that the users of this feature implement. I'm like supporting it here, but maybe script/standard needs to be extended for the address -> scriptPubKey translator (sorry to reiterate, but I don't think we want an address index, just a one way translation function [the other way is for wallets and relay policies, or higher level protocols ala coinjoin and whatnot]).

This all assumes you create a new scriptPubKey -> COutPoint index.

Regarding utxo vs stxo vs txo...
You can easily serve both utxo and stxo (that is, the whole txo) by just calling GetTransaction() (in validation.o) like getrawtransaction does, without needing any consideration on whether the the output is spent or not while getting the better performance if the COutPoint happens to be in the current utxo, which is very nice.
That will just complain if you search when you are pruning or not using -txindex and thus you cannot serve certain outputs unless you do a rescan which is completely out of the question.
You probably want a couple of options -utxoscriptindex and -txoscriptindex or something of the sort.
Having -utxoscriptindex but not -txoscriptindex maybe should force the error when searching for spent txo's. Having both is just reduntant. Having -txoscriptindex implies having the full historic scriptPubKey -> COutPoint index.
You don't even need to remove the spent entries from the index if you only have -utxoscriptindex if you make sure the outputs fail when -utxoscriptindex is set but -txoscriptindex is not. In that case, garbage collecting or "is spent reference counting" or "worrying about reorgs" or "adding a spent bool to the index" would probably be nice additions to have since that selection could prune them and otherwise they will accumulate over running time, even if they're just entries on an index and not the outputs themselves. That seems like it could be an important distraction in the short term.

So although I'm not against gettxoutsbyaddress and I celebrate concept ACK it, but I think it would be nicer to get gettxoutsbyoutpoint and gettxoutsbyscript reviewed and merged in that order first. But I think some other people don't like chained PRs all that much, so just take it as "if we're offering gettxoutsbyaddress, we should be able to offer gettxoutsbyoutpoint and gettxoutsbyscript 'almost for free'(tm) too".

It may seem contradictory: I'm asking you to do more with respect to utxo/stxo, but I'm suggesting to do less with respect to addresses and properly removing things for the users that want this feature but don't want to be full archives, which is something we can optimize further later after leveraging the easy important one, which is just looking in the utxo first.
The way I see it, the utxo is just the subset of the txo that can't be pruned, we happen to cache very efficiently, so it is nice to always search there first just in case and well, people usually care less about stxo because it is already spent...but is simple to also serve it or just complain when you can't or you are not expected to, sorry to reiterate.

Anyway, feel free to note the parts of my long post you like in whatever order you like best and ignore the rest. I plan to have a deeper look either way beyond this concept ack.

All the best but needs rebase.

Sorry, no, using GetTransaction doesn't leverage the utxo index we maintain...One would need to search in the utxo (with CCoinsViewCache::AccessCoin) first and then try with GetTransaction only if we want to also serve stxo, which should perhaps be left out of scope for this PR as a later improvement.

Sorry for the long post, not for being long but for being incorrect.

There's not need for a gettxobyoutpoint or similar because it already exists and is named gettxout!
That means everything I said about "optimize later, it shouldn't be hard" turns into "it's already done".

Only the slower parts were missing, which I plan to serve in #10822 which is just a draft that needs lots of testing.

To reiterate, the important parts of my previous fedback:

1. I'm more than happy to extend whatever you do for utxo for txo too, ideallly almost transparently
2. Don't do an address index, do a more generic script (that means binary or lower level, hex strings on rpc) to outpoint index, and once you have the outpoint you have the whole txo (unless you don't tcindex or you prune).
3. Create a function to translate "an address" into a script so that it can be easily searched for in the outpoint -> script index. There will always be new address types your index doesn't explicitly support yet, make them support your index instead.

The PR that should help extend whatever is done for utxo to txo if you do the index: outpoint -> script; is the following:

#10822

As said it needs testing, but there's no need to wait, let's expose utxo if we can first and then txo "almost for free"(tm).

Sorry again for invading the pr in a distractive way, but this needs rebase.

What's the status of this PR, needs more review and testing? General concept ok?

I think it would be useful to have an index on the UTXO set for the purposes of supporting external and hardware wallets.

Closing as 'up for grabs'. PR author hasn't checked in for about a year.

@droark - if you still want this, feel free to re-open and rebase!

Picked up in " Add address-based index (attempt 4?) #14053 "

Picked up in " Add address-based index (attempt 4?) #14053 "

I think those are different, right? this an indeex over the UTXO set, while that is an index over all addresses in the block chain? Or do I get this wrong.

Indeed looks like it. I wonder why it is called take 4 then.

Though, scanning the utxo set can already be done with scantxoutset, no?

Though, scanning the utxo set can already be done with scantxoutset, no?

Absolutely, that does the same thing. Though it can be pretty slow, which can be awkward if it's part of some user flow, so UTXO indexing is not completely out of the picture probably.