@hbina i know certain specific string sizes which once had to use 16 bit header sds, will now fit into 8 bit header sds. but i'm not sure the extra complexity in the code justifies that improvement.
what's more, i also think there's a chance for performance degradation due to two things: one is the extra operations that are done (shift and or), and the other is the restructure of the into separate blocks to handle each field.

let's consider the memory saving in this case (most significant one):
some strings that are over 255 bytes length which was in the past forced to use 16 bit sds header, will not be able to fit into the 8 bit sds header.
in the most extreme case, you're saving 2 bytes of memory out of a string that's anyway longer than 255 bytes.
truth be told, in some cases this 2 bytes reduction can cause the allocation to use a smaller bin size from the allocator (reducing some 30% of internal fragmentation too), but all in all i don't think the complexity and possible performance hit is worth it.

for the bigger sizes it's even less rewarding (saving 4 bytes out of 65k string)

@oranagra thanks for looking at this! I am new here >.<

some strings that are over 255 bytes length which was in the past forced to use 16 bit sds header, will not be able to fit into the 8 bit sds header.
in the most extreme case, you're saving 2 bytes of memory out of a string that's anyway longer than 255 bytes.

I think (from reading above) you are misunderstanding what my change will do?
This basically changes from using 8/16/32-bits for len/alloc to using 10/18/34-bits.
Meaning that 255 bytes length will definitely fit into this.
In fact, "SDS_TYPE_8" can now store up to 1023 worth of characters.

Notice that I shift the additional bits in the flags to make it a power of (9, 10), (17, 18), and (33, 34) approximately.

what's more, i also think there's a chance for performance degradation due to two things: one is the extra operations that are done (shift and or), and the other is the restructure of the into separate blocks to handle each field.

I think the implementation can be made simpler by simply rethinking what bits are used for len/alloc.
Why didn't I think of this earlier...god

--------------------------------------------------------------------------------------------------
| 8 -bits of len | 8 -bits of alloc |  5-bit unused | 3-bits of flag | ....
--------------------------------------------------------------------------------------------------

to

--------------------------------------------------------------------------------------------------
| 10 -bits of len | 10 -bits of alloc | 1-bit unused | 3-bits of flag | ....
--------------------------------------------------------------------------------------------------

As long its contained within this file and all access be done through helper functions...it should be fine.

@hbina i may be missing your intent indeed (didn't carefully read the changes).
but looking at your response i actually don't think i missed it.

this PR is about memory saving, right?
what is the smallest string that would be affected by your change? and how many bytes will it save?
if i'm not mistaken, the smallest string that will be affected is 256 bytes, and you'll save 2 bytes.
this 2 bytes saving will hold for strings up to 1023 bytes, and anything above it can largely be ignored (saving 4 bytes for string over 16kb is not a real concern).

if i'm still missing your point please correct me.

@oranagra Oh okay, yes, I understand now. What you are claiming is that my changes will have diminishing returns in memory saving as the string gets bigger. Yes, I agree with that. But there's also other benefits from increasing the range of SDS because of this

While are are at it, and I don't want to open an issue just for this, what do you think about

eagerly creating the strings instead of lazily doing so? I think it could be improved to do basically 0 wasted allocations.

@hbina regarding this memory optimization, it's not only that it's diminishing as strings get bigger, even on the smallest string that it affects, the benefit is not that big (considering the size of the string). this is what we avoided going that way in the past.

regarding size class conversion in sdsMakeRoomFor and also in sdsRemoveFreeSpace, i think this is an edge case that we don't want to consider.
if we do want to go that way, we may even want to just avoid changing size class in certain cases when not necessary (like keep using sds16 when shrinking a string from 260 to 240, or pre-allocating an sds16 when the string is 240 since it might soon be expanded).

note that the above waste of bytes in order to reduce the need for size class conversion goes against the main topic of this PR (to reduce space).

if we are worried about that conversion, the most important thing to do is completely drop the use of sds16 and sds32, and always use sds64, since a conversion of a big string is a lot more painful, and the waste of 16 bytes is not is not that bad.
however, keeping in mind that even when not changing a size class, and doing a simple realloc within the same size class, can result in a memory copy (in case the allocator decided to change bin), and in other cases, can be a NOP.

Sounds reasonable. Thanks for looking into this regardless :)