[TUT] Packed vs Unpacked Strings

[Archive-Poster]

Packed vs Unpacked Strings

Recommended pre-reading material if you do not understand how numbers are stored in memory: https://projectgodmode.com/custom_conte ... number.pdf

The unpacked string

An unpacked string is the type of string that you commonly (probably always) see used in AMX-X plugins. It is an array of 4-byte cells, each cell representing an individual character in the string. Each character in a string (both packed and unpacked) is represented by an integer value of 0-255. To represent an integer of 0-255, only 1 byte is needed, however, unpacked strings allocate 4-bytes per character leaving 3-bytes of unused memory per cell. Below illustrates this:

Code: Select all

A byte consists of 8-bits, below is a cell of 4-bytes or 32-bits.
This is how a character is stored in an unpacked string cell. 
Only bits 1-8 are used because that is all that is needed to store values 0-255. 

0000 0000 0000 0000 0000 0000 0000 0000 = 0
0000 0000 0000 0000 0000 0000 1111 1111 = 255

The bits in red are un-used\wasted memory. The waste is 3-bytes per cell (24-bits)

The packed string

A packed string is also an array of 4-byte cells but the memory layout is 'packed' so that all memory in the cell is used to store data. This results in the memory requirement for each character to be 1-byte (8-bits) opposed to 4-bytes (32-bits) in an unpacked string. When you declare a packed string you are still declaring an array of 4-byte cells but each cell will hold 4 characters instead of just one as it would in an unpacked string. Note: A packed string can only hold characters from a single-byte character set, such as ascii or one of the extended ascii sets

Code: Select all

This is how a 4-byte cell will appear in packed form. Each color represents 
the storage of an individual character in the string. Notice, nothing is wasted.

This is the packed version of the word "pack"

0111 0000 0110 0001 0110 0011 0110 1011

When should I use a packed string over an unpacked string?

Packed
Storing any numerical values in the range of 0 - 255
Storing ascii text

Unpacked
Storing numerical values > 255
Storing unicode text or any non-ascii standard

How do I define and manipulate a packed string in AMX-X?

The usage is almost identical to a normal unpacked string with the exception of accessing the string. You cannot print\manipulate the entire packed string as you can an unpacked. You must first unpack it and then print/format/etc.

Here are two small functions to pack and unpack a string. I don't know how 'expensive' using this unpack function on large strings will be but it's the only option for now. If anyone discovers a way to do this without an additional function please PM me.

Code: Select all

public strpack( const szUnpacked[] , szPacked[] ) 
{     
    new i;
    while((szPacked{i} = szUnpacked[i++])){}
}

public strunpack( const szPacked[] , szUnpacked[] ) 
{     
    new i;
    while((szUnpacked[i] = szPacked{i++})){}
} 

To define a string you define it as you normally would with the desired size and add the 'char' identifier after it. You use the same rule of adding 1 to the actual size needed to make room for the null terminating character. You must also include the '!' character before the beginning " when assigning the string to the variable.

Code: Select all

new szString[11 char] = !"The string"; 

To reference an individual cell in the string, it is the same as with an unpacked array but instead of using [] you use {}.

Code: Select all

Unpacked = szString[5];
Packed = szString{5}; 

But exactly how much memory does using a packed string save me?

Your memory savings will vary depending on the number and size of the strings in your plugin. I put together a plugin that can quickly determine memory usage based on the number and size of strings in your plugin. By default it is set to use packed strings, to get the result of using unpacked strings, comment the #defined PACKED_STRINGS line.

Here are some results that I came up with using a various number of strings, all with a size of 512. All results are the 'total requirements' value when compiled. I also did a test using size 33 strings and savings can start being noticed at 4+ strings.

Code: Select all

NUM_STRINGS = 1
PACKED = 19756
UNPACKED = 19088
Net savings = -668 bytes [Uses more memory when packed]

NUM_STRINGS = 5
PACKED = 21820
UNPACKED = 27296
Net savings = 5476 bytes

NUM_STRINGS = 10
PACKED = 24400
UNPACKED = 37556
Net savings = 13156 bytes

NUM_STRINGS = 20
PACKED = 29560
UNPACKED = 58076
Net savings = 28516 bytes

NUM_STRINGS = 30
PACKED = 34720
UNPACKED = 78596
Net savings = 43876 bytes

NUM_STRINGS = 50
PACKED   = 45040
UNPACKED = 119636
Net savings = 74596 bytes

NUM_STRINGS = 100
PACKED = 70840
UNPACKED = 222236
Net savings = 151396 bytes

Code: Select all

#include <amxmodx>
#include <amxmisc>

#define PLUGIN "Packed vs Unpacked"
#define VERSION "1.0"
#define AUTHOR "bugsy"

#define PACK_STRINGS

#define NUM_STRINGS    20
#define MAX_STRING_SIZE    512

#if defined PACK_STRINGS
new g_szUnpackBuffer[MAX_STRING_SIZE];
new g_szStrings[NUM_STRINGS][MAX_STRING_SIZE char]
#else
new g_szStrings[NUM_STRINGS][MAX_STRING_SIZE]
#endif

public plugin_init() 
{
    register_plugin(PLUGIN, VERSION, AUTHOR)
}

public plugin_cfg()
{
    for ( new iTest = 0 ; iTest < NUM_STRINGS ; iTest++ )
    {
        #if defined PACK_STRINGS
        g_szStrings[iTest] = !"Packed strings"    
        strunpack( g_szStrings[iTest] , g_szUnpackBuffer );
        server_print( "%d = %s" , iTest , g_szUnpackBuffer );
        #else
        g_szStrings[iTest] = "Unpacked strings";
        server_print( "%d = %s" , iTest , g_szStrings[iTest] );
        #endif
    }
}

#if defined PACK_STRINGS
public strunpack( szPacked[] , szUnpacked[] ) 
{     
    new i;

    while((szUnpacked[i] = szPacked{i++})){}
}
#endif 

Padding

Since a packed string still uses 4-byte cells, you will usually get from 1 to 3 bytes of wasted space. This is because you cannot have only a partial cell. Suppose you want to store 5 letters in a packed string. Theoretically you only need 5 bytes right? since that's what this tutorial has been explaining. Yes, you only need 5 bytes for the storage of your string but you will need 2 cells since 1 cell can only hold 4 characters. In this case we would have 4 bytes used in the first cell for the first 4 letters and then only use 1 byte in the second cell for the remaining letter, plus 1 extra cell for the null character. The left over 2 cells are referred to as 'padding' and will remain there empty holding 0's. The only time this will not occur is when your string size is divisible by 4.

Here are some sample AMX-X scripts using packed strings

Code: Select all

//This will declare a packed and unpacked string, display the values in 
//each cell (which are identical) and then display the size of each string 
//(in cells) at the end.

//The result: Same data 0 = 0 -> Same data 31 = 31
//Packed Size = 9 cells
//Unpacked Size = 33 cells
//Why is the packed 9 cells? See Padding section above.

new szPacked[33 char];
new szUnpacked[33];

for ( new i = 0 ; i < 33 ; i++ )
{
    szPacked{i} = i;
    szUnpacked[i] = i;
    server_print( "Same data %d = %d" , szPacked{i} , szUnpacked[i] );
}
    
server_print( "Packed Size = %d cells" , sizeof szPacked );
server_print( "Unpacked Size = %d cells" , sizeof szUnpacked ); 

Code: Select all

//This will print out the ascii values for each character
//in the string "Packed"
new szPacked[7 char] = !"Packed";

for ( new i = 0 ; i < 7 ; i++ )
    server_print( "%d" , szPacked{i} ); 
        
//This will print the ascii values for "Packed": 80, 97, 99, 107, 101, 100, 0
//P=80 a=97 c=99 k=107 e=101 d=100 NULL=0

Code: Select all

//Define a packed array used to store bool values for players
new g_szOnline[33 char];

public client_putinserver(id)
{
    g_szOnline{id} = 1;
}

public client_disconnect(id)
{
    g_szOnline{id} = 0;
}

public YourFunc(id)
{
    if ( !g_szOnline{id} )
        //user not online
} 

[Archive-Poster]

AMXX string functions were specially crafted to avoid the need to convert cells to chars. Doesn't converting cells to chars just to have them convert back to cells seem kinda pointless?

But that's not the only thing you can do with packed strings (to save/use strings) as you can see in the last example.

Actually conversions will happen anyway. All pawn literals are cells, therefore they need to be truncated to fit in a packed string, and must be re-converted to cells when being used in any statement, AFAIK.

EDIT: Apparently I'm /partially/ wrong, there are special opcodes for packed chars, but still, if you try to use any cell values into expressions conversions have to happen. I'll look into the Pawn manual to see if there are better explanations on the subject.

[Archive-Poster]

AMXX string functions were specially crafted to avoid the need to convert cells to chars. Doesn't converting cells to chars just to have them convert back to cells seem kinda pointless?

I know what you're getting at but packed strings can conserve memory if done correctly. You can create a single global unpacked array for a universal unpacking buffer.

It is also beneficial for having an array of bools if you choose not to use bit-fields.

Code: Select all

new g_UnpackBuffer[1024];

public Func1()
{
    new szPacked1[100 char];

    strunpack( szPacked1 , g_UnpackBuffer );
}


public Func2()
{
    new szPacked2[512 char];

    strunpack( szPacked2 , g_UnpackBuffer );
} 

[Archive-Poster]

Actually conversions will happen anyway. All pawn literals are cells, therefore they need to be truncated to fit in a pack string, and must be re-converted to cells when being used in any statement, AFAIK.

EDIT: Apparently I'm /partially/ wrong, there are special opcodes for packed chars, but still, if you try to use any cell values into expressions conversions have to happen. I'll look into the Pawn manaul to see if there are better explations on the subject.

Even if you are wrong that was a smart point. I will ask for help to amxx profiler.

Edit:

The results (It's basically the same):

Code: Select all

#include <amxmodx>
#include <fakemeta>

new normal[33]
new packed[33 char]

setNormal(x,y)
{
    normal[x] = y;
}

setPacked(x,y)
{
    packed{x} = y;
}

getNormal(x)
{
    return normal[x];
}

getPacked(x)
{
    return packed{x};
}


public plugin_cfg()
{
    for(new i=0;i<10000;i++)
    {
        new a = random_num(0,32)
        new b = random_num(0,255)
        
        setPacked(a,b)
        setNormal(a,b)
        
        getPacked(a);
        getNormal(a);
        
        setPacked(a,0)
        setNormal(a,0)
        
        getPacked(a);
        getNormal(a);
    }
    
    server_cmd("quit")
    
} 

Code: Select all

date: Tue Apr 21 07:39:32 2009 map: de_dust2
type |                             name |      calls | time / min / max
-------------------------------------------------------------------
   n |                       random_num |      20000 | 0.049776 / 0.000001 / 0.000101
   n |                       server_cmd |          1 | 0.000014 / 0.000014 / 0.000014
   p |                       plugin_cfg |          1 | 0.207914 / 0.207914 / 0.207914
   f |                        setNormal |      20000 | 0.048192 / 0.000001 / 0.000045
   f |                        setPacked |      20000 | 0.048167 / 0.000001 / 0.000092
   f |                        getNormal |      20000 | 0.048578 / 0.000001 / 0.000067
   f |                        getPacked |      20000 | 0.048613 / 0.000001 / 0.000098
0 natives, 0 public callbacks, 1 function calls were not executed.

date: Tue Apr 21 07:40:37 2009 map: de_dust2
type |                             name |      calls | time / min / max
-------------------------------------------------------------------
   n |                       random_num |      20000 | 0.049495 / 0.000001 / 0.000079
   n |                       server_cmd |          1 | 0.000013 / 0.000013 / 0.000013
   p |                       plugin_cfg |          1 | 0.207875 / 0.207875 / 0.207875
   f |                        setNormal |      20000 | 0.048272 / 0.000001 / 0.000079
   f |                        setPacked |      20000 | 0.048243 / 0.000001 / 0.000193
   f |                        getNormal |      20000 | 0.048381 / 0.000001 / 0.000220
   f |                        getPacked |      20000 | 0.048462 / 0.000001 / 0.000080
0 natives, 0 public callbacks, 1 function calls were not executed.

Code: Select all

#include <amxmodx>
#include <fakemeta>

new normal[33]
new packed[33 char]

setNormal(x)
{
    normal[x] = 33 | normal[x];
}

setPacked(x)
{
    packed{x} = 33 | packed{x};
}

getNormal(x)
{
    return 1 | normal[x];
}

getPacked(x)
{
    return 1 | packed{x};
}


public plugin_cfg()
{
    for(new i=0;i<10000;i++)
    {    
        new a = random_num(0,32)
        
        setPacked(a)
        setNormal(a)
        
        getPacked(a);
        getNormal(a);
    }
    
    server_cmd("quit")
    
} 

Code: Select all

date: Tue Apr 21 07:57:38 2009 map: de_dust2
type |                             name |      calls | time / min / max
-------------------------------------------------------------------
   n |                       random_num |      10000 | 0.026228 / 0.000001 / 0.001046
   n |                       server_cmd |          1 | 0.000011 / 0.000011 / 0.000011
   p |                       plugin_cfg |          1 | 0.104097 / 0.104097 / 0.104097
   f |                        setNormal |      10000 | 0.028161 / 0.000001 / 0.001741
   f |                        setPacked |      10000 | 0.024113 / 0.000001 / 0.000031
   f |                        getNormal |      10000 | 0.024541 / 0.000001 / 0.000118
   f |                        getPacked |      10000 | 0.024453 / 0.000001 / 0.000091
0 natives, 0 public callbacks, 1 function calls were not executed.

date: Tue Apr 21 08:05:27 2009 map: de_dust2
type |                             name |      calls | time / min / max
-------------------------------------------------------------------
   n |                       random_num |      10000 | 0.024472 / 0.000001 / 0.000101
   n |                       server_cmd |          1 | 0.000010 / 0.000010 / 0.000010
   p |                       plugin_cfg |          1 | 0.105678 / 0.105678 / 0.105678
   f |                        setNormal |      10000 | 0.024440 / 0.000001 / 0.000139
   f |                        setPacked |      10000 | 0.023782 / 0.000001 / 0.000051
   f |                        getNormal |      10000 | 0.024079 / 0.000001 / 0.000009
   f |                        getPacked |      10000 | 0.024073 / 0.000001 / 0.000031
0 natives, 0 public callbacks, 1 function calls were not executed.

[Archive-Poster]

It's good to see memory usage can be reduced without an expense to the CPU.

I have a situation for an MOTD message that has to be formatted with the plugin.
The array for it is a 2500 array and uses the "len += format(motd[len], ...)" method.
Would using packed strings for this work?

No because show_motd would still need an unpacked string.

[Archive-Poster]

If you only have one large string that you are trying to conserve space on then the packed approach will not work. Since packed strings can not be manipulated as a whole like unpacked strings can you must first unpack them into a normal unpacked string buffer. In your case this would require you to declare a packed MOTD buffer of 2500 new szMOTD[2500 char] and then a buffer to unpack it new szUnpackBuffer[2500] which is obviously only going to create waste instead of reduce it. See the revised section above titled But exactly how much memory does using a packed string save me?

[Archive-Poster]

In all the cases I could use packed strings for my large arrays such as MOTDs or menus, it wouldn't work because the functions require unpacked. So my question is this: why use packed strings if its only use is for smaller variables that wouldn't improve memory usage as much as the large arrays?

[Archive-Poster]

That is not "working with packed string". You are declaring a variable with the "char operator" but you are using it as a normal string. Basically, what you are doing is:

But it will not work. It will show just the 4th character from each cell.

Yes, I realized this literally as I pulled out of my driveway on my way to work shortly after posting; I did not have internet access until now to edit my post. The above code only appeared to work since my code example had such a large buffer so there were enough cells to properly display the string.

What is happening is the array is getting declared with char so that will trim the cell count down to [1024 / 4 = 256]. When passed to format, it is treated as a normal 256 sized buffer [unpacked] and is able to be printed\manipulated. If you trim the buffer down to the actual size and try to format, your server will error.

Code: Select all

public plugin_init() 
 {
     register_plugin( "pack test" , "0.1" , "bugsy" )
     
     //Will error the server
     /// new szTest[22 char];
     
     //Will work as normal
     // new szTest[22];
     
     //Will cause the 'g' to be cut off of the word "string"
     new szTest[84 char];

     format( szTest , sizeof szTest - 1 ,"This is a test string" )
     
     server_print("%s . Size=%d" , szTest , sizeof szTest );