You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The maximum number of elements an array can contain cannot exceed 4194303? #1068
For some reason I generated a large array. But I found that once the number of elements reaches a certain threshold, the array generation fails and the array is completely emptied.
Parameter Expansion.
A parameter is a variable, one or more digits, or any of the characters *, @, #, ?, -, $, and !. A variable is denoted by a vname. To create a variable whose vname contains a ., a variable whose vname consists of everything before the last . must already exist. A variable has a value and zero or more attributes. Variables can be assigned values and attributes by using the typeset special built-in command. The attributes supported by the shell are described later with the typeset special built-in command.
The shell supports both indexed and associative arrays. An element of an array variable is referenced by a subscript. A subscript for an indexed array is denoted by an arithmetic expression (see Arithmetic Evaluation below) between a [ and a ]. To assign values to an indexed array, use vname=(value ...) or set -A vname value ... . The value of all non-negative subscripts must be in the range of 0 through 4,194,303.
But I also found another interesting thing. ${#assoc[@]} counts the number of elements in an associative array. Once the value exceeds 4194303, the statistics themselves will be invalid, but in fact the elements are there.
I tried to expand it to the size of 2^54 - 1 on a 64-bit system, but I found that the dolnod structure also needs to be upgraded simultaneously, otherwise functions like traversing keys cannot be unified, and the amount of modification is very large. And there is no need, no one will use such a large array.
As a trade-off, I limited the maximum index to 2^32 - 1.
I'd rather go with something based on the method AT&T started implementing in the abandoned ksh 93v- beta, back in the day. See the relevant changes in ksh93-history repo commit 62e78f84a (note that this also contains a huge number of unrelated changes).
Currently, the number of elements and the array's flags share the same nelem member of struct Namarray; the 93v- code separated out the flags into a new int flags member of that struct, so that nelem could be used for the number of elements. The nv_putsub function also gains an extra flags argument. Given that long is at least 32 bits long, this allows up to 2147483647 elements. There might be some other relevant changes I'm missing.
But I think we can do even better and allow a 64-bit number of elements. Yes, you're right, struct dolnod (used also for things like positional parameters) also needs upgrading for that. This needs to happen anyway; after @JohnoKing's work to make ksh 64-bit ready, having a huge number of positional parameters suddenly became plausible.
Implementing these changes is a tedious and error-prone job, but these days I can use AI assistance to get it done quite quickly, so I expect these changes to make it into the next release.
I experimented with 64-bit array indexes. Indexed arrays are dense, not sparse, meaning that to allocate element x, all previous elements 0..(x-1) must also be allocated, even if unused. So allocating huge indexes is extremely slow, or the system will run out of memory. Therefore, I'm settling on INT_MAX (2147483647) as the maximum for indexed arrays.
Activity
qindapao commentedon Oct 25, 2026
Is it because the maximum number of elements of the array is a 22-bit maximum integer?
0011 1111 1111 1111 1111 1111It overflow if it exceeds the limit.
qindapao commentedon Oct 25, 2026
I found it in the manual!
https://manpages.debian.org/unstable/ksh93u+m/ksh93.1.en.html
Parameter Expansion.
A parameter is a variable, one or more digits, or any of the characters *, @, #, ?, -, $, and !. A variable is denoted by a vname. To create a variable whose vname contains a ., a variable whose vname consists of everything before the last . must already exist. A variable has a value and zero or more attributes. Variables can be assigned values and attributes by using the typeset special built-in command. The attributes supported by the shell are described later with the typeset special built-in command.
The shell supports both indexed and associative arrays. An element of an array variable is referenced by a subscript. A subscript for an indexed array is denoted by an arithmetic expression (see Arithmetic Evaluation below) between a [ and a ]. To assign values to an indexed array, use vname=(value ...) or set -A vname value ... . The value of all non-negative subscripts must be in the range of 0 through 4,194,303.
qindapao commentedon Oct 25, 2026
But I also found another interesting thing.
${#assoc[@]}counts the number of elements in an associative array. Once the value exceeds 4194303, the statistics themselves will be invalid, but in fact the elements are there.The test results are:
Expand maximum indexed array subscript limit on 64-bit systems (re: k…
Expand maximum indexed array subscript limit on 64-bit systems (re: k…
Expand maximum indexed array subscript limit on 64-bit systems (re: k…
Expand maximum indexed array subscript limit on 64-bit systems (re: k…
Expand maximum indexed array subscript limit on 64-bit systems (re: k…
qindapao commentedon Oct 27, 2026
qindapao@1a58a16
I tried to expand it to the size of 2^54 - 1 on a 64-bit system, but I found that the
dolnodstructure also needs to be upgraded simultaneously, otherwise functions like traversing keys cannot be unified, and the amount of modification is very large. And there is no need, no one will use such a large array.As a trade-off, I limited the maximum index to 2^32 - 1.
Expand maximum indexed array subscript limit on 64-bit systems (re: k…
qindapao commentedon Oct 28, 2026
I set the number of bits to 31 instead of 32 to avoid the risk of overflow during integer conversion.
qindapao@44a9ea8
McDutchie commentedon Nov 19, 2026
Thanks for the report and the commit.
I'd rather go with something based on the method AT&T started implementing in the abandoned ksh 93v- beta, back in the day. See the relevant changes in ksh93-history repo commit 62e78f84a (note that this also contains a huge number of unrelated changes).
Currently, the number of elements and the array's flags share the same
nelemmember ofstruct Namarray; the 93v- code separated out the flags into a newint flagsmember of that struct, so thatnelemcould be used for the number of elements. Thenv_putsubfunction also gains an extraflagsargument. Given thatlongis at least 32 bits long, this allows up to 2147483647 elements. There might be some other relevant changes I'm missing.But I think we can do even better and allow a 64-bit number of elements. Yes, you're right,
struct dolnod(used also for things like positional parameters) also needs upgrading for that. This needs to happen anyway; after @JohnoKing's work to make ksh 64-bit ready, having a huge number of positional parameters suddenly became plausible.Implementing these changes is a tedious and error-prone job, but these days I can use AI assistance to get it done quite quickly, so I expect these changes to make it into the next release.
qindapao commentedon Nov 21, 2026
@McDutchie If it could truly support 64-bit, that would certainly be the best outcome.
McDutchie commentedon Nov 21, 2026
I experimented with 64-bit array indexes. Indexed arrays are dense, not sparse, meaning that to allocate element x, all previous elements 0..(x-1) must also be allocated, even if unused. So allocating huge indexes is extremely slow, or the system will run out of memory. Therefore, I'm settling on
INT_MAX(2147483647) as the maximum for indexed arrays.