Skip to content

The maximum number of elements an array can contain cannot exceed 4194303? #1068

Open

Description

@qindapao

For some reason I generated a large array. But I found that once the number of elements reaches a certain threshold, the array generation fails and the array is completely emptied.

Here is the test script:

#!/usr/bin/env ksh

function test_big_arr {
    typeset -a big_arr1 big_arr2
    typeset -i i

    for((i=0;i<4194303;i++)) ; do
        big_arr1[${#big_arr1[@]}]=1
    done

    echo "big_arr1 num: ${#big_arr1[@]}"

    for((i=0;i<4194304;i++)) ; do
        big_arr2[${#big_arr2[@]}]=1
    done

    echo "print big_arr2"
    typeset -p big_arr2
    echo "big_arr2 num: ${#big_arr2[@]}"
}

print "===================  .sh.version : ${.sh.version}"
print "===================  OS / uname  : $(uname -a)"

test_big_arr

exit 0

This is the terminal printout after the script is executed:

===================  .sh.version : Version AJM 93u+m/1.1.0-alpha 2026-10-22
===================  OS / uname  : MINGW64_NT-10.0-26100 DESKTOP-0MVRMOU 3.6.5-0eeda3e1.x86_64 2025-10-10 14:51 UTC x86_64 Msys
big_arr1 num: 4194303
print big_arr2
typeset -a big_arr2=()
big_arr2 num: 0

Activity

  1. qindapao commented on Oct 25, 2026

    @qindapao
    Author

    Is it because the maximum number of elements of the array is a 22-bit maximum integer?

    0011 1111 1111 1111 1111 1111

    It overflow if it exceeds the limit.

  2. qindapao commented on Oct 25, 2026

    @qindapao
    Author

    I found it in the manual!

    https://manpages.debian.org/unstable/ksh93u+m/ksh93.1.en.html

    Parameter Expansion.
    A parameter is a variable, one or more digits, or any of the characters *, @, #, ?, -, $, and !. A variable is denoted by a vname. To create a variable whose vname contains a ., a variable whose vname consists of everything before the last . must already exist. A variable has a value and zero or more attributes. Variables can be assigned values and attributes by using the typeset special built-in command. The attributes supported by the shell are described later with the typeset special built-in command.

    The shell supports both indexed and associative arrays. An element of an array variable is referenced by a subscript. A subscript for an indexed array is denoted by an arithmetic expression (see Arithmetic Evaluation below) between a [ and a ]. To assign values to an indexed array, use vname=(value ...) or set -A vname value ... . The value of all non-negative subscripts must be in the range of 0 through 4,194,303.

  3. qindapao commented on Oct 25, 2026

    @qindapao
    Author

    But I also found another interesting thing. ${#assoc[@]} counts the number of elements in an associative array. Once the value exceeds 4194303, the statistics themselves will be invalid, but in fact the elements are there.

    #!/usr/bin/env ksh
    
    function test_assoc_len {
        typeset -A big_assoc
        typeset -i i
    
        for((i=0;i<4194305;i++)) ; do
            big_assoc[$i]=1
        done
    
        big_assoc[x1]=1
        big_assoc[x2]=1
        big_assoc[x3]=1
        big_assoc[x4]=1
    
        echo "key x1: value: ${big_assoc[x1]}"
        echo "key x2: value: ${big_assoc[x2]}"
        echo "key x3: value: ${big_assoc[x3]}"
        echo "key x4: value: ${big_assoc[x4]}"
        echo "key 4194304: value: ${big_assoc[4194304]}"
    
        echo "big_arr2 num: ${#big_arr2[@]}"
    }
    
    print "===================  .sh.version : ${.sh.version}"
    print "===================  OS / uname  : $(uname -a)"
    
    test_assoc_len
    
    exit 0

    The test results are:

    ===================  .sh.version : Version AJM 93u+m/1.1.0-alpha 2026-10-22
    ===================  OS / uname  : MINGW64_NT-10.0-26100 DESKTOP-0MVRMOU 3.6.5-0eeda3e1.x86_64 2025-10-10 14:51 UTC x86_64 Msys
    key x1: value: 1
    key x2: value: 1
    key x3: value: 1
    key x4: value: 1
    key 4194304: value: 1
    big_arr2 num: 0
  4. qindapao commented on Oct 27, 2026

    @qindapao
    Author

    qindapao@1a58a16

    I tried to expand it to the size of 2^54 - 1 on a 64-bit system, but I found that the dolnod structure also needs to be upgraded simultaneously, otherwise functions like traversing keys cannot be unified, and the amount of modification is very large. And there is no need, no one will use such a large array.

    As a trade-off, I limited the maximum index to 2^32 - 1.

  5. qindapao commented on Oct 28, 2026

    @qindapao
    Author

    I set the number of bits to 31 instead of 32 to avoid the risk of overflow during integer conversion.

    qindapao@44a9ea8

  6. McDutchie commented on Nov 19, 2026

    @McDutchie

    Thanks for the report and the commit.

    I'd rather go with something based on the method AT&T started implementing in the abandoned ksh 93v- beta, back in the day. See the relevant changes in ksh93-history repo commit 62e78f84a (note that this also contains a huge number of unrelated changes).

    Currently, the number of elements and the array's flags share the same nelem member of struct Namarray; the 93v- code separated out the flags into a new int flags member of that struct, so that nelem could be used for the number of elements. The nv_putsub function also gains an extra flags argument. Given that long is at least 32 bits long, this allows up to 2147483647 elements. There might be some other relevant changes I'm missing.

    But I think we can do even better and allow a 64-bit number of elements. Yes, you're right, struct dolnod (used also for things like positional parameters) also needs upgrading for that. This needs to happen anyway; after @JohnoKing's work to make ksh 64-bit ready, having a huge number of positional parameters suddenly became plausible.

    Implementing these changes is a tedious and error-prone job, but these days I can use AI assistance to get it done quite quickly, so I expect these changes to make it into the next release.

  7. qindapao commented on Nov 21, 2026

    @qindapao
    Author

    @McDutchie If it could truly support 64-bit, that would certainly be the best outcome.

  8. McDutchie commented on Nov 21, 2026

    @McDutchie

    I experimented with 64-bit array indexes. Indexed arrays are dense, not sparse, meaning that to allocate element x, all previous elements 0..(x-1) must also be allocated, even if unused. So allocating huge indexes is extremely slow, or the system will run out of memory. Therefore, I'm settling on INT_MAX (2147483647) as the maximum for indexed arrays.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions