Exploring Paginated Arrays

2026-08-10T00:00:00+00:00

Throughout this text, I work exclusively with signed numbers. This is because one of my personal requirements is that my libraries must be compatible with the CLS independence agreement described in the ECMA-335 CLI standard.

Recently, I was working on a personal project developed in C# that, for various reasons and with the desire to experiment a bit, required an array that could use a long integer as its index, that is, with 64 bits of indexing.

The .NET runtime environment only natively supports arrays with int integers,32 bits, as indices. This is not only a limitation of its standard library but also of its own garbage collector (GC); unless a specific configuration is enabled, objects are limited to 2 GiB.

<gcAllowVeryLargeObjects> element

On 64-bit platforms, enables arrays that are greater than 2 gigabytes (GB) in total size.
<gcAllowVeryLargeObjects enabled="true|false" />
— Microsoft Learn (source)

In any case, creating an array of such a size wouldn't be ideal either, as it would result in large monolithic blocks of managed memory that would be problematic for the GC.

Paginated Arrays

I decided to use a resource that would solve the memory problems and, in my mind, would also allow me to have 64-bit indexing: paginated arrays.

A paginated array is a data structure in which the elements are stored in fixed-size arrays called pages. My theory was that with enough pages, I could simulate 64-bit indexing.

My implementation consisted of a sparse list, a data structure that allows access to non-contiguous indices by inserting null pointers into completely empty pages.

 1namespace Redacted.Collections;
 2
 3internal class SparseList<T>(int pageSize, T tombstone = default!)
 4{
 5    public T this[long index]
 6    {
 7        get => Get(index);
 8        set => Set(index, value);
 9    }
10
11    private readonly List<SparseListPage<T>?> _pages = [];
12
13    private T Get(long index)
14    {
15        var (pageIndex, elementIndex) = GetIndices(index);
16        if (pageIndex >= _pages.Count) return tombstone;
17
18        var page = _pages[pageIndex];
19        if (page == null) return tombstone;
20        
21        return _pages[pageIndex]![elementIndex];
22    }
23
24    private void Set(long index, T value)
25    {
26        var (pageIndex, elementIndex) = GetIndices(index);
27        var isValueTombstone = Equals(value, tombstone);
28
29        var missingPages = (pageIndex + 1) - _pages.Count;
30        if (missingPages > 0 && isValueTombstone) return;
31        
32        for (var i = 0; i < missingPages; i++)
33        {
34            _pages.Add(null);
35        }
36
37        _pages[pageIndex] ??= new SparseListPage<T>(pageSize, tombstone);
38        var page = _pages[pageIndex]!;
39        
40        if (page.Count == 1 && isValueTombstone)
41        {
42            _pages[pageIndex] = null;
43            return;   
44        }
45        
46        page[elementIndex] = value;
47    }
48    
49    private (int pageIndex, int elementIndex) GetIndices(long index)
50    {
51        return ((int)(index / pageSize), (int)(index % pageSize));
52    }
53}⏎

 1namespace Redacted.Collections;
 2
 3internal class SparseListPage<T>
 4{
 5    public int Size => _size;
 6    public int Count;
 7    
 8    public T this[int index]
 9    {
10        get => _data[index];
11        set
12        {
13            var isNewValueTombstone = Equals(value, _tombstone);
14            var isStoredValueTombstone = Equals(_data[index], _tombstone);
15            
16            if (!isNewValueTombstone && isStoredValueTombstone)
17            {
18                Count++;
19            }
20            else if (isNewValueTombstone && !isStoredValueTombstone)
21            {
22                Count--;
23            }
24            
25            _data[index] = value;
26        }
27    }
28    
29    private readonly int _size;
30    private readonly T _tombstone;
31
32    private readonly T[] _data;
33    
34    public SparseListPage(int size, T tombstone = default!)
35    {
36        _size = size;
37        _tombstone = tombstone;
38
39        _data = new T[size];
40        Array.Fill(_data, _tombstone);
41    }
42}

With my implementation working and passing my unit tests, I set about bringing these structures to the real world. I dedicated a few hours to my project, I ran dotnet run in the terminal and...

Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
    at System.Collections.Generic.List`1.get_Item(Int32 index)
    at Redacted.Collections.SparseSet.SparseList`1.Get(Int64 index) in [...]
    at Redacted.Collections.SparseSet.SparseList`1.get_Item(Int64 index) in [...]

At that moment I was surprised. It turns out that when trying to use the highest possible index, long.MaxValue, that is, with the number 9,223,372,036,854,775,807, the GetIndices() method returns the value -1 as the page index.

This is because the index / pageSize operation does not guarantee that it will be a value less than or equal to int.maxValue, that is, the number 2,147,483,648.

This makes sense; it's not hard to see that any long / int operation doesn't guarantee this.

Reason

The reason, which disproves any possibility that my theory was true, is straightforward.

An array with 32-bit indices can store up to $ 2^{31} $ elements. This implies that each page can store that many elements; and since pointers to pages are also stored in an array, there can be up to $ 2^{31} $ pages.

This means there can be up to $ 2^{31} * 2^{31} = 2^{31 + 31} = 2^{62} $ elements in total in a paginated array, but an array with 64-bit indices can accumulate up to $ 2^{63} $ elements. There are $ 2^{63} - 2^{62} $ inaccessible elements; that is, the last $ 4,611,686,018,427,387,904 $ indices of the array are not indexable.

What if we removed the sign?

If we encountered a language that allowed 32-bit unsigned indices, up to $ 2^{32} $ elements could be stored; using the same reasoning as before, we could have up to $ 2^{32} $ pages.

This would allow us to reach $ 2^{32} * 2^{32} = 2^{32 + 32} = 2^{64} $ elements in total when using a paginated array. It's interesting that in this hypothetical language, a paginated array could store not only all the elements of a 64-bit signed index array ($ 2^{63} $ elements) but also of a 64-bit unsigned index array ($ 2^{64} $ elements).

Proof

Given the previous results, it's natural to be interested in finding a mathematical proof:

We set up $a \gt b$, where $a \in \mathbb{N^+}$ is the number of bits of the indices of the array we want to replicate, and $b \in \mathbb{N^+}$ is the number of bits of the indices of the arrays that make up our paginated array (the replica).

For the replica to be valid, all elements of the original array must be able to be included in the replica:

$$ 2^a = 2^b * 2^b \implies 2^a = 2^{b + b} \implies a = 2b \implies a / 2 = b $$

And since we are working with integer arithmetic, the expression $a / 2$ only exists if $a \bmod 2 = 0$; In other words, $ a $ must be an even number $ \blacksquare $.

Alternatives

I doubt there are alternatives to cover this specific case; most systems where such a need might arise are capable of using 64-bit indices.