GGUF huge files size / Split possible ?

[phymbert]

Distributing and storing GGUF files is difficult for 70b models, especially on f16. Lot of issue can happen during file transfers, examples:

• temporary disk full
• network interruption

Typically, they need to be tranferred from huggingface to an internal storage like s3, minio, git lfs, nexus or artifactory, then downloaded by the inference server and stored locally (or on a k8s PvC for example). Also they cannot be stored in a dockerfile, but IMHO this is for good.

Is there a smarter way than splitting the file using split & cat ?

[phymbert]

@ggerganov thoughts? would it be possible to support a split size in GUFF ?

[ggerganov]

We can give it a try - I can't estimate how big would be the change. A rough plan would be:

• implement split tool that given a GGUF file generate N GGUF files
• modify llama_model_loader to load tensors from multiple GGUf files (probably checking for extension .gguf.1, .gguf.2, etc)
• maybe add optional meta data in the GGUF to indicate number of splits

Ideally, no changes to ggml would be needed.

Depending on the complexity of the implementation we will decide wether to support or not

[phymbert]

Ok, I will give it a try. Thanks for the guidance

[ngxson]

Good idea!

I've just had a look on llama_model_loader. What I understand is that we can add one tuple gguf_context, ggml_context for each file. We can then have a std::unordered_map<std::string, ggml_context> that maps the tensor name to the correct context.

One thing that I'm not sure is that create_tensor and create_tensor_for are currently taking both ggml_context and tensor name as input - this maybe tricky because now we need to either select the context outside of create_tensor, or either just pass the mapping to create_tensor and let it search for the context.

Maybe there's better idea than mine?

[slaren]

The ggml_context that create_tensor and create_tensor_for takes is just an user-provided context where the tensor should be created, you don't need to do anything special with these. Just find the ctx_meta where the tensor is, and duplicate it from there into the user-provided context.

[phymbert]

Getting the tensor is OK, loading the tensor data from multiple files is more complicated especially with mmap. Any help is welcomed based on 14442d8

[Artefact2]

Is there a smarter way than splitting the file using split & cat ?

On Linux yes, if your filesystem supports reflink copies (btrfs, xfs do). I've written a couple demo utilities that use copy_file_range() for near-instant merging/splitting of files regardless of their size.

my-split.c

#define _GNU_SOURCE
#define _FILE_OFFSET_BITS 64

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
#include <unistd.h>

int main(int argc, char** argv) {
    if(argc != 2) {
        fprintf(stderr, "Usage: %s file.gguf\n", argv[0]);
        return 1;
    }

    int src_fd = open(argv[1], O_RDONLY);
    if(src_fd == -1) {
        perror("src open()");
        return 1;
    }
    off_t src_len = lseek(src_fd, 0, SEEK_END);
    if(src_len == -1) {
        perror("lseek()");
        return 1;
    }
    lseek(src_fd, 0, SEEK_SET);

    /* 50GB (NOT GiB) */
    const off_t BSIZE = 372*128*1024*1024l; /* Must be a multiple of CFRSIZE */
    const size_t CFRSIZE = 128*1024*1024l;
    if(src_len < BSIZE) return 0; /* No need to split */

    unsigned int n = 0;
    char dest_path[strlen(argv[1]) + 10];
    int dest_fd;
    for(off_t i = 0; i < src_len; i += BSIZE) {
        snprintf(dest_path, sizeof(dest_path)-1, "%s.%03u", argv[1], n);
        dest_fd = open(dest_path, O_CREAT|O_TRUNC|O_WRONLY, 0666);
        if(dest_fd == -1) {
            perror("dest open()");
            return 1;
        }
        for(off_t k = 0; k < BSIZE; k += CFRSIZE) {
            if(copy_file_range(src_fd, NULL, dest_fd, NULL, CFRSIZE, 0) == -1) {
                perror("copy_file_range()");
                return 1;
            }
        }
        close(dest_fd);
        printf("%s\n", dest_path);
        n += 1;
    }
    close(src_fd);
    return 0;
}

my-merge.c

#define _GNU_SOURCE
#define _FILE_OFFSET_BITS 64

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
#include <unistd.h>

int main(int argc, char** argv) {
    if(argc < 3) {
        fprintf(stderr, "Usage: %s file.000 file.001... merge.out\n", argv[0]);
        return 1;
    }

    int dest_fd = open(argv[argc-1], O_CREAT|O_TRUNC|O_WRONLY, 0666);
    if(dest_fd == -1) {
        perror("dest open()");
        return 1;
    }
    const size_t CFRSIZE = 2048*1024*1024l;

    for(int i = 1; i < argc-1; ++i) {
        int in_fd = open(argv[i], O_RDONLY);
        if(in_fd == -1) {
            perror("in open()");
            return 1;
        }

        ssize_t ret;
        /* copy_file_range() returns 0 on EOF */
        while((ret = copy_file_range(in_fd, NULL, dest_fd, NULL, CFRSIZE, 0)) != 0) {
            if(ret == -1) {
                perror("copy_file_range()");
                return 1;
            }
        }
        close(in_fd);
    }

    close(dest_fd);
    return 0;
}

[ngxson]

What I'm not sure (in theory) is that will there be any performance cost if the file is splitted in the middle of a tensor?

That means for example, a file having tensor0 tensor1 tensor2

Now it's splitted into 2 files:

• tensor0 tensor1(first-half)
• tensor1(second-half) tensor2

With copy_file_range it will not consolidate these "fragments" of a file, but just link them together. Maybe when mmap tries to load tensor1 to memory, it will need to make a copy of tensor1 into RAM?

[Artefact2]

No, that's not how it works. My code is semantically equivalent to split and cat, minus some abstractions that are internal to the filesystem. How the extents are laid on disk is also completely irrelevant once the model is loaded into ram (either the page cache if you're using mmap, or anon pages if you aren't).

[ngxson]

Thanks for pointing it out. Yeah I had a misconception about mmap. Let's consider my question as wrong question then.

[phymbert]

@ggerganov as we merged:

• gguf-split: split and merge gguf per batch of tensors #6135

I am working on llama_model_loader to support split gguf in 14442d8.

I need your advice to continue further on, should we have multiple std::unique_ptr<llama_mmap> mapping or llama_mmap should support multiple input files ?
Thanks

[ngxson]

To me, only the part where we load metadata that we care which i_split it is. After that, we don't care from which split the tensor come from - so we should prevent checking n_split < 2 in these cases.

[ngxson]

Another tips is maybe you can firstly make a non-mmap version that works (which should be easier), then bring back mmap after that.

[phymbert]

Thanks for your early review but how you can load tensor data to the backend if you dont know in which files/mapping/offset it is ? Metadata are not enough... Anyway I broke the branch where n_split < 2, so I am fixing this first.

[ngxson]

The offset of a tensor is already stored inside its ctx_gguf, so you can just store the ctx_gguf of the split into a std::map for example, which map name ==> ctx_gguf

Then we delegate the task of finding the offset to file_offset, as we already had this function. The modified version will looks like:

    size_t file_offset(const char * name) const {
        const split_ctx_gguf = map_ctx_gguf[name]; // map name to context
        const int idx = gguf_find_tensor(split_ctx_gguf, name);

        if (idx < 0) {
            throw std::runtime_error(format("%s: tensor '%s' not found in the file", __func__, name));
        }

        return gguf_get_data_offset(ctx_gguf) + gguf_get_tensor_offset(ctx_gguf, idx);
    }

Doing this will reduce one occurrence where you need to duplicate the line gguf_get_data_offset(ctx_gguf) + gguf_get_tensor_offset(ctx_gguf, idx)

[phymbert]

I have a working version for both CUDA and CPU backend. Cleaning it up and opening a draft PR. Thanks all

[phymbert]

Completed in:

• gguf-split: split and merge gguf per batch of tensors #6135
• llama_model_loader: support multiple split/shard GGUFs #6187
• common: llama_load_model_from_url split support  #6192

[ego]

Awesome job guys!

Only one mention about arg --split-max-size 3G sometimes it produce largest files, but --split-max-tensors works like charm 👍

[ngxson]

@ego Can you let me know the model that you're using?

@phymbert I though about this bug before, maybe because I removed the code to calculate size of metadata because the code was quite too complicated. Maybe I'll need to fix this.

[phymbert]

We need tests <3

[ego]

Model is CodeLlama-7b-Python size ~26G

./gguf-split --split --split-max-size 3G ./models/CodeLlama-7b-Python/ggml-model-f32.gguf ./models/CodeLlama-7b-Python/splited/

and its produce only 3 files.

./main --version
version: 2584 (33a52448)
built with Apple clang version 15.0.0 (clang-1500.3.9.4) for arm64-apple-darwin23.4.0

./gguf-split --version
version: 2584 (33a52448)
built with Apple clang version 15.0.0 (clang-1500.3.9.4) for arm64-apple-darwin23.4.0

[trollkotze]

What's wrong with split and cat? How can anything be "smarter" than that?
This seems like an absolutely worthless complication that is now sadly becoming the new standard.

[ngxson]

Technical details: the new gguf-split tool splits the file at tensor level, not at byte level, so it never cuts a tensor in half (which will be the case when using split & cat). This allows us to:

• Load all shards without doing any cat, because a tensor never be cut in half
• Quantize a big model while keeping all the shards: Implement '--keep-split' to quantize model into several shards #6688
• ...

Also, this is the same approach used by safetensors.

Whether it's "smarter" or "complicated", it's up to you to judge. For me, it's just more useful this way.

[phymbert]

https://twitter.com/mishig25/status/1783108187119329301

[trollkotze]

Okay, I understand now. Sorry for being rude and obnoxious.