• GreatDong3000@lemm.ee
    link
    fedilink
    arrow-up
    7
    arrow-down
    3
    ·
    5 months ago

    Man the models can’t store verbatim its training data, the amount of data is turned into a model that is hundreds or thousands of times smaller than the original source data. If it was capable of simply recovering everything that it was trained on this would be some magical compression algorithm and that by itself would be extremely impressive.

      • GreatDong3000@lemm.ee
        link
        fedilink
        arrow-up
        5
        arrow-down
        3
        ·
        edit-2
        5 months ago

        Oh ok, you want to claim this is compressing the entirety of the internet in a model that isn’t even 1 terabyte of data and be unimpressed that is something.

        But it isn’t compression. It is a mathematical fact that neural networks are universal function approximators, this is undisputed, and analytic functions are continuous so to be an analytical function approximator it must be able to fill in the gaps between discrete data points by itself, which necessarily means spiting out data outside of the input distribution, data it has not seen.

        • 31337@sh.itjust.works
          link
          fedilink
          arrow-up
          3
          arrow-down
          1
          ·
          5 months ago

          TBF, compression is related to ML. Hence, the Hutter Prize. Thinking of LLMs as lossy compression algorithms is a decent analogy.

          • GreatDong3000@lemm.ee
            link
            fedilink
            arrow-up
            1
            arrow-down
            1
            ·
            5 months ago

            It is a partial analogy, it takes into consideration the outputs which are related to some specific training data and disconsiders the outputs which cannot be directly related to any specific training data.

            For example, make up a new meme template and a new joke on the spot, it couldn’t have seen it before if you make sure your joke and template are new. If the AI can explain it then compression is a horrendous analogy.

            Lossy compression explains outputs being similar but not identical when trying to recover the original data, it doesn’t explain brand new content that makes sense standalone. Imagine a lossy audio compression resulting in a brand new song midway through playback, or a lossy image compression resulting in a brand new coherent image being overlayed onto some pixels of the original image. That is not what happens, lossy audio compression results in noise, lossy image compression results in noise, not in coherent unheard songs and unseen images.

        • SpaceNoodle@lemmy.world
          link
          fedilink
          arrow-up
          1
          arrow-down
          8
          ·
          5 months ago

          Not sure why you feel the need to put words in my mouth. It wasn’t trained on “the entirety of the Internet,” but rather less than a terabyte of it. So yeah, that would probably take up less than a terabyte.