NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
How does lossless compression in Fuji RAF files work? (2020) (capnfabs.net)
yboulkaid 7 hours ago [-]
I wonder if something about this explains why Fuji RAF files are the only RAW files that make Lightroom struggle.

I never heard the fans on my M1 mac until I started processing those RAF files, they're not even that large but something about them makes Lightroom struggle. And judging by posts on photography forums, I'm not the only one either

jcynix 5 hours ago [-]
Lightroom struggles with Fuji's unconventional sensor pattern. Other programs like Darktable, GraphicConverter, Luminar Neo or FastRawViewer don't make fans of my M1 audible.
MaxGripe 5 hours ago [-]
I'm not sure what causes the terrible RAF handling in Lightroom, but it's a fairly well-known issue. Lightroom produces so-called "worming" and apparently this is related to the X-Trans sensor which requires a different decoding algorithm. On my end, I can recommend Capture One - the handling here is perfect, though you do have to get used to the UI unfortunately
ImPleadThe5th 7 hours ago [-]
I moved away from Adobe recently and ended up trying out Affinity Photo. Opening a RAF file would regularly seize my entire computer.

I always wondered if it was a deadlock situation, because it doesn't spit out any overflow errors.

kjkjadksj 6 hours ago [-]
Lightroom still is terrible for fuji raws from a worm standpoint. Almost looks like they use bilinear debayering. Dcraw is night and day better.
the_lucifer 9 hours ago [-]
As someone who shoots on Fujifilm (XT-3), this was an intensely fascinating read. Thanks, now I have half a mind to sit down and re-implement this code, just to get a feel of how it works.
vadansky 8 hours ago [-]
I've been thinking of buying that camera for a while, do you recommend it? Do you have anything to say that will finally push me over the edge to actually but it?
ebbi 1 hours ago [-]
I have an X-T3 and I love it. I went from an X-E2, to a Sony set up, and then quickly went back to Fuji. There's just something about Fuji that made it more enjoyable to shoot, for me (mostly travel photos).

I will say the only thing that gives me FOMO is the lack of the Classic Negative film sim, as a lot of recipes that I see online that I really like uses that film sim as the base.

If what appeals to you about Fuji's are the recipes and film sims, I'd make sure to research which ones you like, and then work out which model has the film sim you need to recreate it.

jagaerglad 8 hours ago [-]
I have the same one and I can definitely recommend it. It depends what your camera experience is, but if you have had one that collected dust on a shelf in the past, I can guarantee you that this one is more fun to use and has a much lower risk of dust collection
PetitPrince 7 hours ago [-]
Another happy X-T3 owner here (I had in my hand a Nikon D40X, D300s, D810 before getting a X-T1 and then upgrading to X-T3 ; thanks dad).

Yes, this is a very good camera. I love UI of Fujifilm cameras; and by that I do not mean the menu system (which is... serviceable) but the physical dial for each of the main setting. Putting them in "A" for automatic just make sense compared to the usual PSAM modes.

MaxGripe 5 hours ago [-]
I own 4 Fujifilm cameras and personally, I'd recommend being VERY careful and thinking hard about this purchase. This isn't the same Fujifilm as it used to be. The company was once known for its "Kaizen" approach, which has long since disappeared. Prices are now inflated because they're riding on popularity. Autofocus in Fuji is simply weak.

The question is whether you actually need such a camera for anything. With a new smartphone that has multiple lenses, out-of-the-box photos will turn out MUCH NICER than from a camera, because initial processing is built into the software. Digital cameras don't have this. You need to take RAW and work pretty hard on it to make the photo look as good as what a smartphone delivers right away.

In tourist destinations, you can often find middle-aged guys running around with huge cameras when in reality most of their photos are quite poor. Because they don't realize that with a regular phone, their pictures would be much nicer.

crab_galaxy 4 hours ago [-]
> The question is whether you actually need such a camera for anything. With a new smartphone that has multiple lenses, out-of-the-box photos will turn out MUCH NICER than from a camera, because initial processing is built into the software. Digital cameras don't have this. You need to take RAW and work pretty hard on it to make the photo look as good as what a smartphone delivers right away.

You’re completely neglecting to highlight Fuji’s film simulations. I use Fuji’s specifically because they produce excellent jpgs out of camera. Not really sure where your take is coming from, an xt3 on auto will blow any smartphone picture FAR out of the water.

tkcranny 2 hours ago [-]
It’s true that phones cameras are miracles of technology, especially considering their size. But I take a modern Fuji traveling because the modern phone camera look is so over-processed and distinct. There’s no faking the real optics a large aperture and sensor gives, the portrait mode on phones is still a poor imitation of the real thing.

Fuji then has the whole film simulation system with all their colour science from the last century. It’s a ton of fun, and the jpgs it produces are distinct and beautiful, and I believe better than 99% of people could achieve from post processing the raws, myself included.

The middle-age guy part is accurate though, I got it as a thirtieth present.

barnabee 42 minutes ago [-]
I don’t find this at all, even compared to my (now rather old) X-T1.

For quick shots to remember an event or night out, modern phone cameras are fine.

For anything that I’d call photography and actually want to print, display, etc. I rarely if ever get results I’m really happy with from a phone camera.

If you’re in any way interested in photography beyond taking a few snaps at parties and on holidays, I highly recommend getting a real camera. I’ve found the Fuji system to be great, from the lenses to the out of camera JPEGs and film simulations that mean you can pretty much avoid doing any significant editing or post-processing if, like me, you find that all quite tedious.

dsego 3 hours ago [-]
This aligns with my experience as well. The bigger sensor does generate pictures that look more crisp in big prints or zoomed in. In theory it should gather more light, but in reality, phones stitch together multiple exposures, and frequently produce nicer low light images without much noise. For sharing on social media, it's hard to notice a difference. For me its event worse with the x100 since the wide lens doesn't have that signature compression and depth of field, so the photos don't really stand out that much, no wonder most x100 photographers rely on color filters (film sims) and high contrast to draw attention.
krior 2 hours ago [-]
Even if phone cameras were twice as good, for me its simply more fun to take pictures with my camera.
Analemma_ 8 hours ago [-]
I recently upgraded from an XT-3 to an XT-5, but loved my XT-3 and would still recommend it as a good purchase if you can find a decent deal on one in good condition. Fuji’s AF is not the best in the business, so I wouldn’t recommend one if you’re planning on using it for e.g. sports photography, but apart from that the XT series has no real downsides. The physical dials for ISO+exposure+shutter speed are fantastic and Fuji’s color processing makes images that I just enjoy looking at, even if they’re not as strictly neutral and accurate as what you’d get from someone else.
guillego 6 hours ago [-]
I have an XT-1 from 2015 (still working!) and recently started considering upgrading to an XT-5 but I'm a little hesitant to buy a "new" camera first released in 2023 that still retails for almost the same price as two years ago. I'm so torn between just going for it and waiting (who knows how long) for the X-T6 to come out. Perhaps I should just try to find a good deal on an X-T4.
smugglerFlynn 5 hours ago [-]
Innovation is very slow in photography world these days, X-T5 made a big jump in MP count compared to X-T4, but resolution aside image quality is pretty much the same, and other improvements were marginal.

I still use X-T2, and it has not really aged, even when compared to my X100V. Infamous Fuji AF is where they progress slowly but steadily, so that's the primary feature that I'd look into when choosing between generations.

Analemma_ 6 hours ago [-]
If it helps, I pay reasonably close attention to Fuji rumors because I'm deep in the ecosystem, and at present there appears to be no indication that an XT-6 is coming any time soon. They just released the GFX100RF and XE-5, plus there are rumors of an X-T30 III soon, and with all that in the pipeline I doubt they are also finishing up an XT-6. The -4 and -5 are still great cameras, I would just go for whichever of those you think is a better deal.
FireBeyond 29 minutes ago [-]
> an XT-6 is coming any time soon

Besides, it's still near impossible to get an X100 VI. B&H's backlog must be over a year at this point.

ninjin 7 hours ago [-]
Fujifilm's whole X-mount series is wonderful and while I shoot "full-frame" M mount to remain interoperable between digital and film, there is no doubt in my mind that I would have a Fujifilm X-mount series if I only shot digital based on how much fun they have been when I have borrowed/tested them. Great "enthusiast level" cameras, great glass, solid build, everything has a button/dial, does not break the bank, and I actually know more than one professional photographer that shoots them and one of them even shooting sports!
vjvjvjvjghv 10 hours ago [-]
There are always these crazy discussions in photo forums where people are arguing that uncompressed RAW files are better than lossless compressed.
matja 9 hours ago [-]
Those discussions I feel are fuelled by manufacturers like Sony saying [0] things like:

  Lossless compressed RAW:
  ...
  This is a popular format that occupies less space with minimal quality loss.
"minimal" "loss"? That's not "no loss", so what exactly is it?

[0] https://www.sony.co.uk/electronics/support/articles/00257081

gappa 9 hours ago [-]
I actually asked Sony Support about that. Their reply: "We can confirm that with Lossless compressed RAW there is a minimal quality loss. To have no impact on image quality I suggest using Uncompressed RAW files." Lossless isn't what it used to be.
smugglerFlynn 5 hours ago [-]
Seems you just got 1st line robo-reply repeating what public resources state. Does not say much about actual compression algorithm Sony uses.
gappa 4 hours ago [-]
Their first reply was "we have passed your question to a higher technical team", then they came back four days later with the above reply. I was enquiring about the A7R mark V, which introduced the much needed "lossless" option. I think I asked because I wondered why they kept the uncompressed option and because experts warned that Sony did that before with "lossless" formats.
adgjlsfhk1 4 hours ago [-]
It is a shame that Sony has such an obsession with weird proprietary formats.
benoliver999 7 hours ago [-]
They read it lessloss I guess
vjvjvjvjghv 6 hours ago [-]
That's insane. I honestly thought that lossless basically means to run zip over the file and not more.

Gotta hate companies these days with their dishonesty. "Lossless" means "lossy". "Unlimited" means "limited to 50GB".

Scaevolus 6 hours ago [-]
A lot of cameras write "lossless" to mean "perceptually lossless". This is easy to do because ~12-bit ADCs have lots of noise in the low order bits.
matja 5 hours ago [-]
"lossless" has always referred to compression, not sampling - but it seems camera manufacturers want to change that for marketing reasons.

Similarly (without starting an audiophile thread): Recording a vinyl record and compressing to a MP3 is "perceptually lossless" but will be different to compressing to a FLAC, never mind that the sampling output will always have random noise.

manoDev 9 hours ago [-]
It’s because manufacturers keep inventing their own formats instead of using plain DNG, and you can only figure out the difference reverse engineering the file. Even for the same manufacturer and file type, they may change encoding details between models or firmware revisions.

There are situations where they may decrease the bit depth of the final image if there’s not enough dynamic range, there are situations where even though the file is “uncompressed” the camera already does noise reduction and essentially compresses detail in the image, and so on…

tecleandor 7 hours ago [-]
DNG license might include some legal stuff that probably they don't want in their devices.

https://helpx.adobe.com/camera-raw/digital-negative.html#dng

graemep 5 hours ago [-]
What is the problem there? It essential provides an unlimited license any Adobe patents needed to implement DNG, and revocation if you assert a patent against Adobe related to DNG.
danhau 8 hours ago [-]
Manufacturers not supporting DNG is so annoying. But of course they do it. Allows them to sell licenses to software vendors like Adobe or Capture One.
brnt 8 hours ago [-]
How is DNG tooling these days? It's been ages since I checked, but it used have near zero support in (FLOSS) tools (including ffmpeg/imagemagick). You even had to use Adobe's Windows converter.
scblock 6 hours ago [-]
DNG is well supported by the typical raw libraries in my experience. My Ricoh records DNG natively, and I work with the results in rawtherapee and darktable. I am less sure about ffmpeg or imagemagick as I don't really automate raw processing.
jsheard 9 hours ago [-]
Glad to hear that audiophiles have some competition in who can invent the most pointless things to split hairs over.

The photo guys need to start taping magic rocks to their cameras to really keep up though.

miladyincontrol 8 hours ago [-]
The worst part is its often in areas people improperly understand the benefits and downsides.

Biggest problem with good raw compression is you have a linear DNG, half processed essentially. Great, the file size is smaller, but now you miss data that processes like AI denoise can benefit from as the image is already debayered.

On the flip side, good compression like DNG 1.7 spec's jpeg-xl compression is borderline magic. Lossless is actually lossless. The lossy flavour is so good even at 105 megapixels in 16 bit (per color channel) I would challenge anyone to spot a noticeable difference compared to the original, a file possibly 20x it's size.

On a tangent, bits per channel is yet another part people split hairs over. 14 vs 16 has almost no difference, no the colours are not 'better' even in a full 16 bit workflow, the only real world perceivable difference is your darkest darks are more precise and under extreme editing conditions do look a little better if being raised extensibly in post. I digress 16 is bigger than 14 and yay marketing.

Looping back to compression, 14 bit raws without compression are padded to 16 bit lengths due to word sizes and file constraints. This bit throws off the less technically minded who make all sorts of assumptions about file sizes and being 'more lightweight to edit'.

adgjlsfhk1 6 hours ago [-]
I'm kind of surprised we haven't seen things like 16 bit luminence, 12 bit chromanance. (I guess to do that before debayering would require RGBW pixels or something like that)
badlibrarian 9 hours ago [-]
They weren't just rocks. They were brilliant pebbles.

https://web.archive.org/web/20150220012844/http://www.machin...

beAbU 1 hours ago [-]
There's probably an untapped market for sd cards with "special" holographic stickers on them retailing for $100 each.
jsheard 5 minutes ago [-]
Maybe an audiophile SD card would also produce better photos in a camera? You never know until you try.

https://www.whathifi.com/news/sony-claims-high-end-sd-card-o...

mlyle 8 hours ago [-]
The problem is camera marketing makes it really unclear what you're actually getting. Really lossless? Does it trigger more on-body processing? Does it throw away bits to be "visually lossless"?

If you shoot a few thousand photos and then find you can't fix exposure as well as you'd hoped and the whole batch is worse, it's a pretty big disappointment, so it's smart to be risk adverse and skeptical.

jcynix 6 hours ago [-]
>The problem is camera marketing [...]

IMHO marketing is almost always (part of) the problem. They shouldn't drink or smoke that stuff …

Equating "lossless" with "visually lossless" or some other phrase is newspeak. We could call a JPEG of quality >= 95 (or 96 .. 99) visually lossless too then.

Losslessness is easy to define: compress something, then uncompress it again and both the original and the uncompressed file should compute to the same (cryptographic) checksum.

jcynix 5 hours ago [-]
These forum discussions seem unconnected to reality IMO. Sometimes funny, more often strange, like audiophile discussions about oxygen free copper cables. Why?

The Fuji X-T4 and X-S20 here produce images of 6240x4160 pixels, but I almost never look at images in 1:1. My 4K monitors, mostly set to 2K, display 2560 x 1440 pixels. And even when switch them to their full 4K resolution I don't view my images in 1:1 obviously. And the tablet l'm typing this comment on offers "meager" 1800 × 2880 pixels. Most family members look at images on their "smart" devices nowadays, where 2K or 4K aren't present. So even decompressed images are fine for them.

I have my cameras configured to take both JPEG (fine) + RAW (lossless) of course. Fuji JPEGs (and Canon ... etc too) are fine for most casual viewers. And if I want to crop certain parts, or adjust certain details (esp exposure), I have my RAW images as a fallback.

Storage? My SD cards are 256GB and my disks are definitely not a problem either.

LeifCarrotson 5 hours ago [-]
I almost never look at full images in 1:1, either, but I find very often that I have the wrong lens for an application and/or the wrong exposure.

You don't have to go very far before you've cropped a 20 MP source image into a 4K or 2K image, and if that part of the image that you've wanted to highlight is not well-lit, well, exposure is logarithmic and I want all of that RAW color depth that the camera can find if I'm going to turn black or white into perceptually accurate colors.

It's true that when my framing and exposure are great out of the box, I probably wouldn't notice or care if JPEG compression cut my file size by a factor of 4...but that's not always the case.

cogman10 3 hours ago [-]
The main reason my wife shoots lossless is touch-up editing in light room. You can rescue some pretty crappy photos when all the sensor data is present.
FireBeyond 16 minutes ago [-]
> My 4K monitors, mostly set to 2K, display 2560 x 1440 pixels. And even when switch them to their full 4K resolution I don't view my images in 1:1 obviously.

macOS, but my understanding of Retina is that even if your effective resolution is lower (e.g., my 6K ProDisplay has a resolution of 3008x1692), applications can designate regions to be "original resolution," so I'm getting the image's original resolution in the editing window or region.

agloe_dreams 7 hours ago [-]
Honestly the best argument for uncompressed is actually nothing to do with file quality or loss - it's that Apple only supports uncompressed Fuji RAWs.

You cannot preview or process lossless compressed Fuji RAWs on iOS natively but the uncompressed files are equal to Apple's own RAWs in support. On the field, it is sadly worth every byte to be able to grab a file directly off the camera and tweak it or send it to an editor. :/

vjvjvjvjghv 6 hours ago [-]
The problem is that the size difference between compressed and uncompressed is enormous. This adds up quickly if you shoot sports or wildlife with 20fps bursts.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 22:32:53 GMT+0000 (Coordinated Universal Time) with Vercel.