Question about RAW - retaining 'linear' data for image analysis

[JMC]

Yeah, I'm not always the best at explaining myself Andrea. Anyway, I have downloaded RawDigger, and will try to explain a bit better with the following.

 

I took my 8 Labsphere diffuse reflectance standards, and photographed them in sunlight using my monochrome EOS 5DSR and Asahi UAT 85mm lens (Baader U filter, f8, ISO800, 1/30th second). I captured a monochrome JPEG file and RAW file in the camera.

 

1. With the JPEG file I cropped the areas from within each of the standards, so I ended up with 8 small JPEGs, one of the middle part of each standard. These were then loaded into ImageJ (image analysis program) and the histogram assessed. This gave a score of between 0 (black) and 255 (white) for each one.

 

2. With the Raw file, I loaded it into RawDigger. I drew square areas in each of the 8 standards and recorded the R, G, B and G2 values for each tile (varies between 0 and 25,000ish). I then averaged all of these for each one, to give an average score for each standard.

 

I took both of these and plotted them against the % Reflectance, and got this.

 

Obviously the JPEG does not show a straight line response between the grey scale score and the reflectance of the standard - it emphasises the midtones making them brighter. The RAW data shows the linear response I am after - something which varies in a straight line with the amount of light hitting the sensor, the more light hitting the sensor the higher the score, in a nice linear fashion. This was very easy in RawDigger and will be my software of choice for this type of information :)

 

One slightly odd thing - I would have thought with my monochrome camera, the R, G, B and G2 score would be the same as each other, but the R score was consistently higher than the others. I must follow up with MaxMax about why that is.

[nfoto]

The jpg data have been gamma-corrected, the 'linear' obviously not. The RAW converter will apply a gamma transformation, though, to make the final image for visual observation.

[Andy Perrin]

Jonathan, in Excel I believe you can force the best fit line to pass through zero (it is part of the Trendline options) which may make sense here.

[Alaun]

Jonathan, in Excel I believe you can force the best fit line to pass through zero (it is part of the Trendline options) which may make sense here.

 

That's the noise, why it should not hit zero :)

[Andy Perrin]

Yeah, it is, so since we know it is the noise, we can get a more accurate fit by constraining it to pass through zero. Or that was my thought process.

[Andrea B.]

but the R score was consistently higher than the others. I must follow up with MaxMax about why that is.

 

I"m wondering if the native white point of your camera might have anything to do with the readout in such a way that there is a little skew to things?

 

Wouldn't it be better to remove the sensor, wire it up and test it that way?

 

Are your standards calibrated? If not, shouldn't you be using a set which are calibrated for such an experiment? No bias on my part, I am simply being curious. And asking the questions as they come to mind.

 

Also, ISO-800 is fairly noisy on an EOS. How is noise going to affect your calculations? Probably best to shoot and analyze each standard separately and not all in one photo. (If already doing this, please ignore.)

 

Also, forgot to mention that some cameras cook even the raw data. So how to determine whether your particular EOS does or does not cook its raw data?

 

are you shooting 12 or 14? uncompressed raw or lossless compressed raw or lossy compressed raw?

 

At ISO does this particular Canon apply any gain?

[Andy Perrin]

Also, forgot to mention that some cameras cook even the raw data. So how to determine whether your particular EOS does or does not cook its raw data?

(I mentioned this upthread. For what it's worth, the fact that he got a linear response seems to suggest it is NOT cooked, or at least not cooked with any transform that affects the average value nonlinearly.)

[Andrea B.]

I wanted to see raw data from a Monochrome camera to try to understand more about them.

MaxMax offers some raw file downloads.

 

D800 Mono

Histogram with linear Y-axis.

Note difference between green and others on the right-hand wall. Look at green peaks relative to similar locations on red & blue. This is evidence of a little tiny bit of cooking called "white balance pre-conditioning".

 

 

Fuji XPro-1

Histogram with linear Y-axis.

This has a non-standard CFA, so all green is shown in one histo.

Red and blue histos are not exactly alike.

[JMC]

Hi Andrea, in response to your questions...

 

but the R score was consistently higher than the others. I must follow up with MaxMax about why that is.

I"m wondering if the native white point of your camera might have anything to do with the readout in such a way that there is a little skew to things?

 

Not sure. Native white point is a new term to me. However if the Bayer filter has been removed, I would have thought the R, G and B channels would behave the same way, unless there is a fundamental difference between them in addition to the Bayer filter. I wonder if shots of the same scene with different filters leads to changes in the R, G and B curves on a my monochrome camera?

 

Wouldn't it be better to remove the sensor, wire it up and test it that way?

 

It may well be, but that is beyond what i would be willing to do.

 

Are your standards calibrated? If not, shouldn't you be using a set which are calibrated for such an experiment? No bias on my part, I am simply being curious. And asking the questions as they come to mind.

 

They are Labsphere certified standards. I have not had them rechecked for calibration though, but the straight line in the RAW graph would suggest to me that they are not a million miles away from where they should be. I only included them here as a quick test to explain the difference between JPEG and RAW, and to help with what I was trying to achieve.

 

Also, ISO-800 is fairly noisy on an EOS. How is noise going to affect your calculations? Probably best to shoot and analyze each standard separately and not all in one photo. (If already doing this, please ignore.)

 

No idea how noise will effect my calculation, other than I would expect it to increase variability. I shot them all at once here as the weather was variable (as it often is in the UK) and I wanted to ensure the same lighting for all 8.

 

Also, forgot to mention that some cameras cook even the raw data. So how to determine whether your particular EOS does or does not cook its raw data?

are you shooting 12 or 14? uncompressed raw or lossless compressed raw or lossy compressed raw?

At ISO does this particular Canon apply any gain?

 

Not sure on any of these - I'm a RAW newbie and to me RAW is RAW is RAW. Gives me plenty more to look into and to try and learn about though :)

[Shane]

An autoscaled file has had its histogram "stretched" to cover a file's EV range with the camera's basic EV range also taken into account.

(I should try to find a more accurate description of that. Please everyone feel free to jump in on describing Autoscale.)

 

 

12 and 14 bit bit RAW data needs to be scaled to 16 bit as follows:

Create untagged greyscale RAW data linear image using dcraw -D -4 -T

Open in photoshop and add levels adjustment layer.

For 12 bit set the highlight input level to 15 and for 14 bit set it to 63.

You now have a scaled RAW file.

This is not the same as autoscaling.

[Shane]

alternately use -d -4 -T for a scaled version of the RAW data.

 

If you want this file without WB then use -d -r 1 1 1 1 -4 -T

[Andrea B.]

I will try to run dcraw on my examples above in the next couple of days. We have been sidetracked here at home by various "events". :)

Shane has mentioned two good command line settings for dcraw.

 

 

I would have thought the R, G and B channels would behave the same way,

A channel is a software construction. Given that the camera does not know it is monochrome, the raw data will be dumped into the usual r, g and b channels. As you have seen from the examples I showed. some cameras double the green rather than capturing it as 2 separate green channels. Other cameras do slightly cook the data as you can see from the green in the nikon channels. So it would be important to see what the raw data of your mono EOS looks like. The manual should tell you how it saves raw, uncompressed, lossy comp or non-lossy comp and whether it is 12 or 14 bit.

I don't see any of this as a problem, but you do need to know the facts arout your camera. :)

[Andrea B.]

Here are some dcrawU extractions of the file used previously.

 

The first, rawest version is made as "dcrawU -v -D -4 -T"

where

-v = verbose (just so I can follow what is happening)

-D = the Big D, for an undemosaiced raw with no scaling

-4 = linear 16-bit output (that is, gamma is 1 1)

-T = write output to a TIF file.

As before, you can't see too much in this because no midtone lift has been applied.

 

 

 

The next level of raw is made as "dcrawU -v -d -4 -T"

where

-d = the little d, for an undemosaiced raw with the black point adjusted and a rescale to a 16-bit range.

This scaled version shows a little bit more because its histogram has been "stretched" a bit. Or however you chose to term it. :D If this were enlarged, you could see the grid.

 

Here is an unresized crop from the preceding to show the grid.

 

 

 

And now the classic "raw composite" -- a demosaiced, scaled, gamma-ed version with a uniform white balance made using "dcrawU -v -r 1 1 1 1 -6 -T"

where

-r = indicates a white balance defined by r g b g coordinates --in this case, 1 1 1 1.

-6 = gamma-ed 16-bit output

This version does seem a bit more saturated than the corresponding Raw Digger version?

 

 

 

*******

There are lots of other things you can do with Dcraw, but perhaps some other day when I have more time available.

I will happily accept corrections on my terminology for these Dcraw examples!!! I'm feeling like I don't quite have it right? Let me know!

[Shane]

As before, you can't see too much in this because no midtone lift has been applied.

The image is currently 12 bit (4096 levels) in a 16 bit (65536) histogram, applying midtone lift would do nothing as there are no "midtones" at this stage, it's all shadow because it is unscaled. It's 16 bit scaling that makes it appear "lifted". (Damn spell check...kept turning midtone into midtown).

 

-d = the little d, for an undemosaiced raw with the black point adjusted and a rescale to a 16-bit range

Unless -d is accompanied by -r 1 1 1 1 the resulting image, although greyscale, will have WB multipliers applied to the luminance data.

 

-d is the same as -D but also applies:

1) scaling to 16 bit

2) black offset subtraction (if applicable)

3) camera saturation point ( not necessarily 4096 or 16385) adjusted to the maximum of the 16 bit range.

4) linear WB multipliers are applied to scale the three "colour" channels.

[Shane]

the -d version is actually likely to be a little brighter than the non WB version of -d -r 1 1 1 1 because, for instance, daylight WB multipliers are greater than one.

[Shane]

a demosaiced, scaled, gamma-ed version with a uniform white balance made using "dcrawU -v -r 1 1 1 1 -6 -T"

"uniform"?

re: UniWhite Balance

 

dcrawg using the g parameter permits optional gamma choices.

 

Not used dcrawU, which I assume is Unix version?

 

I run dcraw on my mac using parallels and windows XP (because I can't live without windows only BRU file renaming utility).

[Andrea B.]

Yes, dcrawU is the unix version which I use on the command line in the Mac Terminal window. Sometimes I write some Unix ksh around the dcrawU to automate things a bit.

 

I do have the optional gamma choices using the -g option with dcrawU.

 

I do not have a particularly up-to-date "manual" for dcrawU. There might not even be one?? The man page I had labeled the Big D as being raw with no adjustments. So I thank you very much for your previous comments!!! :D

I will go back through my dcraw posts and try to clean them up a bit. I could see everything you mentioned about the brightness differences, but had no way to read what was going on. Somewhere it must be documented! There's only so far I can go with analogies from Raw Digger before running into snafus with dcrawU.

 

I'm also going to look at Raw Therapee and Raw Photo Processor 64 to see what kind of raw files they can produce. But I have foto work due on Wednesday and Thursday, so prolly won't get to those apps until the weekend.

 

*****

 

UniWB == unitary white balance, I suppose? I've also seen uniform white balance. I'm not sure which is the correct one. As long as we see that the WB is 1 1 1 1............. :lol:

[Shane]

Uni Unitary infers 1 where white balance multipliers are 1, uniform WB seems a bit more vague........that's my take.

 

Yes a manual.....there's a concept. Never found anything close, experimenting and testing outcome produced some info.

 

I got Rawtherapee a few years back, played with it a bit but it didn't do what I wanted, doesn't mean it's not a good option to try, so interested to know what you find.

Darktable may be of use, downloaded recently but no time to try it. (Damn it...spellcheck....darktable becomes marketable and almost didn't catch it).

[Shane]

guillermo lujik? probably had some of the best info on dcraw

[Andrea B.]

Yes, Guillermo's work is still there. :D

[Andrea B.]

Just prowling around the web, I found uniWB labeled "unitary", "uniform" and "universal" white balance. Take your pick !!!!

 

 

But on Guillermo's website (link below), he calls it "unitary". Let it be Unitary then. :D

http://www.guillermoluijk.com/tutorial/uniwb/index_en.htm

[Shane]

I think Iliah Borg first came up with the idea and use of a WB with unitary multipliers early this century (feels odd to say it that way) and I believe named it based on the Unitary WB multipliers. I remember downloading ones he provided for the Nikon D70 along with a linear "tone curve".

[Andrea B.]

I know Iliah was in the lead on that. It was a huge thing back then, wasn't it? Remember when we were trying to write up algorithms for everyone to use to create UniWB??

Now the cameras are so very much better that I no longer use UniWB for "regular" photography -- preferring instead to just shoot a CC Passport and make an in-camera white balance preset.

 

However, for our slightly strange modified Nikons as used for UV, a UniWB is still a very nice thing to have.

 

Added: Just remembered that in the early incarnations of Capture NX we could create linear curves (and other curves) in the Picture Control app and load them directly into the camera. I don't know if that capability still exists.

 

Added again: Shane, I wonder if we could get Kickstarter Funding for ordering a nice spectro set-up to measure transmission of UV-pass filters, UV-capable lenses and modified camera sensors if we promised to share the data with the Entire World? Do you think we would get any money? :lol: What would we need again? A monochromator, a dual gated something, an integrating sphere, a specro something or other. Argh. I have to go look it up. I've forgotten the terms.

 

FOUND IT: Ideally you need a CCD spectrometer, or spectrophotometer (preferably in a dual beam configuration) with a monochromator, an integrating sphere, and a stable output UV-V-IR source. The integrating sphere helps ensure that the readings are independent of the incoming light ray angle. This will provide you with wavelength specific transmission response.

 

I think I'll go pick out a nice set at Ocean Optics and put it on a Wish List. Any guidance from anyone as to what to pick?

[Shane]

Look to Avantes for the Integrating sphere due to better baffling (although it and OO sphere is a little on the small side) and also Avantes D2H light source for better stability and output, and the OO 200-1100nm spectrometer with UV enhancement and 2nd order sorting filter, UV capable fiber optics......etc,

[Shane]

Think you can still use View NX and View NX2 for curves, depending on which camera you have. (Haven't done it for a while.