Jump to content
UltravioletPhotography

Question about RAW - retaining 'linear' data for image analysis


JMC

Recommended Posts

I am very much a RAW 'newbie' and need some advice please. I am looking to capture images for my work on calibrating the spectral sensitivity of camera/lens combinations to wavelengths between 280nm and 480nm. The aim is to take photos of the exit port of an integrating sphere, at different wavelengths of light. I'd then use the brightness of image, in combination with the intensity of the light at different wavelengths as measured with a spectrometer, to work out how sensitive the camera is to that light of that wavelength.

 

My understanding is that the camera output to brightness is linear to intensity but processing of images changes this linearity to make the final images more appealing to the eye. To create a 'linear' image from the RAW file my approach was this - open in Darktable and create a 'base curve' which is linear. See screenshot below;

post-148-0-77515600-1518606322.jpg

 

I then save the file as Tiff file, and finally crop it in Photoshop, to isolate the middle of the illuminated area from the port of the integrating sphere, and finally do image analysis in Image J to get the histogram which gives me the grey scale value from 0 (black) - 255 (white) of the light coming from the integrating sphere. This does give me different images to straight jpeg files, so something different is definitely happening, but to be honest I have no idea if this is the right approach, or if I should be doing anything different.

 

Before I do too much on this, if anyone can comment on whether what I am doing is correct, or if there is a better way, it would be much appreciated. Thanks.

Link to comment
This does give me different images to straight jpeg files
You mean, if you save the cropped image as JPEG instead of TIFF? Or where do the JPEGs come from? By the way: JPEG is not the format of choice, since it compresses pictures and information is lost. BMP or PNG would be a better choice. But I am not sure, if this is the reason. Can you provide some more details on how you processed the images?
Link to comment

I save 2 images in the camera - Jpeg and RAW. When I say different to Jpeg, I mean that my images processed from RAW using the linear base curve, look significantly different to the Jpegs straight from the camera. It doesn't surprise me there are differences, I'm just not sure my approach in Dark table is the correct one. I'll write down a step by step to what I'm doing next time I'm in front of the PC.

 

By the way, would RawDigger be a better approach at getting info straight from the RAW files, without all these intermediate steps?

Link to comment
JMC, I’m pretty sure you want to be using RawDigger. On top of that, certain brands of camera (e.g. Sony) have been caught using a form of lossy compression in-camera on the “RAW” images, so they turned out not to be as RAW as claimed on some models. You might want to check yours.
Link to comment

Several years ago I used a similar method, different software, to evaluate UV film dosimeters. As I understand it, saving the RAW directly to TIFF should not apply any color correction and the assumption is that the resulting 0-256 scale is linear. You should be able confirm linearity by applying the analysis to your Spectralon standards.

 

You have probably already reviewed this: Garcia Papers in Open Access Journals, particularly Andrea's comment on Raw Digger Research Edition

Link to comment
Thanks All, It sounds like RawDigger is the way to go (and yes, I like the idea to check with the Spectralon standards John). I'll look into whether there is in camera compression - I use Canon (5Ds r, and 7D mk1) and Nikon (d810) cameras.
Link to comment

Another vote for RawDigger from me.

By using RawDigger you can use the original 12- or 14-bit information for rgb-values and not limit the scale to 0 - 255.

Link to comment

There are two ways to go: Raw Digger or the Dcraw program which is used from the command line.

I will post an illustration from each if you like?

Alternately, I can analyze one of your raw files if you provide a link for download.

 

There very well may be another commercial or open source converter app which can supply the raw file in various forms. I don't mean to leave anything out -- it is simply that Raw Digger and Dcraw are the two I know and have used. Also both are widely known for their use in analyzing raw data from a foto file.

 

Also Exiftool can do raw extractions. Andy has been using that for his FLIR files.

Link to comment

Here is a extraction series from Raw Digger. After I get it all posted, please, other members add your comments and suggestions.

 

Gear: Nikon D600-mod + UV-Nikkor 105/2.5 + BaaderU UV-Pass Filter + Sunlight

This photo was taken for the purpose of making a white balance preset in a converter. It is perhaps slightly underexposed in the red -- as you will soon see in the histogram. And, as always, the blue channel shows noise in the darkest EV areas. As is always recommended, the photo is slightly soft to prevent grain (or whatever) from influencing colour or WB. (Probably not needed given that a WB dropper can be dragged around an area to average out the WB?)

 

All fotos were converted to JPG (q80) and resized in Photo Mechanic. No sharpening was applied.

 

Photo as shot at f/8 for 1/6" @ ISO-200.

My in-camera WB setting was a preset made to be close to the raw composite colours (shown later). Those are not Jpg artifacts in the background. I had set the standards on a piece of cardboard. So you're seeing the corrugation pattern.

d600_uvBaader_asShot.jpg

 

There are 3 basic steps to conversion of a raw file.

  • demosaic the Bayer filter
  • autoscale the histogram
  • apply a gamma curve to raise the midtones

The Raw Linear File

Here is the raw, raw, raw file for which no demosaicing, no autoscaling and no gamma-curving have been applied. It is, of necessity, a greyscale file. Some apps won't deal with a greyscale file, so I have had to convert it to an RGB file to better subsequent versions of it here. No harm done for purposes of this illustration.

Of course, you can see nothing in this linear file.

EDIT: I overstated the case! You can't see much, but you can see maybe a little bit in this file depending on your monitor, etc.

d600_uvBaader_greyscalePixelMap.jpg

 

So how do we know anything is in this file? :D

Let me push it up to show you what is in there. I used Photo Ninja's illumination slider. Any brightening tool would work here. I did this to show you the mosaic grid. (Not here, but in the next two versions.)

d600_uvBaader_greyscalePixelMap_toRGB_illum50_v1.jpg

 

 

I blew the photo up to 200% in Photo Ninja and made a screen shot. The forum software will resize this on the page. Click it up for a larger view.

d600_uvBaader_greyscalePixelMap_toRGB_illum50_v2.jpg

 

 

ADDED: oh la!! That screen shot is all jpeg-y artifact-y. That is not what the mosaicing grid looks like. I'll make a crop at full size. Stay tuned!

 

Here you go. This is 645 pixels wide and should display on the page with no resizing downward. You can see a streak in the lower right here which is a marred area on the white standard (since refurbished).

d600_uvBaader_greyscalePixelMap_toRGB_illum50_v2_crop.jpg

Link to comment

Now I will demosaic the file and show how the Autoscaling and Gamma Curve affect the data.

An autoscaled file has had its histogram "stretched" to cover a file's EV range with the camera's basic EV range also taken into account.

(I should try to find a more accurate description of that. Please everyone feel free to jump in on describing Autoscale.)

 

Demosaiced Linear File

First, the demosaiced file with No Autoscale and No Gamma applied. It should be no surprise that you still can't see anything much in this linear file even though it is now "in color".

EDIT: I overstated the case here. You can see something in this linear version, but not much.

d600_uvBaader_2x2NoAutoNoGam.jpg

 

Demosaiced, Autoscaled File -- No Gamma Curve

d600_uvBaader_2x2AutoNoGamma.jpg

 

Demosaiced, Gamma 2.2 File -- No Autoscaling

d600_uvBaader_2x2GammaNoAuto.jpg

 

Demosaiced File with Both Autoscale and Gamma 2.2 Curve Applied.

This is what is commonly called the "raw composite" in Raw Digger. Obviously this is not a presentation ready file. You would still have to make a white balance, adjust white/black points, deal with shadows/highlights, adjust contrast and so forth.

d600_uvBaader_2x2AutoGamma.jpg

 

 

 

Photo Ninja Histogram for Preceding JPGs

These are screen grabs. They help illustrate the effects of the demosaic, autoscale and gamma operations. Not raw, but there would be analogous effects if I could show the raw histos for these operations.

 

Demosaic Only (from JPG)

pnHisto_demosaicOnly.jpg

 

Demosaic + Autoscale but No Gamma (from JPG)

The histogram is stretched but does not reach the right wall because there is about a 1/2 stop underexposure. (Or, sometimes there are simply no really bright areas in a photo.)

pnHisto_demAutoNoGamma.jpg

 

Demosaic + Gamma but No Autoscale (from JPG)

The 3 little peaks on the right look taller than in the preceding.

pnHisto_demGammaNoAuto.jpg

 

Demosaic + Autoscale + Gamma (from JPG)

pnHisto_demAutoGamma.jpg

 

 

 

Histograms of Raw Data

The Raw Digger histograms present a count of the raw data which may be presented in various ways in the chart. The count itself along the Y-axis can be shown linearly or in logarithmic or square-root form. I'm using a linear Y-axis in both views here and letting Raw Digger determine the best height. The X-axis data can be presented linearly or using EV markers based on the file's (and camera's) range. You can set the size of the X-axis bins. Here I am just letting RD decide the binning.

 

Histogram with Linear X-axis

This can be pulled out a bit by shortening the range on the right. But it is more useful (just an opinion, not a fact) to use EVs along the X-axis, I think.

d600_uvBaader_autoXLinearAutoYLinear.jpg

 

Histogram with EV X-axis

This is the "photographic" representation of the raw data.

d600_uvBaader_autoXEvAutoYLinear.jpg

Link to comment

I became curious what the demosaic, autoscale and gamma would look like in just the Red Channel. So here ya go!

 

Red Channel: Demosaic Only

This time we can actually see something more easily because so much of that UV light (all?) is recorded in the Red Channel.

d600_uvBaader_redChan2x2noAutoNoGamma.jpg

 

Red Channel: Demosaic + Autoscale, No Gamma

yak!! These poor standards have been through a lot at this point. Scratches, dirt! As mentioned somewhere, they have since been sanded and cleaned up. But I'm seeing such "discoloration" here in the white one. I'd better try this red channel shot again this spring to see how the Stds are looking. The cruft on the right grey one is something that blew in with the gale force winds from Norway that Bjørn periodically sends over to test my patience when shooting UV outdoors.

B) B) B) :lol: :rolleyes:

d600_uvBaader_redChan2x2AutoNoGamma.jpg

 

Red Channel: Demosaic + Gamma 2.2, No Autoscale

d600_uvBaader_redChan2x2GammaNoAuto.jpg

 

Red Channel: Demosaic + Autoscale + Gamma 2.2

d600_uvBaader_redChan2x2AutoGamma.jpg

Link to comment
Blimey, thanks Andrea. A lot to think about there. I guess though I would not want to do the auto scaling though, as I wouldn't want to include any step which I was relying on an 'automatic' process.
Link to comment

Now, Jonathan, of all those extractions from Raw Digger, do you see anything which will help you in what you are doing?

 

We posted simultaneously. :D

I'm not entirely sure what you want to do. Is there anything above which looks helpful?

 

I think insofar as it is possible you should analyze the raw file and not any TIF or JPG version??

Link to comment

When you create the "linear" image in Dark Table, you are undoing (perhaps not completely) the gamma curve, but the autoscaling is still there? So that is not truly the raw linear image?

 

Here is a very bright Visible image. Its JPG histo hits both the left and right walls. But as per Raw Digger there is no real overexposure and only minor underexposure in the blue channel. (Shadows are noisy in blue channel.)

 

As shot, no edits.

Nikon D750 + 60/28 Micro-Nikkor AFS + Sunlight

f/9 for 1/4000" @ ISO-1250 (I was experimenting......)

chineseStore_20170502romaItalia_52680.jpg

 

Demoisaiced only.

I suppose my point is is that you can't see anything very much in a linear image. Which is why gamma/autoscale is needed to make sense of these raw versions.

EDIT: I have repeatedly overstated the case. Excuse me please! Let's please say that is often very difficult to make out parts of the linear image, but you can usually see something. I've redone this JPG to illustrate that.

chineseStore_20170502romaItalia_52680_linear.jpg

Link to comment

Now, Jonathan, of all those extractions from Raw Digger, do you see anything which will help you in what you are doing?

 

We posted simultaneously. :D

I'm not entirely sure what you want to do. Is there anything above which looks helpful?

 

I think insofar as it is possible you should analyze the raw file and not any TIF or JPG version??

There is a lot which looks helpful Andrea. I would definitely prefer to analyse the RAW file itself if possible, with as little processing as possible (and without resorting to Jpeg and or TIFF). I think the best thing for me to is download RawDigger and have a play, as it looks like it could be very useful.

 

As for what I want to do, that is a good question. This is what I think I want to do - to use the camera as a way of measuring the amount of light hitting the sensor at different wavelengths using my light source/monochromator/integrating sphere setup. Ideally I want to be able to get a 'number' from the image (RAW file) from which I can say how much light is hitting the sensor. This should be as linear scale as possible - so, say I have a scale which goes from 0 (when the image is completely black) to 255 (completely white). I would like to be able to say with confidence that if I have three images which are a '10' a '20' and a '40' that the '20' is twice as bright as the '10' and half as bright as the '40'. Hopefully that makes sense.

Link to comment

And there is so much I've omitted with just the simple display above. You can do all sorts of sample analysis, for example. Be sure to download the PDF manual and look through that, too. :D

 

The raw digger folks are very helpful. Once you have a bit of a feel for the use of the program, then you can ask them for advice.

Link to comment
CORRECTION: I was being too exaggerative above about how hard it was to see anything in a Linear image. The brighter areas are slightly visible in these linear, demosaiced only images. I replaced the linear flower image above with a higher quality one to make this point more obvious.
Link to comment

You want the rgb-values from a selected area in the raw-image.

With RawDigger (I have research version) it is quite easy, you open the raw file, select the desired area and you have an analysis of the selected area, square in the image below.

You get min, max, average and std deviation values for r, g, b, and g2 (?) for the selected area.

These are calculated from the raw, so you also get more digits :)

 

post-182-0-84452600-1518644032.jpg

Link to comment

How would you relate that to "brightness"? I've been trying to think about Jonathan's initial comments.

 

What does it mean for a sensor to be sensitive?

How does exposure length play into this?

With long exposures you can capture more "brightness".

With enough time and patience, as the saying goes, you can get UV light through any lens+filter onto any sensor? Onto some sensors? :lol: I really have no answers here.

Link to comment
I'm using the term 'brightness' for want of a better word here. I'm keeping as much constant as possible (exposure time, iso, f stop of the lens, how the images are recorded), with the aim of trying to extract some data from the image about how much light is hitting the sensor.
Link to comment

Jonathan, I don't think it is the image you want to analyze, but rather the raw data?

 

Raw Digger is counting the number of pixels for each "signal strength" (along the x-axis). But what is being considered as a pixel? The actual physical photosite? Or the "pixel" constructed from the readout and demosaicing data? I have no idea.

 

And the next question -- in these raw histograms what is the unit of measurement for "signal strength"? I have no idea. It would seem to be some measurement which reflects how many photons each photosite is capturing, wouldn't it? So the more photons gathered in a photosite, stronger the signal, and thus the taller the amplitude in the histogram would be for that photosite. But are you looking for tall midtone signals? Or tall bright tone signals (on the right of the histo)? Or would we be looking at the total area under the curve -- the total signal strength over all photosites? And if it is the latter, how do we calculate that?

 

I want to pull a sample histogram from a CC Passport color patch to see what one uniform colour looks like. Stay tuned.

 

****

 

John the Profile edition is basically for creating profiles from standards and color patch charts. Although the more advanced sampling analysis is offered in Profile edition. Being unclear as to what Jonathan wants to analyze, I can't really say which edition is best right now.

Link to comment

Here is a very simple sample analysis. A square sample was taken over each of the Labsphere standards which were photographed under a 3mm thick S8612 blue-green filter which passes UV and visible light and blocks IR light.

 

This is the Raw Digger raw composite of the photograph

Its Jpg conversion, crop and resize were made in Photo Mechanic.

Nikon D610-mod + UV-Nikkor 105/4.5 + Schott S8612-3 + Sunlight

f/5.6 for 1/200" @ ISO-100

 

 d610_uvNikkor_s8612-300_sun_20170709shrCottSwhME_2875_nef01.jpg

 

 

A square sample was drawn over each standard.The Raw Digger sample histogram for the square area was exported.

For each histogram the EV range along the x-axis was adjusted so that each of the R, G, G and B data peaks would show. So please note that EV0 may vary in position from page to page.

For each histogram the maximum height along the y-axis was adjusted so that the entire height of each peak could be seen. So please note that the raw height varies from page to page.

 

 

White 99% reflective sample, center

Raw Digger considered there to be a 10% overexposure in the Green channel and an almost 6% overexposure in the Blue channel. The areas in the photo exhibiting the overexposure are the the white 99% reflective sample and the (next) 75% reflective standard. Thus the Green and Blue data in this histogram has a very narrow width (is confined to a very bright EV region) and is very tall.

I'm not sure what the Red data is telling us! Any thoughts?

 d610_uvNikkor_s8612-300_sun_20170709shrCottSwhME_2875middlePatch.png

 

 

Light Grey 75% reflective sample, upper left

 

 d610_uvNikkor_s8612-300_sun_20170709shrCottSwhME_2875upperLeft.png

 

 

Middle Grey 50% reflective sample, lower left

Clearly the blue and green data is too close to EV3 to be considered midtone. Thus the need for working with a curve tool to produce a final photo more reflective of the actual reflectivity. "-)

 d610_uvNikkor_s8612-300_sun_20170709shrCottSwhME_2875lowerLeft.png

 

 

Dark Grey 25% reflective sample, upper right

 d610_uvNikkor_s8612-300_sun_20170709shrCottSwhME_2875upperRight.png

 

 

Black 2% reflective sample, lower right

 d610_uvNikkor_s8612-300_sun_20170709shrCottSwhME_2875blackPatch.png

Link to comment

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
×
×
  • Create New...