• Ultraviolet Photography
  •  

Lens and camera tests with sparticle and monochromator

60 replies to this topic

#41 JMC

    Member

  • Members+G
  • 1,012 posts
  • Location: London, UK

Posted 19 August 2018 - 15:47

You also need to keep in mind how the image is handled and the data extracted Jim. Quick demonstration. Take 8 Spectralon diffuse reflectance calibration standards with reflectances between 2% and 99%. Put them out in sunshine, and take a picture of them with Monochrome 5DSR camera, Asahi UAT 85mm lens, and Baader U filter. ISO800, f8, and 1/30s exposure, saving both RAW file and a JPEG within the camera, and you get something that looks like this;
Attached Image: 0I8A4864 ISO800 small.jpg

Take the JPEG and extract grey scale scores from the 8 tiles in ImageJ (numbers between 0 and 255), and for the RAW file put it into RawDigger and open as RAW composite. Then read the channel information from each of the 8 tiles, and average them together to get and average response. Plot both of these against reflectance of the tiles, and you get these two graphs (JPEG data on the left RAW composite data on the right);
Attached Image: Graph closeup JPEG RAW.jpg

I'll need to go and re read the Garcia paper, but depending on how they handled the RAW files they could be dealing with data which has already been heavily processed, which they then need to process further to get the curve out of it again. This is why I do all my work using RAW composite files as I get a linear response for changes in intensity.

Edited by JMC, 19 August 2018 - 15:48.

Jonathan M. Crowther

http://jmcscientificconsulting.com

#42 Jim Lloyd

    Member

  • Members(+)
  • 312 posts

Posted 19 August 2018 - 17:17

Thanks Jonathan - a very nice demonstration.

I'm a bit puzzled now. I now recall that actually the sensors are quite linear inherently. This is what Garcia et al say in their paper on measurement of the RGB values:

"Images were recorded in the native raw format for each cam- era and subsequently
converted into the sRGB IEC61966-2.1 color space and encoded into an 8-bit scale
employing Adobe Camera Raw plug-in (v 6.2) for Photoshop version CS5 (Adobe
Systems, San Jose, CA)."

So I guess it is that Adobe Raw plug-in conversion that is introducing the non-linearity in their case. It seems a bit crazy then that they have gone to such lengths to correct for it - I guess maybe they didn't have any other way to access the actual raw values?

Anyway - regarding my work, then yes for the monochromator work I used the RAW RGB values from rawdigger (I think I am right in saying the values given are raw composite, independently of the display method used)

So I think then the spectral response curves should be OK and my concerns over non-linearity there are unfounded.

To compare my predictions with the model I applied White balance using Photoninja to the sparticle images (based on the PTFE tape in the image) - I guess this is where the non-linearity comes in?

I suppose I should check the predictions without white balance against the raw composite image of the sparticle to clarify ...

Edited by Jim Lloyd, 19 August 2018 - 17:23.


#43 Cadmium

    Member

  • Members+G
  • 3,595 posts

Posted 19 August 2018 - 19:32

(this is pertaining to several posts back)
Given that all the BP filters are shot at the same time, with the same exposure, it seems there is ample sensor sensitivity to record the green at about the same brightness as the other BP filters.
However, This is from back lighting, not reflected UV from objects.
I can't say what amounts of 340BP10 is being reflected from objects, but that range of UV light is illuminating objects, and it would seem that the sensor is able to see strong enough compared with the other bandpass sections of the range.

#44 Jim Lloyd

    Member

  • Members(+)
  • 312 posts

Posted 20 August 2018 - 15:07

So further to earlier post I have now investigated further.

The sparticle image seen in Rawdigger as raw composite is shown here:

Attached Image: sparticle2-8-20180820-114130-RawDigger-ScreenShot.png

(This is 5 sec exposure iso 800, Nikkor EL 80 mm (old style) D3200 + UG1 2mm + BG40 2 mm)

In this shot some of the filters are just over-saturating (in the R channel). but if I used an example with lower exposure some of the filters are very difficult to see.

I used rawdigger to obtain RGB values (raw values) average in 100x100 squares (on equivalent image to above but with 2 sec exposure to avoid over saturation). I just used one of the G values.

Then I scaled these according to the total area under the filter transmission curves to compensate for the variable amounts of total transmission

I have plotted these together with my monochromator measurements (to recap these are in 10 nm bands, values also obatined as raw values using Rawdigger, same lens, camera and BG40 2mm filter, UG1 2mm filter and sunlight modeled afterwards)

These are shown below - the overall normalization is consistent within each group of curves, but arbitary

.Attached Image: Capture un wb responses.JPG


This looks good to me - not an exact match, but sufficiently close to me to indicate that the response curves are essentially correct as they match based on two separate measurement methods.

Of course I should have been more aware of some of this before, but the key learning point for me is that when the image is rendered from the raw values, as well as applying the white balance correction, it is likely that a gamma correction will also be applied (and has been in the examples I have shown in previous posts).

This is explained nicely here (although of course you all knew this already ...)

If one also considers that green appears perceptually brighter for the same physical value (maybe opening a can of worms here ...) then we can explain why the 340 nm band pass filter can appear quite bright (after wb and gamma correction) despite the sensor response being pretty low at this wavelength. However in a real life scene it is unlikely that there will be much reflecting just in a narrow band around 340.

Quick demo of apparent brightness of green:

Here RGB set separately to the same value, but green looks brighter.

Attached Image: Capture RGB.JPG

Incidentally along the way in this latest work I discovered a small error what I had done before relating to the green channels - such that the values were all slightly (~10%) higher than they should be. I am never sure really whether to use the average of G and G2 or sum them, but if not stated I have taken the average. In the above I actually compared the average of G and G2 for the monochromator measurement with just the G values as I found that there was very little difference between G and G2.

Edited by Jim Lloyd, 20 August 2018 - 15:09.


#45 Jim Lloyd

    Member

  • Members(+)
  • 312 posts

Posted 20 August 2018 - 15:33

For completeness here is the same image as above when WB applied based on the teflon tape (top row far right) using view NX2

Attached Image: Capture nikkor 80 el wb viewnx2.JPG

#46 Jim Lloyd

    Member

  • Members(+)
  • 312 posts

Posted 20 August 2018 - 18:26

If I apply a gamma correction after white balance to my monochromator data this is what I get:

(What I mean is that first I normalize the curves so that each have the same area (=1) then apply the formula Vnew = Vold ^1/gamma - where Vold is the original value, Vnew is the new value and gamma = 2.2

Attached Image: Capture nikkon el with gamma.JPG

(for ease of comparison here is the graph without gamma correction)

Attached Image: Capture fitted responses.JPG

Seems plausible that this accounts for the observed sparticle images

#47 Jim Lloyd

    Member

  • Members(+)
  • 312 posts

Posted 28 August 2018 - 18:56

I have done some further work and refinements on this and got about as far as I can take it now. Very useful learning experience for me and hopefully of interest to others.

A quick recap: I measured camera response using Xe arc lamp based radiation monochromator - Camera full spectrum D3200 + UG40 2mm + Nikkor EL 80 mm. Then added model of UG1 2mm and sunlight - this response was fitted to bi-normal distribution (RGB channels).
To check that model appears reasonable I imaged a series of narrow band filters in sunlight and used the response model and filter transmission measurements to predict appearance of filters ("sparticle).

Here are the predicted and actual filter colours (shown as image and graphically)

Attached Image: Capture raw spartucle responses.JPG

This is improvement on previous - I updated the solar model parameters (see here) as I previously had the wrong time of day. This can have a big impact on relative irradiance at different wavelengths.
I also made some changes to the filter transmission. The major change was a reduction to the transmission through the 365 nm filter - this may be a coincidence but it seemed to be too high roughly in proportion to its area (its bigger than the others)

As this looked like a reasonable match to the raw RGB values, I moved on to predicting colours after white balancing.
Previous work showed that the lower filter images at either end needed to be boosted so I introduced a gamma correction of 2.2. This was done after white balancing, individually for RGB channles.
White balance was simply determined by measuring the area under the response model and scaling the blue and green channels in inverse proportion to their ratios to the red channel.

This is the result compared to the image obtained in View NX2 by white balance to teflon tape

Attached Image: Capture sparticle prediction.JPG

This looks convincing enough to me.

I will come back and add more comments later ...

#48 Andrea B.

    Desert Dancer

  • Owner-Administrator
  • 7,957 posts
  • Location: USA

Posted 28 August 2018 - 19:03

Are you using any color profiles for this system of camera+lens+filter+light?

Something which might be useful....take each of your predicted and actual colors and find their color wheel reference color at full sat and brightness. It isn't all that important to do this, but does sometimes give a better "feel" for what a color is. For example your 407 color bars are so dark that I cannot see what color they might be.
Andrea G. Blum
Often found hanging out with flowers & bees.

#49 Jim Lloyd

    Member

  • Members(+)
  • 312 posts

Posted 28 August 2018 - 21:06

I have to admit ignorance here, as I am not sure what you mean by colour profiles. Could you educate me?

I have been working with RGB, but I will have a look at HSL values

#50 Jim Lloyd

    Member

  • Members(+)
  • 312 posts

Posted 28 August 2018 - 21:14

Reflections on this work:

1. As proof of principle / pilot work it has been succesful
2. Monochromator method works, akthough there is probably some low level of stray light, which I would need to investigate and quantity.
3. To compare with sparticle accurately, quantifying fiter relative transmission is important and a potential source of error
4. Important to get accurate solar spectrum, although white balance will reduce solar spectrum variations
5. Exact method used to convert from RAW to rendered white balanced image varies with software in ways that might not be clear to user.

#51 Jim Lloyd

    Member

  • Members(+)
  • 312 posts

Posted 29 August 2018 - 14:20

View PostAndrea B., on 28 August 2018 - 19:03, said:

Are you using any color profiles for this system of camera+lens+filter+light?

Something which might be useful....take each of your predicted and actual colors and find their color wheel reference color at full sat and brightness. It isn't all that important to do this, but does sometimes give a better "feel" for what a color is. For example your 407 color bars are so dark that I cannot see what color they might be.

Here is the same set of actual and predicted colours as above

Attached Image: Capture sparticle predictions at 125 lum.JPG

Here i have set lumiance to 125 in all cases (I am using Excel which has luminance values between 1 and 255 - so about 50%)

This shows colour predicition is very close.

#52 Andrea B.

    Desert Dancer

  • Owner-Administrator
  • 7,957 posts
  • Location: USA

Posted 29 August 2018 - 18:22

Thanks for indulging me! That was interesting to see. I do know we won't ever see much of those fully saturated, fully bright colours, if any. But I had no "feel" for those dark versions. :D
Andrea G. Blum
Often found hanging out with flowers & bees.

#53 Cadmium

    Member

  • Members+G
  • 3,595 posts

Posted 29 August 2018 - 19:22

Jim, Not sure why your green and yellow look so brown when white balanced. Similar to you, I use very thin PTFE installed in one of the Sparticle holes,
it should be about the same as the tape I guess, never tried the tape that way though.
It still works for what it is for, finding the depth of transmission, the color doesn't matter for testing transmission depth.
I don't see the reason for RAW colors, they are white balanced too, everything has a white balance, but the RAW white balance is just not optimal, but still works pretty good for testing depth I guess.
By the way, I think I have an extra Sparticle holder around here somewhere if you want to pay for shipping, I would give it to you. It is PTFE and I would include the thin PTFE I use for white balance.
Fist Class would probably be about $12? to you, I think.
What size diameter on the outside are your BP filters? 12.5mm diameter?

Edited by Cadmium, 29 August 2018 - 19:25.


#54 Cadmium

    Member

  • Members+G
  • 3,595 posts

Posted 29 August 2018 - 19:29

I have a couple that look like this around here somewhere... they come apart so you can change filters.
Attached Image: DSC_3356_a_8.jpg

Edited by Cadmium, 29 August 2018 - 19:31.


#55 Andrea B.

    Desert Dancer

  • Owner-Administrator
  • 7,957 posts
  • Location: USA

Posted 29 August 2018 - 20:57

Not sure why your green and yellow look so brown when white balanced.

Our color perception is non-linear. At a certain point, as a color is lightened or darkened, we perceive a hue shift even though there isn't one. The Abney effect.

Example: As you desaturate pure blue by adding white, eventually the eye sees "lavender" or some pale purplish hue.

When I see yellow becoming darker, it eventually looks olive green to me rather than dark yellow.

One of Jim's colors is yellow. The other two have lesser amounts of green. As these colors darken, we see the hue shift.
Attached Image: colorDot.jpg

Actually there is a bit of an anomaly there.
The green in the color associated with 342 is 225.
The green in the color associated with 356 is 195.
Then in the color associated with 366, the green increases again to 204.
From top to bottom, the color wheel bumps from 53° to 46° and then back up to 48°.
Weird.
Could just be the effect of stuffing it all into a color space.
There are a lot of reasons why we need to be very very careful with these false color analyses. It is useful to understand color spaces and how they are constructed. I do not wish to again belabor the lack of a 1-1 wavelength-color mapping, but that is not the only snag which needs explaining to make this all work. For example, what is the proper intent to be used in the color space to handle anything which might lie outside the gamut? Oh well.....another day perhaps....I've barely scratched the surface myself in trying to understand color - whether false or non-false.
Andrea G. Blum
Often found hanging out with flowers & bees.

#56 Jim Lloyd

    Member

  • Members(+)
  • 312 posts

Posted 29 August 2018 - 21:42

View PostCadmium, on 29 August 2018 - 19:22, said:

Jim, Not sure why your green and yellow look so brown when white balanced. Similar to you, I use very thin PTFE installed in one of the Sparticle holes,
it should be about the same as the tape I guess, never tried the tape that way though.
It still works for what it is for, finding the depth of transmission, the color doesn't matter for testing transmission depth.
I don't see the reason for RAW colors, they are white balanced too, everything has a white balance, but the RAW white balance is just not optimal, but still works pretty good for testing depth I guess.
By the way, I think I have an extra Sparticle holder around here somewhere if you want to pay for shipping, I would give it to you. It is PTFE and I would include the thin PTFE I use for white balance.
Fist Class would probably be about $12? to you, I think.
What size diameter on the outside are your BP filters? 12.5mm diameter?

Re: RAW - in this case when I am refering to RAW there is no white balance applied. In the sparticle images they are the values given in Rawdigger and displayed as raw composite. In the model predictions they are derived from the camera response and filter transmission.

Re the colours - I have spent ages looking at this - basically there are a few factors which interact so its complicated. And there is of course a difference between my prediction and the actual.

Regarding my intitial measurements it is likley there is some stray light which has the effect of giving some false response which overall has the effect of reducing saturation in the predictions. Although this effect is small it will get amplified during the gamma correction and white balance process.

Having said that, I do find that the clarity of the yellow and green seen in the actual sparticle image can be rather "dirty", but the exact colour percived depends a lot on the software being used and exactly how its done.

In the end I am satisfied that the predicted colours and actual colours are close enough to give me confidence that my plots of the sensor spectral response are close to the truth (say within 10%?).

#57 Andy Perrin

    Member

  • Members+G
  • 3,142 posts
  • Location: United States

Posted 29 August 2018 - 21:43

Quote

At a certain point, as a color is lightened or darkened, we perceive a hue shift even though there isn't one. The Abney effect. When I see yellow becoming darker, it eventually looks olive green to me rather than dark yellow.

I ran into this effect with my mom - we disagreed over the hue in the buoy here. I was pulling for yellow, she declared it was green, and because I have some color blindness, I couldn't argue against it (even though my color picker also said yellow).

Attached Image: _DSC8812 aerochrome edit small.jpg

#58 Andrea B.

    Desert Dancer

  • Owner-Administrator
  • 7,957 posts
  • Location: USA

Posted 29 August 2018 - 21:55

yes, I see it going a bit towards lime green myself.
Andrea G. Blum
Often found hanging out with flowers & bees.

#59 Jim Lloyd

    Member

  • Members(+)
  • 312 posts

Posted 29 August 2018 - 21:58

and to make matters more complicated we dont percieve colours in isolation, but perception depends on other colours around the object of interest. The RGB receptors (I know that is inaccurate terminology) work to detect R-G and Blue-Yellow(R+G) differences rather than detecting each channel separately.

#60 Cadmium

    Member

  • Members+G
  • 3,595 posts

Posted 30 August 2018 - 01:23

I don't know why your 340/350/360 range is all brown, not green or yellow like mine, and other's show when optimally white balanced.
Yours are the only examples I have seen from anyone using this method that don't show a green and yellow.
Your lens filter stack 'should' be OK for removing any OOB leak.
Basically, I am saying your brown filters 'should' be green and yellow, should meaning according to results I have seen other people show.
Not sure what the difference is.
Your examples in the posts below shows more of a green/yellow.
#7
http://www.ultraviol...dpost__p__22874
#38
http://www.ultraviol...dpost__p__23193

Attached Image: UG1_2mm_BG40_2mm_DIA_LIN.jpg

Edited by Cadmium, 30 August 2018 - 01:35.