Dirty Pictures

[Andy Perrin]

Nfoto, you make a Photoshop action and then batch process all 100 images. The dots don’t move between shots, only the image does, so the masks shouldn’t need individual adjustments.

 

I took a similar approach with the fixed noise in the TriWave images, with a whiteframe that I subtracted off in a batch process for 46 images. I described it at the bottom of post 1 here:

https://www.ultravioletphotography.com/content/index.php/topic/3290-multispectral-examination-of-a-188-year-old-book-cover-uv-swir/page__view__findpost__p__27685

[nfoto]

I don't use PS for processing the RAW files ... thus using an action in PS is another cumbersome step. The final retouching is required whatever measures are taken previously, as the stacking itself enlarges any defect however small. Running daily hundreds and hundreds of 45MPix files through the workflow taxes my patience even though I have set up dedicated PC modules for the processing. I mainly use Linux boxes for that purpose and they don't run PS, although Photo Ninja works perfectly. Zerene Stacker has a native Linux version as well. So I run my Stackshot setups, feed images to PN, and launch ZS directly thereafter. It all happens in a parallel world as it were to my ordinary Windows systems, but as all modules are on the same network, the output from Zerene is immediately available for my Windows work stations.

 

Bernard: the reason why Zerene retouching can help is the different approaches, and results, of the two stacking methods provide there. Often the DMax output by some magic has avoided the worst of the "moving dots" and then you can carefully brush details in from DMax to PMax images. For backgrounds which are deliberately left defocused, a single image in the stack suffices to brush out most artefacts.

[Andy Perrin]

I don’t use PS for batch processing either, it was a “for instance.” But you may be able to adapt the basic idea of creating a mask and batch processing those parts to your own favorite software.

[dabateman]

Thank you for testing out all cleaning methods.

I have always just done the Q-tip, 99.9% IPA method.

Interesting that the tape method didn't work.

 

Are you sure the troublesome dust is above the converted glass? Maybe when your magnifyer arrives you will know for sure. With all the trouble sounds like it might be under.

[bvf]

Are you sure the troublesome dust is above the converted glass? Maybe when your magnifyer arrives you will know for sure. With all the trouble sounds like it might be under.

 

Actually I tried the methods out on my second-string non-converted camera - didn't fancy exprimenting on my full-spectrum or main full-frame "normal" camera. So the full-spectrum conversion process was not a factor. Also I checked with the person who did my conversion, and they removed the UVIR cut filter but did not replace it with anything else. This means that there is no filter to trap dirt under - but also means the dust is closer to the sensor and so will cast stronger shadows.

 

I was surprised I had so much trouble with the 811 tape (although I tried it twice) - in the link you provided there was a message from a second person who tried that method and it worked fine for them, and that's what encouraged me to have a go. But it was bad news for me.

 

As I mentioned somewhere in this thread, I think I set my target too high with going for "extreme" macro - the cleaning methods (apart from 811 tape) eem fine for normal photography and close up to about 2:1 magnification.

[bvf]

Some success ...

 

I looked at retouching in Zerene, but don't think that's going to help. The DMax images didn't provide anything better that I could clone to the PMax image, and using the stack component images looked like an impossible task - each streak is composed of n dots (n = number of images in stack) and for each dot you would have to find a suitable component source image and clone an area over each dot. Maybe I have misunderstood how to use this tool, but for now I've abandoned it.

 

I did have some success with Canon Digital Photo Professional v4. This is freeware, and whilst it will handle only Canon's RAW formats, it can be used with JPEG and TIFF images. The user interface is designed by someone from the planet Zog, but after going up several blind alleys I did get some success.

 

DPP has its own spot-healing/cloning tool, so you can work your way round an image, removing dust spots. The really useful feature is that having processed an image by doing this and whatever else you want to do, you can copy the "Recipe" (in planet Zog nomenclature) and paste it to as many other images as you want, and then have the same set of actions (inc. dust removal) applied to those images. You can also save the recipe to file for recall later.

 

Probably the best approach is to take one shot of a plain out-of-focus background as the source for the dust spots and to do the spot healing on this image to create the recipe.

 

So by going through this I was able to achieve a significant improvement - see images below. The problems I encountered were:

• The user interface. Having copied the recipe to all the images, I then saved them - only to find that they were no different from the originals. It's not obvious, but you need to use the File > Batch process option to actually effect the changes.
  
• There was a huge number of dust spots in the image because I was using the worst case scenario of full bellows extension (about 23 cm, inc. an extension tube) with an El Nik 50mm lens @ f/8. I started trying to fix every spot, but had to give up and just do the more serious ones. So the final image still has lots of lesser streaks.
  
• It is slow. My PC is quite old now, and the batch process took 2 minutes per image. This stack had 60 images, and I was creating a stereo pair. On a modern, higher-spec PC I guess you could halve this time, and probably speed things up further by fixing less dust spots. But it means (for me at least) I could process only a couple of stereo pairs overnight.
  

 

[bostwickenator]

The lens pen probably won't get you better than what you are already showing but I use it to clean the whole sensor. You just do a bunch of parallel strokes in the same direction across the sensor. The antistatic coating helps prevent new dust pickup while you have the body open. Another option to reduce the rate as which stray dust gets onto your sensor is to use some kind of tacky glue (glue dots or similar) as a dust trap. I've stuck them to the backs of lenses, not on moving parts or elements obviously though. People are saying you need a clean room but maybe you could just make a cleanbox? Basically a HEPA filtered clear storage box with glove ports. I've done similar for other projects.

[bostwickenator]

Also I would suspect your issue with the tape could be because you have a bare sensor. That is going to have exposed micro lenses which I bet cause more tape adhesion.

[bvf]

Thanks for the info, bost... I've thought abut a clean box, but it probably won't happen.

 

The camera I tried the tape on was a non-converted one, so there woould have been a UV/IR Cut filte rin front of the sensor.

[ulf]

Also I would suspect your issue with the tape could be because you have a bare sensor. That is going to have exposed micro lenses which I bet cause more tape adhesion.

Normally the conversion by removing the BG- and dust-shaker glass filters do not expose the micro lenses.

They are a part of the sensor structure, protected with a transparent window, bonded to the sensor-part.

 

The sensor's window is only removed during monochrome-conversions, or to be replaced by a better material.

This window is sitting closer to the sensor's surface with the integrated micro-lenses and Bayer matrix.

Normally they are not exposed after the conversion as the sensor-chip and extremely thin bonding wires, if exposed, are very sensitive.

 

During some conversions the filters are replaced by a second transparent, or IR-pass filter window to restore the optical path length to not affect focussing.

Some times it i possible to readjust the sensor's position instead.

 

The buildup is normally like this:

 

Sensor:

Semiconductor chip

Bayer Matrix

Micro Lenses

Air

Transparent Window

 

Filters:

Ir-suppressing BG-glass

Dust-shaker with dichroic filter

 

"Cleaning the sensor" of a non-converted camera is normally a cleaning of the dust shaker.

"Cleaning the sensor" of a converted camera is either cleaning of the second replacement (filter?)-window or cleaning of the sensor's protection window, depending of type of conversion.

[ulf]

Bernhard-

 

I have not tried this myself yet, but it looks promising:

https://photomacrography.net/forum/viewtopic.php?p=260235&sid=00dfb57c16343efc74fd7772a7e23ab0

[bvf]

Bernhard- I have not tried this myself yet, but it looks promising: https://photomacrography.net/forum/viewtopic.php?p=260235&sid=00dfb57c16343efc74fd7772a7e23ab0

 

I got really excited when I went to that link, Ulf, but there are a couple of issues:

 

1. The process for creating the dust spot mask sounds quite painfull.

2. The dust removal feature is not available in the Personal license - I'd have to upgrade to a Pro licence, which costs more than twice as much.

 

But perhaps I'll trial it.

[bvf]

I have trialled Zerene's Beta dust removal utility. It works, but it took a bit of time to get there, and it's fairly labour-intensive. But it's the only way right now to get streak-free extreme macro shots.

 

I have posted some images below, from the top:

1. A standard Zerene stack (of tiny scabious flowers).
   
2. The same Zerene stack, but using the dust removal option. I still missed a few streaks.
   
3. The dust mask to create image 2.
   
4. As the objective was to create a stereo image, a stereo anaglyph using a traditional image pair (if you have red/cyan stereo glasses).
   
5. Just for interest, a stereo anaglyph creted from a single image using the Zerene Synthetic Stereo utility.
   

You can see from the mask that there was a lot of dust, and it was a bit of a painful task creating the mask. The sensor looked to be clean when viewed through a loupe, but extreme macro really shows up lots of otherwise invisible blemishes. For Zerene, the dots in the mask have to be perfect circles. This means:

• the mask can't be generated automatically. This is a major drawback of the Zerene utility
  
• the dots can't overlap - so you can't get rid of line blemishes.
  
• because saving the mask as a JPEG can cause loss of definition of the edge of the spot, the mask needs to be saved as a TIFF (even if the image files are JPEGs).
  

I'm considering changing my Zerene licence to a Pro licence so I can keep using this utility.

 

On the synthetic stereo image, it's rather good (I had tried it once before and was not impressed at that time), although the depth effect is not so great. But because it is created from a single image stack it does have the advantage of avoiding the problem of bits of the flower moving before and during the creation of a second image stack for the tradtionsl approach.

 

 

 

 

 

[bvf]

Update on the above post:

 

Following discussion with the developer of Zerene, I now have a way of automatically generating dust removal masks. It's relatively straightforward, but there are a few gotchas. I suspect this is a niche interest, so I won't publish details here - if you're interested in the details please message me.

 

I'm so pleased with the results that I have upgraded my licence to the Pro version in order to have permanent access to this feature.

[nfoto]

Zernee users are always on the outlook for new twists to the software, so indeed, do post the procedure here. UVP on its own is already "niche".

 

I gather the dust mask still is confined to the Beta version? I have Zerene Pro and couldn't find the feature there.

[dabateman]

I think there are more people whom do UV macro than UV portrait.

Just not too many members here seem to post.

Please share your recipe. I find this interesting and hopefully we more time can get back to some fun imaging.

 

I have almost finished building my pi camera setup and hopefully can post about that in the next 2 weeks. Which too would be a niche. But may actually be the cheapest way into UV photography.

[bvf]

Birna - yes, it's still only in the latest beta version, which you can download from the Zerene site.

 

Here are my notes - they are cut & pasted from a formatted Word document, so formatting is a bit off. Before the notes, an example to show the improvement using an automatically generated mask: there is still a bit of streaking in the detail-free areas near the right-side corners but these would be trivial to remove (e.g. by blurring in software). Also a stereo anaglyph, if you have red/cyan glasses. The flower is about 5mm across, so the macro magnification is around 5:1, the area where I have found dust streaking to be a real problem. This is a visible image of course - I've had image quality and lighting problems when trying to do this in UV, but I'll be trying that again soon.

 

Enjoy.

 

 

 

 

 

Dust Removal in Zerene Stacker

1. At the Zerene End

 

You will need Beta Version T2020-06-01-1033-beta or later or a full release version dated later than July 2020 to get this capability.

To turn the feature on go to Options > Preferences > Preprocessing and check the Use dust and defects mask box. Select the file that contains the mask (see below).

To use this feature you will need a Pro version of Zerene. With Student and Consumer licences you can trial it for 30 days.

2. Make a Grey Background Image (GBI)

 

You will need an image which contains your current dust spots. You could use one of the images that you are going to stack, but it is easier if you make an image (referred to as GBI here) with a plain grey background using the same lens, aperture, and bellows/tube extension (if relevant) that will be used for your real images.

IMPORTANT: make sure the GBI is the same image type (e.g. JPEG) as the images you are going to stack. If you create, for example, a TIFF version of the GBI from RAW output from your camera, it will probably have slightly different dimensions (in pixel count) from a JPEG output by your camera.

IMPORTANT: Zerene cannot detect image orientation, so the orientation of the GBI (and the mask created from it) must be the same as that of the images being stacked. If, for example, you made the GBI in landscape but your stacking images are in portrait then you will have to rotate either the GBI or the stacking images.

3. Making a Dust Mask manually

 

This description uses GIMP, but a similar process must be possible using Photoshop.

• Load the GBI into GIMP.
  
• Create a 2^nd layer, pure white
  
• Move the GBI to the top and set Opacity to about 90%.
  
• Activate the lower, GBI layer.
  
• Select the Pencil tool and set a size big enough to cover typical dust spots – 20 pixels worked well for me. Set pencil colour to black.
  
• Zoom the display to 200% or more.
  
• Scroll around the image and click on the centre of each dust spot. This should create a black disk under the dust spot on the lower white layer. By having the GBI layer Opacity at 90%, you can faintly see this black disk.
  
• For larger blemishes:
  
  • Increase the Pencil size, or
    
  • Keep the left mouse button down and paint over the blemish.
    

  [*]When you have done all the dust spots, delete the top, GBI layer.

  [*]Save the lower white layer as your manual mask. IMPORTANT: save this as an uncompressed TIFF.

4. Making a Dust Mask automatically

 

There are 2 stages to this:

1. Capture the dust spots
   
2. Expand the Dust Spots,
   

4.1 Capture the Dust Spots

 

There will be multiple ways of achieving this, but this approach uses GIMP and is easy.

• Load the GBI into GIMP
  
• Set sharpening to maximum ((Filters>Enhance>Sharpen (Unsharp mask)).
  
• Set image to Greyscale (Image>Mode>Grayscale)
  
• Set Contrast to maximum (Colors>Brightness-Contrast)
  
• Set Brightness to the minimum that does not cause noise dots to appear (best to view the image at 200-400% so you can see this.)
  
• You should now have a pure white field with pure black dust spots. Save this interim mask as a TIFF file.
  

4.2 Expand the Dust Spots

 

This ensures that the dust spots in the mask are:

1. slightly larger than the dust spots on the target images
   
2. are solid. (In macrophotography, the dust spots may be concentric light and dark rings because of diffraction.)
   

This method used Photoshop Elements – it may be possible to achieve the same using GIMP, but at first sight I could not see all the necessary options.

• Load the GBI into Photoshop.
  
• Select the Magic Wand tool.
  
• Make sure the tool property Contiguous is unchecked.
  
• Enlarge the image ( Ctrl + ) so that the dots are a good working size on the screen.
  
• Click on any black dot. This will select all black dots in the image. (If you look carefully, you can see shimmering outlines).
  
• Go to Select>Modify>Contract and set Contract By to 2 pixels (you can play around with this setting later if you want). This causes very small dots to be ignored.
  
• Go to Select>Modify>Expand and set Expand By to 5 pixels (you can play around with this setting later if you want). This enlarges the dust spots that have not been ignored.
  
• Go to Edit>Fill Selection and select Contents Use = Black and Opacity = 100%.
  
• Save the resulting image as a TIFF – this is your Dust Mask.
  

5. Updating the Dust Mask

 

The Dust Mask file can be retained for use in subsequent sessions. However, there are likely to be new dust spots which cause new dust streaks to appear. You can update the Dust Mask manually by:

• Loading the Dust Mask into GIMP.
  
• Load one of your images into GIMP as a top layer, and set opacity to 90%.
  
• Expand the view to 200%
  
• Set the Pencil Tool as described above for manual masks
  
• Find the dust spot(s) that are causing the streak and click on them.
  
• Delete the top layer and save the lower layer as a TIFF as your updated Dust Mask.
  

[Andy Perrin]

Bernard, this is almost word for word what I was attempting to describe above? I guess I'm glad people understood the method in the end, but I feel frustrated that I couldn't communicate it!

[Andrea B.]

Bernard, thank you for the tutorial on dust removal in Zerene Stacker. If you would like to make a copy of that as a stand-alone topic (for easier reference), please do so when you have a moment.

 

---

 

This is my opinion only. YMMV

 

Of all the methods of cleaning, only a Lens Pen/Sensor Pen and the (diluted?) methanol solution sold as Eclipse and used to moisten a Pec Pad-wrapped swab work really well.

Isopropyl alcohol does not evaporate as quickly as Eclipse, so it often leaves streaks and residues. Same goes for de-ionized water or certain "soap" solutions which are used to clean monitor glasss.

 

Sensor Cleaning Method: Blow Out, Wet Clean, Dry Clean, Blow Out

 

1. Blow out the dust from the sensor chamber.

You want to get rid of any fibers, hairs and "big" grit before swabbing so that you do not scratch the sensor glass.

 

2. Fold a Pec Pad and rubber-band it over a plastic swab form. Put ONE drop of Eclipse on it. ONE !!!!

Pull left over the sensor with one side of the Pec Padded swab and without lifting pull back right over the sensor with the other side of the wrapped swab. Now stop because your Pec Pad has been contaminated and its fibers have been loosened.

Make sure the sensor glass is dry. (If it is not, then maybe you used two drops of Eclipse??)

 

3. If needed (and it usually is), then repeat number 2 with a clean fresh Pec Pad and ONE drop of Eclipse.

 

4. Prime the Sensor Pen by rotating it in the graphite cap. "Polish" the clean sensor glass by placing the pen tip on center of glass and make a spiral cleaning motion out to the edges of the glass.

(Yes, I know the glass is a rectangle, but you get what I mean by starting in the center and working your way out to the edges.)

 

5. Blow out the graphite dust from the sensor chamber.

 

6. Make test shots to determine whether you have removed most of the dirt, specks, fibers, pollen and whatnot. I have had whatnot be a small insect when out shooting in the field !!! Yuk.

 

I have had sensors for which I wrapped about a dozen swabs before I finally could get the thing clean.

 

---

 

Notes: Again, my humble opinion on the following points. YMMV

• Cotton swabs no matter how tightly wrapped tend to unwind and transmit fibers. They also concentrate the liquid cleaner and can sometimes leave wet streaks.
  

• Old-fashioned lens tissue can be used in place of Pec Pads.
  

• Sensor pens should be replaced regularly because you cannot always tell whether anything gritty has gotten embedded in its felt tip. If the edge of the felt tip looks frayed or wrinkled, replace. Shine a light in the cap to see if the graphite has been depleted. Dedicate one felt/graphite pen to the sensor and don't use it on anything else. You don't want to get the felt full of (possible) oils, moisture and other crud which might be on a lens cleaned with that pen.
  

• Nikon used to use the end of a chopstick wrapped with a Pec Pad or lens tissue and ONE drop of Eclipse on it. (Or whatever solution was being used at the time.) Then the chopstick was spiraled out from the center ONCE. Any repeats were made with clean Pec Pads. Please make sure you have a smooth-ended plastic chopstick or a very, very smooth-ended wooden one.
  

• I usually make sure there is a double/quad layer of Pec Pad over either swab or chopstick.
  

• Delicate little brushes which must first be electrostatically charged do work for sweeping out fibers, dust and anything not stuck to the glass. But it is so easy to accidentally bump a brush against some part of the inner chamber and pick up black (rubber?) contamination thus rendering that expensive little brush useless.
  

• Expensive sensor lights are not really needed. I have a couple but prefer a torch. Sometimes a little flashlight held between the teeth. (Not good for teeth? La!)
  

• If you are currently experiencing very low humidity and have lots of dust or floating pet hair in your house, then run the shower for a couple of minutes in the bathroom and wait about 10(?) minutes. There should be no floating dust in there after that. You don't want too much humidity in the bathroom else you'll get condensation. Just enough to quell the dust motes.
  

---

 

And, yes, I sometimes get crazy and use canned air to blow off the sensor. I seem to think that I can use canned air without getting a blast of that glop one gets when holding the can at the wrong angle. Someday I will be so sorry for using canned air. So don't do this, OK? :devil:

Please don't tell anyone that I do that.

[Andy Perrin]

6. Make test shots to determine whether you have removed most of the dirt, specks, fibers, pollen and whatnot. I have had whatnot be a small insect when out shooting in the field !!! Yuk.

I have spent a lot of time in my life debugging, but never with actual bugs.

[bvf]

Bernard, this is almost word for word what I was attempting to describe above? I guess I'm glad people understood the method in the end, but I feel frustrated that I couldn't communicate it!

 

Andy, I'm assuming you're referring to your earlier post that said:

 

Bernard, I would just take a photo of a blank surface using the same settings as the rest of the photos, use it to make a mask by thresholding, and use content aware fill on the masked regions of each photo. In Photoshop this could even be automated.

 

Yes, you're right - you've proposed the same process. My PS and workflow skills are somewhat rudimentary, so what made it do-able for me was the new facility in Zerene to provide the automation bit, and hand-holding by Rik @ Zerene to flesh out the process and patiently explain how to do this in PS (BTW - I've worked out how it can all be done within GIMP now). And then there were the detailed gotchas that the conversation with Rik brought to light and which had wasted quite a lot of effort - the need to expand the dust dots, the difference in size between TIFFs and JPEGs, the need to have the mask as a TIFF, the need to synchronise the rotation of mask and stacking images.

 

Canon provide a Dust Deletion Data tool which I think uses the same principle - but I just couldn't get it to work (and it took forever while it was failing to working!).

[bvf]

Bernard, thank you for the tutorial on dust removal in Zerene Stacker. If you would like to make a copy of that as a stand-alone topic (for easier reference), please do so when you have a moment.

 

---

 

This is my opinion only. YMMV

 

 

I'll do that, Andrea.

 

Good tips on sensor cleaning.

 

As I've mentioned before, the dust problem is an order of magnitude worse when you get to doing high-magnification macro. I've just done a high-magnification (around 5:1) macro stack and the sensor looked filthy - and I used the Zerene dust removal feature to save it. Then I did a close-up stack at a magnification of about 1:2, and there wasn't a dust spot to be seen on the output.

 

The dustspots that cause a problem in migh-magnification macro are too small to be seen with a loupe, and I find that trying to clean them "blind" works in the sense that the dust spots are removed, but a whole load of new ones are deposited (or perhaps the old ones are just moved around).

 

I think would give up on high-magnification macro if the Zerene dust removla tool wan't there.

[ulf]

I finally found the product for a paint on sensor cleaning product:

https://www.sensor-film.com/shop/

I'm not sure I would dare to try it.

Also not for Sony cameras!

[OlDoinyo]

If you want to create a dust map of your sensor, a pinhole is the perfect tool to do so. You will find specks you did not even realize were there.

[Andy Perrin]

I've been going through hell trying to get my sensor clean lately. (Bernard: I feel your pain, man!)

 

I tried the SensorKlear pen method with the loupe after the testimonials here, and it mostly just seemed to swish the dirt around and sometimes even drop fibers on the sensor. Possibly user error there - I got a bit better at just using the edge of the pen after awhile. However I kept ending up with areas where I basically just swept all the dust specks together into a pile but couldn't pick up or dislodge the pile. I finally decided I needed a professional cleaning.

 

I went to Hunt's Photo in Kenmore and asked them to clean it. There were only two employees, and by the sound of it, they are the last two left. One of them started to take the camera from me for cleaning and then the other started to remonstrate with him for saying he could get it back to me in an hour. ("What if we get other customers?") Anyway, they ultimately agreed to do it in an hour. As I was going out, I noticed the guy was working on it right on the shop counter, not under a hood or anything — he just popped the lens right out and got to work. I thought that was a bad sign, and I was right. They did give me the camera back in an hour, and I asked if he got everything because I'd had trouble. He said he did, and made a show of taking a photo and scrutinizing it on the back of the camera (as if you could see anything that way). When I got home I noticed he'd left the lens at F/2.8. Anyway, I figured I didn't care if he wanted to be show-bizzy so long as he got it clean, but when I took a test shot and checked it out, it was WORSE than when I gave it to him.

 

I immediately called and asked for a refund, which he said he gave me (although it has not shown up yet -- I will give it a day or two since they sometimes take a while). Long story short, I won't be going back there for sensor cleanings anyway.

--

 

Just now I managed to get the larger bits of dust out, so it's "only" as bad as when I started now.